[JBoss JIRA] (RFPL-1467) @Test annotation param dependsOnMethods in Metamer's ftest causes IllegalStateException
by Jiří Štefek (JIRA)
[ https://issues.jboss.org/browse/RFPL-1467?page=com.atlassian.jira.plugin.... ]
Jiří Štefek resolved RFPL-1467.
-------------------------------
Resolution: Out of Date
Was fixed with resolving of https://issues.jboss.org/browse/RFPL-3268
> @Test annotation param dependsOnMethods in Metamer's ftest causes IllegalStateException
> ---------------------------------------------------------------------------------------
>
> Key: RFPL-1467
> URL: https://issues.jboss.org/browse/RFPL-1467
> Project: RichFaces Planning
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Components: QE
> Environment: Metamer 4.1.0-snapshot, r.22501
> Reporter: Ján Jamrich
> Assignee: Pavol Pitonak
> Fix For: 5.0.0.Alpha4
>
>
> When tried set dependsOnMethods param to @Test annotation in Metamer's ftests, get IllegalStateException thrown from MatrixConfigurator.
> There aren't any other methods annotated by @Test with dependsOnMethods param, so its possible that this issue is in Metamer longer time without being discovered.
> By debugging observed that list of methods set to be executed by TestRunner depends on this param. In other word, when set dependsOnMethods param to @Test annotation - this method (on which was this param set to annotation) doesn't appear in list of methods being executed (LinkedList<Method> methods attribute of MatrixConfigurator).
> This attribute is initialized in method
> public List<IMethodInstance> intercept(List<IMethodInstance> methodInstances, ITestContext context)
> which is called directly from org.testng.TestRunner, which give param List<IMethodInstance> methodInstances depending on dependsOnMethods param set to @Test annotation as well.
> This behavior should be ok, since methods which are not depending on another else method should be executed first.
> But what seems worse is that when run this again - for method anotated by @Test with param dependsOnMethod, then list of methods which going to be executed is empty - do no test method called.
> And in case when all methods except one depends on another method, different exception is result of run testsuite:
> Running TestSuite
> org.apache.maven.surefire.booter.SurefireExecutionException: null; nested exception is java.lang.NullPointerException: null
> java.lang.NullPointerException
> at org.richfaces.tests.metamer.ftest.MetamerTestInfo.getConfigurationInfo(MetamerTestInfo.java:50)
> at org.richfaces.tests.metamer.ftest.MetamerTestInfo.getConfigurationInfoInParenthesses(MetamerTestInfo.java:77)
> at org.richfaces.tests.metamer.ftest.MetamerSeleniumLoggingTestListener.getMessage(MetamerSeleniumLoggingTestListener.java:35)
> at org.jboss.test.selenium.listener.SeleniumLoggingTestListener.logStatus(SeleniumLoggingTestListener.java:82)
> at org.jboss.test.selenium.listener.SeleniumLoggingTestListener.onTestFailure(SeleniumLoggingTestListener.java:55)
> at org.testng.internal.Invoker.runTestListeners(Invoker.java:1634)
> at org.testng.internal.Invoker.runTestListeners(Invoker.java:1618)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1101)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:137)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:121)
> at org.testng.TestRunner.runWorkers(TestRunner.java:1030)
> at org.testng.TestRunner.privateRun(TestRunner.java:709)
> at org.testng.TestRunner.run(TestRunner.java:579)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:331)
> at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:326)
> at org.testng.SuiteRunner.privateRun(SuiteRunner.java:288)
> at org.testng.SuiteRunner.run(SuiteRunner.java:193)
> at org.testng.TestNG.createAndRunSuiteRunners(TestNG.java:910)
> at org.testng.TestNG.runSuitesLocally(TestNG.java:879)
> at org.testng.TestNG.run(TestNG.java:787)
> at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:62)
> at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:141)
> at org.apache.maven.surefire.Surefire.run(Surefire.java:177)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:345)
> at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:1009)
> So this should be issue related to TestNG itself (current version 5.12.1), tried 5.13, but the same behavior.
> There is stack trace:
> java.lang.IllegalStateException: can't find more configured methods
> at org.richfaces.tests.metamer.ftest.MatrixConfigurator.configureMethod(MatrixConfigurator.java:150)
> at org.richfaces.tests.metamer.ftest.MatrixConfigurator.beforeInvocation(MatrixConfigurator.java:101)
> at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:62)
> at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:141)
> at org.apache.maven.surefire.Surefire.run(Surefire.java:177)
> at org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:345)
> at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:1009)
> ... Removed 22 stack frames
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 10 months
[JBoss JIRA] (RFPL-1467) @Test annotation param dependsOnMethods in Metamer's ftest causes IllegalStateException
by Jiří Štefek (JIRA)
[ https://issues.jboss.org/browse/RFPL-1467?page=com.atlassian.jira.plugin.... ]
Jiří Štefek updated RFPL-1467:
------------------------------
Fix Version/s: 5.0.0.Alpha4
(was: 5-Future)
> @Test annotation param dependsOnMethods in Metamer's ftest causes IllegalStateException
> ---------------------------------------------------------------------------------------
>
> Key: RFPL-1467
> URL: https://issues.jboss.org/browse/RFPL-1467
> Project: RichFaces Planning
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Components: QE
> Environment: Metamer 4.1.0-snapshot, r.22501
> Reporter: Ján Jamrich
> Assignee: Pavol Pitonak
> Fix For: 5.0.0.Alpha4
>
>
> When tried set dependsOnMethods param to @Test annotation in Metamer's ftests, get IllegalStateException thrown from MatrixConfigurator.
> There aren't any other methods annotated by @Test with dependsOnMethods param, so its possible that this issue is in Metamer longer time without being discovered.
> By debugging observed that list of methods set to be executed by TestRunner depends on this param. In other word, when set dependsOnMethods param to @Test annotation - this method (on which was this param set to annotation) doesn't appear in list of methods being executed (LinkedList<Method> methods attribute of MatrixConfigurator).
> This attribute is initialized in method
> public List<IMethodInstance> intercept(List<IMethodInstance> methodInstances, ITestContext context)
> which is called directly from org.testng.TestRunner, which give param List<IMethodInstance> methodInstances depending on dependsOnMethods param set to @Test annotation as well.
> This behavior should be ok, since methods which are not depending on another else method should be executed first.
> But what seems worse is that when run this again - for method anotated by @Test with param dependsOnMethod, then list of methods which going to be executed is empty - do no test method called.
> And in case when all methods except one depends on another method, different exception is result of run testsuite:
> Running TestSuite
> org.apache.maven.surefire.booter.SurefireExecutionException: null; nested exception is java.lang.NullPointerException: null
> java.lang.NullPointerException
> at org.richfaces.tests.metamer.ftest.MetamerTestInfo.getConfigurationInfo(MetamerTestInfo.java:50)
> at org.richfaces.tests.metamer.ftest.MetamerTestInfo.getConfigurationInfoInParenthesses(MetamerTestInfo.java:77)
> at org.richfaces.tests.metamer.ftest.MetamerSeleniumLoggingTestListener.getMessage(MetamerSeleniumLoggingTestListener.java:35)
> at org.jboss.test.selenium.listener.SeleniumLoggingTestListener.logStatus(SeleniumLoggingTestListener.java:82)
> at org.jboss.test.selenium.listener.SeleniumLoggingTestListener.onTestFailure(SeleniumLoggingTestListener.java:55)
> at org.testng.internal.Invoker.runTestListeners(Invoker.java:1634)
> at org.testng.internal.Invoker.runTestListeners(Invoker.java:1618)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1101)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:137)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:121)
> at org.testng.TestRunner.runWorkers(TestRunner.java:1030)
> at org.testng.TestRunner.privateRun(TestRunner.java:709)
> at org.testng.TestRunner.run(TestRunner.java:579)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:331)
> at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:326)
> at org.testng.SuiteRunner.privateRun(SuiteRunner.java:288)
> at org.testng.SuiteRunner.run(SuiteRunner.java:193)
> at org.testng.TestNG.createAndRunSuiteRunners(TestNG.java:910)
> at org.testng.TestNG.runSuitesLocally(TestNG.java:879)
> at org.testng.TestNG.run(TestNG.java:787)
> at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:62)
> at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:141)
> at org.apache.maven.surefire.Surefire.run(Surefire.java:177)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:345)
> at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:1009)
> So this should be issue related to TestNG itself (current version 5.12.1), tried 5.13, but the same behavior.
> There is stack trace:
> java.lang.IllegalStateException: can't find more configured methods
> at org.richfaces.tests.metamer.ftest.MatrixConfigurator.configureMethod(MatrixConfigurator.java:150)
> at org.richfaces.tests.metamer.ftest.MatrixConfigurator.beforeInvocation(MatrixConfigurator.java:101)
> at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:62)
> at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:141)
> at org.apache.maven.surefire.Surefire.run(Surefire.java:177)
> at org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:345)
> at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:1009)
> ... Removed 22 stack frames
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 10 months
[JBoss JIRA] (RFPL-3268) Reimplement MatrixConfigurator in Metamer tests
by Jiří Štefek (JIRA)
[ https://issues.jboss.org/browse/RFPL-3268?page=com.atlassian.jira.plugin.... ]
Jiří Štefek edited comment on RFPL-3268 at 3/11/14 6:01 AM:
------------------------------------------------------------
Because the Arquillian extension was not easily implementable as a testing framework independent solution, the MatrixConfigurator was replaced by using of a TestNG's @DataProvider.
The TestNG was updated to latest version.
The features:
* The tested/supported templates are specified in value of {{@Templates}} annotation of first found field in hierarchy of test (from offspring to parent) ( == maybe as before).
* The plain template is activated by default as before.
* To run tests in all templates one can now use either {{*}} or {{all}}.
* To run tests in specific template one can still use everything as before plus I've added some aliases for each template. One can now use e.g. {{-Dtemplates=edt}} for running tests in richExtendedDataTable. The aliases are in [Template|https://github.com/richfaces/richfaces-qa/blob/master/metamer/ap...] class of Metamer Application. The aliases are case insensitive.
* The templates can now be run separately, e.g.: {{-Dtemplates=plain,edt,cp}} (plain, extendedDataTable, collapsiblePanel)
** all tests will run in each specified template, templates separator is {{comma}}
** one can even use {{-Dtemplates=plain,*}} which will run all tests in plain template and in all supported templates (there will be 2 tests for plain template).
* The templates can now be run concurrently, e.g.: {{-Dtemplates=acc>edt,cp}} (1st configuration: EDT inside accordion, 2nd configuration: collapsible panel)
* One can configure a test class/method to run or to not run with specific templates using @Templates (as before)
* The {{@Use}} parameter was separated to {{@UseForAllTests}} and {{@UseWithField}}.
** {{@Inject}} was deleted
** {{@UseForAllTests}} for marking field, that will be used for configuration in all tests
** {{@UseWithField}} for marking method, that will use a field for configuration
** {{@Uses}} can now only accept {{@UseWithField}} annotations
** {{@UseForAllTests}} and {{@UseWithField}} can be used concurrently
** the supported types for fields are Strings, Enums, and everything not primitive (can be fixed) from field
was (Author: jstefek):
Because the Arquillian extension was not easily implementable as a testing framework independent solution, the MatrixConfigurator was replaced by using of a TestNG's @DataProvider.
The TestNG was updated to latest version.
The features:
* The tested/supported templates are specified in value of {{@Templates}} annotation of first found field in hierarchy of test (from offspring to parent) ( == maybe as before).
* The plain template is activated by default as before.
* To run tests in all templates one can now use either {{*}} or {{all}}.
* To run tests in specific template one can still use everything as before plus I've added some aliases for each template. One can now use e.g. {{-Dtemplates=edt}} for running tests in richExtendedDataTable. The aliases are in [Template|https://github.com/richfaces/richfaces-qa/blob/master/metamer/ap...] class of Metamer Application. The aliases are case insensitive.
* The templates can now be run separately, e.g.: {{-Dtemplates=plain,edt,cp}} (plain, extendedDataTable, collapsiblePanel)
** all tests will run in each specified template, templates separator is {{comma}}
** one can even use {{-Dtemplates=plain,*}} which will run all tests in plain template and in all supported templates (there will be 2 tests for plain template).
* The templates can now be run concurrently, e.g.: {{-Dtemplates=acc>edt,cp}} (1st configuration: EDT inside accordion, 2nd configuration: collapsible panel)
* One can configure a test class/method to run or to not run with specific templates using @Templates (as before)
* The {{@Use}} parameter was separated to {{@UseForAllTests}} and {{@UseWithField}}.
** {{@Inject}} was deleted
** {{@UseForAllTests}} for marking field, that will be used for configuration in all tests
** {{@UseWithField}} for marking method to use a field for configuration
** {{@Uses}} can now only accept {{@UseWithField}} annotations
** {{@UseForAllTests}} and {{@UseWithField}} can be used concurrently
** the supported types for fields are Strings, Enums, and everything not primitive (can be fixed) from field
> Reimplement MatrixConfigurator in Metamer tests
> -----------------------------------------------
>
> Key: RFPL-3268
> URL: https://issues.jboss.org/browse/RFPL-3268
> Project: RichFaces Planning
> Issue Type: Task
> Security Level: Public(Everyone can see)
> Components: QE
> Reporter: Pavol Pitonak
> Assignee: Jiří Štefek
> Fix For: 5.0.0.Alpha4
>
>
> Metamer component enabling us to run tests in various templates and to run tests with parameters.
> Problems of current implementation:
> * Implemented as a TestNG interceptor, i.e. it cannot work with JUnit (which is used by all other RichFaces tests)
> * We cannot upgrade to latest version of TestNG (not sure why, probably a bug in TestNG interceptors, it skips tests)
> Requirements for new implementation (almost identical to current implementation):
> * It should work with both TestNG and JUnit (preferably implemented as an Arquillian extension)
> * I can run tests in plain template: mvn clean verify
> * I can run tests in all templates: mvn clean verify -Dtemplates=*
> * I can run tests in all templates using "all" (asterisk cannot be used in Jenkins matrix): mvn clean verify -Dtemplates=all
> * I can run tests in one specific template: mvn clean verify -Dtemplates=richList
> * I can run tests in two or more specific templates (each separately): mvn clean verify -Dtemplates=richList,richDataTable, i.e. each test will run twice - once in richList template and once in richDataTable template
> * I can run tests in two or more specific templates (applied all at once): mvn clean verify -Dtemplates=richList,richDataTable, i.e. each test will run once - with both richList and richDataTable templates activated concurently, comma cannot be used as separator - optional feature?
> * I can configure a test class/method to run with specific templates using @Templates (as is now)
> * I can configure a test class/method not to run with specific templates using @Templates (as is now)
> * I can configure a test to run with multiple parameters using @Inject, @Use (as is now)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 10 months