[JBoss JIRA] (RFPL-3367) Split suites to decrease job run times and to acquire better feedback
by Pavol Pitonak (JIRA)
[ https://issues.jboss.org/browse/RFPL-3367?page=com.atlassian.jira.plugin.... ]
Pavol Pitonak commented on RFPL-3367:
-------------------------------------
Works perfectly for 6 components as in the job you linked. However, I think that it wouldn't work for all 83 components - the table would be too big.
Can we create about 20 groups somehow (not using 20 almost identical testng.xml descriptors)?
> Split suites to decrease job run times and to acquire better feedback
> ---------------------------------------------------------------------
>
> Key: RFPL-3367
> URL: https://issues.jboss.org/browse/RFPL-3367
> Project: RichFaces Planning
> Issue Type: Task
> Security Level: Public(Everyone can see)
> Components: QE
> Reporter: Jiří Štefek
>
> The suite for tables is now too big and it cannot run on Jenkins under {color:red}900 mins{color}, we need to split it somehow. But, imho, it is not just problem of this suite, we should divide all suites somehow to gain at least a better feedback.
> Any ideas how to divide the suites?
--
This message was sent by Atlassian JIRA
(v6.2.3#6260)
9 years, 11 months
[JBoss JIRA] (RFPL-3268) Reimplement MatrixConfigurator in Metamer tests
by Jiří Štefek (JIRA)
[ https://issues.jboss.org/browse/RFPL-3268?page=com.atlassian.jira.plugin.... ]
Jiří Štefek edited comment on RFPL-3268 at 5/12/14 5:23 AM:
------------------------------------------------------------
Because the Arquillian extension was not easily implementable as a testing framework independent solution, the MatrixConfigurator was replaced by using of a TestNG's @DataProvider.
The TestNG was updated to latest version.
The features:
* The tested/supported templates are specified in value of {{@Templates}} annotation of first found field in hierarchy of test (from offspring to parent) ( == maybe as before).
* The plain template is activated by default as before.
* To run tests in all templates one can now use either {{*}} or {{all}}.
* To run tests in specific template one can still use everything as before plus I've added some aliases for each template. One can now use e.g. {{-Dtemplates=edt}} for running tests in richExtendedDataTable. The aliases are in [Template|https://github.com/richfaces/richfaces-qa/blob/master/metamer/ap...] class of Metamer Application. The aliases are case insensitive.
* The templates can now be run separately, e.g.: {{-Dtemplates=plain,edt,cp}} (plain, extendedDataTable, collapsiblePanel)
** all tests will run in each specified template, templates separator is {{comma}}
** one can even use {{-Dtemplates=plain,*}} which will run all tests in plain template and in all supported templates (there will be 2 tests for plain template).
* The templates can now be run concurrently, e.g.: {{-Dtemplates=acc+edt,cp}} (1st configuration: EDT inside accordion, 2nd configuration: collapsible panel)
** to open the sample with such templates in browser, one should add {{?templates=acc>edt}} (just change {{+}} to {{>}})
* One can configure a test class/method to run or to not run with specific templates using @Templates (as before)
* The {{@Use}} parameter was separated to {{@UseForAllTests}} and {{@UseWithField}}.
** {{@Inject}} was deleted
** {{@UseForAllTests}} for marking field, that will be used for configuration in all tests
** {{@UseWithField}} for marking method, that will use a field for configuration
** {{@Uses}} can now only accept {{@UseWithField}} annotations
** {{@UseForAllTests}} and {{@UseWithField}} can be used concurrently
** the supported types for fields are Strings, Enums, and everything not primitive (can be fixed) from field
was (Author: jstefek):
Because the Arquillian extension was not easily implementable as a testing framework independent solution, the MatrixConfigurator was replaced by using of a TestNG's @DataProvider.
The TestNG was updated to latest version.
The features:
* The tested/supported templates are specified in value of {{@Templates}} annotation of first found field in hierarchy of test (from offspring to parent) ( == maybe as before).
* The plain template is activated by default as before.
* To run tests in all templates one can now use either {{*}} or {{all}}.
* To run tests in specific template one can still use everything as before plus I've added some aliases for each template. One can now use e.g. {{-Dtemplates=edt}} for running tests in richExtendedDataTable. The aliases are in [Template|https://github.com/richfaces/richfaces-qa/blob/master/metamer/ap...] class of Metamer Application. The aliases are case insensitive.
* The templates can now be run separately, e.g.: {{-Dtemplates=plain,edt,cp}} (plain, extendedDataTable, collapsiblePanel)
** all tests will run in each specified template, templates separator is {{comma}}
** one can even use {{-Dtemplates=plain,*}} which will run all tests in plain template and in all supported templates (there will be 2 tests for plain template).
* The templates can now be run concurrently, e.g.: {{-Dtemplates=acc+edt,cp}} (1st configuration: EDT inside accordion, 2nd configuration: collapsible panel)
* One can configure a test class/method to run or to not run with specific templates using @Templates (as before)
* The {{@Use}} parameter was separated to {{@UseForAllTests}} and {{@UseWithField}}.
** {{@Inject}} was deleted
** {{@UseForAllTests}} for marking field, that will be used for configuration in all tests
** {{@UseWithField}} for marking method, that will use a field for configuration
** {{@Uses}} can now only accept {{@UseWithField}} annotations
** {{@UseForAllTests}} and {{@UseWithField}} can be used concurrently
** the supported types for fields are Strings, Enums, and everything not primitive (can be fixed) from field
> Reimplement MatrixConfigurator in Metamer tests
> -----------------------------------------------
>
> Key: RFPL-3268
> URL: https://issues.jboss.org/browse/RFPL-3268
> Project: RichFaces Planning
> Issue Type: Task
> Security Level: Public(Everyone can see)
> Components: QE
> Reporter: Pavol Pitonak
> Assignee: Jiří Štefek
> Fix For: 5.0.0.Alpha4
>
>
> Metamer component enabling us to run tests in various templates and to run tests with parameters.
> Problems of current implementation:
> * Implemented as a TestNG interceptor, i.e. it cannot work with JUnit (which is used by all other RichFaces tests)
> * We cannot upgrade to latest version of TestNG (not sure why, probably a bug in TestNG interceptors, it skips tests)
> Requirements for new implementation (almost identical to current implementation):
> * It should work with both TestNG and JUnit (preferably implemented as an Arquillian extension)
> * I can run tests in plain template: mvn clean verify
> * I can run tests in all templates: mvn clean verify -Dtemplates=*
> * I can run tests in all templates using "all" (asterisk cannot be used in Jenkins matrix): mvn clean verify -Dtemplates=all
> * I can run tests in one specific template: mvn clean verify -Dtemplates=richList
> * I can run tests in two or more specific templates (each separately): mvn clean verify -Dtemplates=richList,richDataTable, i.e. each test will run twice - once in richList template and once in richDataTable template
> * I can run tests in two or more specific templates (applied all at once): mvn clean verify -Dtemplates=richList,richDataTable, i.e. each test will run once - with both richList and richDataTable templates activated concurently, comma cannot be used as separator - optional feature?
> * I can configure a test class/method to run with specific templates using @Templates (as is now)
> * I can configure a test class/method not to run with specific templates using @Templates (as is now)
> * I can configure a test to run with multiple parameters using @Inject, @Use (as is now)
--
This message was sent by Atlassian JIRA
(v6.2.3#6260)
9 years, 12 months
[JBoss JIRA] (RFPL-3268) Reimplement MatrixConfigurator in Metamer tests
by Jiří Štefek (JIRA)
[ https://issues.jboss.org/browse/RFPL-3268?page=com.atlassian.jira.plugin.... ]
Jiří Štefek edited comment on RFPL-3268 at 5/12/14 5:04 AM:
------------------------------------------------------------
Because the Arquillian extension was not easily implementable as a testing framework independent solution, the MatrixConfigurator was replaced by using of a TestNG's @DataProvider.
The TestNG was updated to latest version.
The features:
* The tested/supported templates are specified in value of {{@Templates}} annotation of first found field in hierarchy of test (from offspring to parent) ( == maybe as before).
* The plain template is activated by default as before.
* To run tests in all templates one can now use either {{*}} or {{all}}.
* To run tests in specific template one can still use everything as before plus I've added some aliases for each template. One can now use e.g. {{-Dtemplates=edt}} for running tests in richExtendedDataTable. The aliases are in [Template|https://github.com/richfaces/richfaces-qa/blob/master/metamer/ap...] class of Metamer Application. The aliases are case insensitive.
* The templates can now be run separately, e.g.: {{-Dtemplates=plain,edt,cp}} (plain, extendedDataTable, collapsiblePanel)
** all tests will run in each specified template, templates separator is {{comma}}
** one can even use {{-Dtemplates=plain,*}} which will run all tests in plain template and in all supported templates (there will be 2 tests for plain template).
* The templates can now be run concurrently, e.g.: {{-Dtemplates=acc+edt,cp}} (1st configuration: EDT inside accordion, 2nd configuration: collapsible panel)
* One can configure a test class/method to run or to not run with specific templates using @Templates (as before)
* The {{@Use}} parameter was separated to {{@UseForAllTests}} and {{@UseWithField}}.
** {{@Inject}} was deleted
** {{@UseForAllTests}} for marking field, that will be used for configuration in all tests
** {{@UseWithField}} for marking method, that will use a field for configuration
** {{@Uses}} can now only accept {{@UseWithField}} annotations
** {{@UseForAllTests}} and {{@UseWithField}} can be used concurrently
** the supported types for fields are Strings, Enums, and everything not primitive (can be fixed) from field
was (Author: jstefek):
Because the Arquillian extension was not easily implementable as a testing framework independent solution, the MatrixConfigurator was replaced by using of a TestNG's @DataProvider.
The TestNG was updated to latest version.
The features:
* The tested/supported templates are specified in value of {{@Templates}} annotation of first found field in hierarchy of test (from offspring to parent) ( == maybe as before).
* The plain template is activated by default as before.
* To run tests in all templates one can now use either {{*}} or {{all}}.
* To run tests in specific template one can still use everything as before plus I've added some aliases for each template. One can now use e.g. {{-Dtemplates=edt}} for running tests in richExtendedDataTable. The aliases are in [Template|https://github.com/richfaces/richfaces-qa/blob/master/metamer/ap...] class of Metamer Application. The aliases are case insensitive.
* The templates can now be run separately, e.g.: {{-Dtemplates=plain,edt,cp}} (plain, extendedDataTable, collapsiblePanel)
** all tests will run in each specified template, templates separator is {{comma}}
** one can even use {{-Dtemplates=plain,*}} which will run all tests in plain template and in all supported templates (there will be 2 tests for plain template).
* The templates can now be run concurrently, e.g.: {{-Dtemplates=acc>edt,cp}} (1st configuration: EDT inside accordion, 2nd configuration: collapsible panel)
* One can configure a test class/method to run or to not run with specific templates using @Templates (as before)
* The {{@Use}} parameter was separated to {{@UseForAllTests}} and {{@UseWithField}}.
** {{@Inject}} was deleted
** {{@UseForAllTests}} for marking field, that will be used for configuration in all tests
** {{@UseWithField}} for marking method, that will use a field for configuration
** {{@Uses}} can now only accept {{@UseWithField}} annotations
** {{@UseForAllTests}} and {{@UseWithField}} can be used concurrently
** the supported types for fields are Strings, Enums, and everything not primitive (can be fixed) from field
> Reimplement MatrixConfigurator in Metamer tests
> -----------------------------------------------
>
> Key: RFPL-3268
> URL: https://issues.jboss.org/browse/RFPL-3268
> Project: RichFaces Planning
> Issue Type: Task
> Security Level: Public(Everyone can see)
> Components: QE
> Reporter: Pavol Pitonak
> Assignee: Jiří Štefek
> Fix For: 5.0.0.Alpha4
>
>
> Metamer component enabling us to run tests in various templates and to run tests with parameters.
> Problems of current implementation:
> * Implemented as a TestNG interceptor, i.e. it cannot work with JUnit (which is used by all other RichFaces tests)
> * We cannot upgrade to latest version of TestNG (not sure why, probably a bug in TestNG interceptors, it skips tests)
> Requirements for new implementation (almost identical to current implementation):
> * It should work with both TestNG and JUnit (preferably implemented as an Arquillian extension)
> * I can run tests in plain template: mvn clean verify
> * I can run tests in all templates: mvn clean verify -Dtemplates=*
> * I can run tests in all templates using "all" (asterisk cannot be used in Jenkins matrix): mvn clean verify -Dtemplates=all
> * I can run tests in one specific template: mvn clean verify -Dtemplates=richList
> * I can run tests in two or more specific templates (each separately): mvn clean verify -Dtemplates=richList,richDataTable, i.e. each test will run twice - once in richList template and once in richDataTable template
> * I can run tests in two or more specific templates (applied all at once): mvn clean verify -Dtemplates=richList,richDataTable, i.e. each test will run once - with both richList and richDataTable templates activated concurently, comma cannot be used as separator - optional feature?
> * I can configure a test class/method to run with specific templates using @Templates (as is now)
> * I can configure a test class/method not to run with specific templates using @Templates (as is now)
> * I can configure a test to run with multiple parameters using @Inject, @Use (as is now)
--
This message was sent by Atlassian JIRA
(v6.2.3#6260)
9 years, 12 months
[JBoss JIRA] (RFPL-3367) Split suites to decrease job run times and to acquire better feedback
by Matej Novotny (JIRA)
[ https://issues.jboss.org/browse/RFPL-3367?page=com.atlassian.jira.plugin.... ]
Matej Novotny commented on RFPL-3367:
-------------------------------------
The matrix you proposed looks pretty awesome, that would allow us to browse just the results we are interested in.
Besides we can run just part of the whole matrix build (thanks for explanation btw!) if we need faster feedback for some specific setting.
So for me its +1 for this idea.
> Split suites to decrease job run times and to acquire better feedback
> ---------------------------------------------------------------------
>
> Key: RFPL-3367
> URL: https://issues.jboss.org/browse/RFPL-3367
> Project: RichFaces Planning
> Issue Type: Task
> Security Level: Public(Everyone can see)
> Components: QE
> Reporter: Jiří Štefek
>
> The suite for tables is now too big and it cannot run on Jenkins under {color:red}900 mins{color}, we need to split it somehow. But, imho, it is not just problem of this suite, we should divide all suites somehow to gain at least a better feedback.
> Any ideas how to divide the suites?
--
This message was sent by Atlassian JIRA
(v6.2.3#6260)
9 years, 12 months
[JBoss JIRA] (RFPL-3367) Split suites to decrease job run times and to acquire better feedback
by Jiří Štefek (JIRA)
[ https://issues.jboss.org/browse/RFPL-3367?page=com.atlassian.jira.plugin.... ]
Jiří Štefek edited comment on RFPL-3367 at 5/7/14 5:48 AM:
-----------------------------------------------------------
Do we need the suites at all? Couldn't we create one parametrized matrix job for all components ( = one axis for component, second axis for container)? Or will it be unmaintainable? Will it be a big overload for the slaves?
Something like this job: [richfaces-4.3-release-metamer-ftest--server-compatibility-tables-2|https:...].
was (Author: jstefek):
Do we need the suites at all? Couldn't we create one parametrized matrix job for all components ( = one axis for component, second axis for container)? Or will it be unmaintainable? Will it be a big overload for the slaves?
> Split suites to decrease job run times and to acquire better feedback
> ---------------------------------------------------------------------
>
> Key: RFPL-3367
> URL: https://issues.jboss.org/browse/RFPL-3367
> Project: RichFaces Planning
> Issue Type: Task
> Security Level: Public(Everyone can see)
> Components: QE
> Reporter: Jiří Štefek
>
> The suite for tables is now too big and it cannot run on Jenkins under {color:red}900 mins{color}, we need to split it somehow. But, imho, it is not just problem of this suite, we should divide all suites somehow to gain at least a better feedback.
> Any ideas how to divide the suites?
--
This message was sent by Atlassian JIRA
(v6.2.3#6260)
10 years
[JBoss JIRA] (RFPL-3367) Split suites to decrease job run times and to acquire better feedback
by Jiří Štefek (JIRA)
[ https://issues.jboss.org/browse/RFPL-3367?page=com.atlassian.jira.plugin.... ]
Jiří Štefek updated RFPL-3367:
------------------------------
Description:
The suite for tables is now too big and it cannot run on Jenkins under {color:red}900 mins{color}, we need to split it somehow. But, imho, it is not just problem of this suite, we should divide all suites somehow to gain at least a better feedback.
Any ideas how to divide the suites?
was:
The suite for tables is now too big and it cannot run under {color:red}900 mins{color}, we need to split it somehow. But, imho, it is not just problem of this suite, we should divide all suites somehow to gain at least a better feedback.
Any ideas how to divide the suites?
> Split suites to decrease job run times and to acquire better feedback
> ---------------------------------------------------------------------
>
> Key: RFPL-3367
> URL: https://issues.jboss.org/browse/RFPL-3367
> Project: RichFaces Planning
> Issue Type: Task
> Security Level: Public(Everyone can see)
> Components: QE
> Reporter: Jiří Štefek
>
> The suite for tables is now too big and it cannot run on Jenkins under {color:red}900 mins{color}, we need to split it somehow. But, imho, it is not just problem of this suite, we should divide all suites somehow to gain at least a better feedback.
> Any ideas how to divide the suites?
--
This message was sent by Atlassian JIRA
(v6.2.3#6260)
10 years