Hi Richard, On 04/02/14 15:23, Richard Achmatowicz wrote: > Hi Bela > > I have been thinking about how we might exclude those tests which are not relevant to EAP testing from the JGroups testsuite. > > The testsuite has a lot of structure which is not easily customised: > - runtest gets passed a lot of parameters which determines the suites to run and the runtime parameters to use > - the report processing depends on runtest output being in a certain format and in certain directories > - not easy to add in lists of new tests which should be excluded for each runtest target > > Because you don't want to add in EAP-related stuff into the JGroups community release, the only clean approach I can see is to do this in the Jenkins job which runs the testsuite: > - checkout and run the entire JGroups 3.2.13 testsuite > - after the testsuite has completed and before the reports are generated, delete all test report directories relating to non-EAP tests > - generate the EAP-only reports and let Jenkins display them An alternative (#1) could be to compile everything, but before running the individual tests (functional, udp, tcp etc), a task would remove all files matching files on the exclusion list from the ./classes directory. The advantage over removing xml files is that we don't need to run all of the tests. > The "delete non-EAP test results" step can be packaged up as an ant file which deletes the specified directories and can be used on all hosts which run the job. > > WDYT? Otherwise, we are going to have to customise the JGroups testsuite in some way, after the branch is checked out and before it is executed. Hmm, not really nice. Also, I don't really like my alternative #1. Perhaps adding an "unsupported" element into the groups annotation attr is not so bad after all, e.g. @Test(groups={Global.FUNCTIONAL,Global.UNSUPPORTED}) public void testFoo() {} I'm thinking we could create an eap-build.xml which uses <import> to import the regular build.xml and overrides the test targets. Or perhaps this could be done inside of the existing build.xml, in a more elegant way. OK, while writing this email, I realized that adding an "unsupported" element to the groups tag is not so bad, so - unless you object - this is the way we're going to do this. Do we agree on "unsupported", or would you like it to be named differently ?

Hi Richard, On 04/02/14 15:23, Richard Achmatowicz wrote: > Hi Bela > > I have been thinking about how we might exclude those tests which are > not relevant to EAP testing from the JGroups testsuite. > > The testsuite has a lot of structure which is not easily customised: > - runtest gets passed a lot of parameters which determines the suites to > run and the runtime parameters to use > - the report processing depends on runtest output being in a certain > format and in certain directories > - not easy to add in lists of new tests which should be excluded for > each runtest target > > Because you don't want to add in EAP-related stuff into the JGroups > community release, the only clean approach I can see is to do this in > the Jenkins job which runs the testsuite: > - checkout and run the entire JGroups 3.2.13 testsuite > - after the testsuite has completed and before the reports are > generated, delete all test report directories relating to non-EAP tests > - generate the EAP-only reports and let Jenkins display them An alternative (#1) could be to compile everything, but before running the individual tests (functional, udp, tcp etc), a task would remove all files matching files on the exclusion list from the ./classes directory. The advantage over removing xml files is that we don't need to run all of the tests. > The "delete non-EAP test results" step can be packaged up as an ant file > which deletes the specified directories and can be used on all hosts > which run the job. > > WDYT? Otherwise, we are going to have to customise the JGroups > testsuite in some way, after the branch is checked out and before it is > executed. Hmm, not really nice. Also, I don't really like my alternative #1. Perhaps adding an "unsupported" element into the groups annotation attr is not so bad after all, e.g. @Test(groups={Global.FUNCTIONAL,Global.UNSUPPORTED}) public void testFoo() {} I'm thinking we could create an eap-build.xml which uses <import> to import the regular build.xml and overrides the test targets. Or perhaps this could be done inside of the existing build.xml, in a more elegant way. OK, while writing this email, I realized that adding an "unsupported" element to the groups tag is not so bad, so - unless you object - this is the way we're going to do this. Do we agree on "unsupported", or would you like it to be named differently ?

Allow exclusion of specific test cases from the testsuite execution ------------------------------------------------------------------- Key: JGRP-1790 URL: https://issues.jboss.org/browse/JGRP-1790 Project: JGroups Issue Type: Feature Request Affects Versions: 3.2.13 Reporter: Richard Achmatowicz Assignee: Bela Ban Fix For: 3.2.13, 3.5 There is a need to run the JGroups testsuite, while at the same time excluding a subset of tests from test execution and report generation. This is used, for example, in executing JGroups tests which are only relevant to EAP. .