[jboss-jira] [JBoss JIRA] (JGRP-1790) Allow exclusion of specific test cases from the testsuite execution

[Richard Achmatowicz (JIRA)]

    [ https://issues.jboss.org/browse/JGRP-1790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12941264#comment-12941264 ] 

Richard Achmatowicz edited comment on JGRP-1790 at 2/4/14 11:34 AM:
--------------------------------------------------------------------


On 02/04/2014 10:50 AM, Bela Ban wrote:
> Hi Richard,
>
>
> On 04/02/14 15:23, Richard Achmatowicz wrote:
>> Hi Bela
>>
>> I have been thinking about how we might exclude those tests which are
>> not relevant to EAP testing from the JGroups testsuite.
>>
>> The testsuite has a lot of structure which is not easily customised:
>> - runtest gets passed a lot of parameters which determines the suites to
>> run and the runtime parameters to use
>> - the report processing depends on runtest output being in a certain
>> format and in certain directories
>> - not easy to add in lists of new tests which should be excluded for
>> each runtest target
>>
>> Because you don't want to add in EAP-related stuff into the JGroups
>> community release, the only clean approach I can see is to do this in
>> the Jenkins job which runs the testsuite:
>> - checkout and run the entire JGroups 3.2.13 testsuite
>> - after the testsuite has completed and before the reports are
>> generated, delete all test report directories relating to non-EAP tests
>> - generate the EAP-only reports and let Jenkins display them 
>
>
> An alternative (#1) could be to compile everything, but before running the individual tests (functional, udp, tcp etc), a task would remove all files matching files on the exclusion list from the ./classes directory.
>
> The advantage over removing xml files is that we don't need to run all of the tests.
>
>> The "delete non-EAP test results" step can be packaged up as an ant file
>> which deletes the specified directories and can be used on all hosts
>> which run the job.
>>
>> WDYT?  Otherwise, we are going to have to customise the JGroups
>> testsuite in some way, after the branch is checked out and before it is
>> executed. 
>
>
> Hmm, not really nice. Also, I don't really like my alternative #1. Perhaps adding an "unsupported" element into the groups annotation attr is not so bad after all, e.g.
>
> @Test(groups={Global.FUNCTIONAL,Global.UNSUPPORTED}) public void testFoo() {}
>
> I'm thinking we could create an eap-build.xml which uses <import> to import the regular build.xml and overrides the test targets. Or perhaps this could be done inside of the existing build.xml, in a more elegant way.
>
> OK, while writing this email, I realized that adding an "unsupported" element to the groups tag is not so bad, so - unless you object - this is the way we're going to do this.
>
> Do we agree on "unsupported", or would you like it to be named differently ?
>


                
      was (Author: rachmato):
    From email discussion:
<quote>
Hi Bela

I have been thinking about how we might exclude those tests which are not relevant to EAP testing from the JGroups testsuite.

The testsuite has a lot of structure which is not easily customised:
- runtest gets passed a lot of parameters which determines the suites to run and the runtime parameters to use
- the report processing depends on runtest output being in a certain format and in certain directories
- not easy to add in lists of new tests which should be excluded for each runtest target

Because you don't want to add in EAP-related stuff into the JGroups community release, the only clean approach I can see is to do this in the Jenkins job which runs the testsuite:
- checkout and run the entire JGroups 3.2.13 testsuite
- after the testsuite has completed and before the reports are generated, delete all test report directories relating to non-EAP tests
- generate the EAP-only reports and let Jenkins display them

The "delete non-EAP test results" step can be packaged up as an ant file which deletes the specified directories and can be used on all hosts which run the job.

WDYT?  Otherwise, we are going to have to customise the JGroups testsuite in some way, after the branch is checked out and before it is executed.
<quote>
                  
> Allow exclusion of specific test cases from the testsuite execution
> -------------------------------------------------------------------
>
>                 Key: JGRP-1790
>                 URL: https://issues.jboss.org/browse/JGRP-1790
>             Project: JGroups
>          Issue Type: Feature Request
>    Affects Versions: 3.2.13
>            Reporter: Richard Achmatowicz
>            Assignee: Bela Ban
>             Fix For: 3.2.13
>
>
> There is a need to run the JGroups testsuite, while at the same time excluding a subset of tests from test execution and report generation.
> This is used, for example, in executing JGroups tests which are only relevant to EAP.
> .

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira