[jboss-jira] [JBoss JIRA] (JGRP-1790) Allow exclusion of specific test cases from the testsuite execution

[Richard Achmatowicz (JIRA)]

    [ https://issues.jboss.org/browse/JGRP-1790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12941264#comment-12941264 ] 

Richard Achmatowicz edited comment on JGRP-1790 at 2/4/14 11:35 AM:
--------------------------------------------------------------------

On 02/04/2014 10:50 AM, Bela Ban wrote:
> Hi Richard,
>
> On 04/02/14 15:23, Richard Achmatowicz wrote:
>> Hi Bela
>>
>> I have been thinking about how we might exclude those tests which are not relevant to EAP testing from the JGroups testsuite.
>>
>> The testsuite has a lot of structure which is not easily customised:
>> - runtest gets passed a lot of parameters which determines the suites to run and the runtime parameters to use
>> - the report processing depends on runtest output being in a certain format and in certain directories
>> - not easy to add in lists of new tests which should be excluded for each runtest target
>>
>> Because you don't want to add in EAP-related stuff into the JGroups community release, the only clean approach I can see is to do this in the Jenkins job which runs the testsuite:
>> - checkout and run the entire JGroups 3.2.13 testsuite
>> - after the testsuite has completed and before the reports are generated, delete all test report directories relating to non-EAP tests
>> - generate the EAP-only reports and let Jenkins display them 
>
>
> An alternative (#1) could be to compile everything, but before running the individual tests (functional, udp, tcp etc), a task would remove all files matching files on the exclusion list from the ./classes directory.
>
> The advantage over removing xml files is that we don't need to run all of the tests.
>
>> The "delete non-EAP test results" step can be packaged up as an ant file which deletes the specified directories and can be used on all hosts which run the job.
>>
>> WDYT?  Otherwise, we are going to have to customise the JGroups testsuite in some way, after the branch is checked out and before it is executed. 
>
>
> Hmm, not really nice. Also, I don't really like my alternative #1. Perhaps adding an "unsupported" element into the groups annotation attr is not so bad after all, e.g.
>
> @Test(groups={Global.FUNCTIONAL,Global.UNSUPPORTED}) public void testFoo() {}
>
> I'm thinking we could create an eap-build.xml which uses <import> to import the regular build.xml and overrides the test targets. Or perhaps this could be done inside of the existing build.xml, in a more elegant way.
>
> OK, while writing this email, I realized that adding an "unsupported" element to the groups tag is not so bad, so - unless you object - this is the way we're going to do this.
>
> Do we agree on "unsupported", or would you like it to be named differently ?
>


                
      was (Author: rachmato):
    
On 02/04/2014 10:50 AM, Bela Ban wrote:
> Hi Richard,
>
>
> On 04/02/14 15:23, Richard Achmatowicz wrote:
>> Hi Bela
>>
>> I have been thinking about how we might exclude those tests which are
>> not relevant to EAP testing from the JGroups testsuite.
>>
>> The testsuite has a lot of structure which is not easily customised:
>> - runtest gets passed a lot of parameters which determines the suites to
>> run and the runtime parameters to use
>> - the report processing depends on runtest output being in a certain
>> format and in certain directories
>> - not easy to add in lists of new tests which should be excluded for
>> each runtest target
>>
>> Because you don't want to add in EAP-related stuff into the JGroups
>> community release, the only clean approach I can see is to do this in
>> the Jenkins job which runs the testsuite:
>> - checkout and run the entire JGroups 3.2.13 testsuite
>> - after the testsuite has completed and before the reports are
>> generated, delete all test report directories relating to non-EAP tests
>> - generate the EAP-only reports and let Jenkins display them 
>
>
> An alternative (#1) could be to compile everything, but before running the individual tests (functional, udp, tcp etc), a task would remove all files matching files on the exclusion list from the ./classes directory.
>
> The advantage over removing xml files is that we don't need to run all of the tests.
>
>> The "delete non-EAP test results" step can be packaged up as an ant file
>> which deletes the specified directories and can be used on all hosts
>> which run the job.
>>
>> WDYT?  Otherwise, we are going to have to customise the JGroups
>> testsuite in some way, after the branch is checked out and before it is
>> executed. 
>
>
> Hmm, not really nice. Also, I don't really like my alternative #1. Perhaps adding an "unsupported" element into the groups annotation attr is not so bad after all, e.g.
>
> @Test(groups={Global.FUNCTIONAL,Global.UNSUPPORTED}) public void testFoo() {}
>
> I'm thinking we could create an eap-build.xml which uses <import> to import the regular build.xml and overrides the test targets. Or perhaps this could be done inside of the existing build.xml, in a more elegant way.
>
> OK, while writing this email, I realized that adding an "unsupported" element to the groups tag is not so bad, so - unless you object - this is the way we're going to do this.
>
> Do we agree on "unsupported", or would you like it to be named differently ?
>


                  
> Allow exclusion of specific test cases from the testsuite execution
> -------------------------------------------------------------------
>
>                 Key: JGRP-1790
>                 URL: https://issues.jboss.org/browse/JGRP-1790
>             Project: JGroups
>          Issue Type: Feature Request
>    Affects Versions: 3.2.13
>            Reporter: Richard Achmatowicz
>            Assignee: Bela Ban
>             Fix For: 3.2.13, 3.5
>
>
> There is a need to run the JGroups testsuite, while at the same time excluding a subset of tests from test execution and report generation.
> This is used, for example, in executing JGroups tests which are only relevant to EAP.
> .

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira