[jboss-as7-dev] Test Suite Changes
Kabir Khan
kabir.khan at jboss.com
Fri Nov 4 08:24:31 EDT 2011
I'll be talking to Ondra a bit later today, in the meantime I've created a POC for the testsuite https://github.com/kabir/testsuite-poc. It is a very simple project with hardly any classes/dependencies to just concentrate on the profile selection issues we have been having. If someone has time to look at this in the next few hours and has some feedback that would be great.
The maven modules are
* build/ - the "main codebase"
* testsuite - the testsuite aggregator for a stripped down similar structure to what we have in AS 7
* testsuite/integration - Illustrates a project with several sets of functionality:
- smoke tests
- A fuller set of integration tests
- Some 'special' tests which must be run in a separate surefire execution from the integration tests
* testsuite/spec - Just contains one test
* testsuite/domain - Just contains one test.
In testsuite and testsuite/integration there are some profiles for AS 7 developers and some for QE.
**AS 7 developer profiles**
1) Smoke tests only
Run by: mvn install
The 'activeByDefault' profiles of testsuite/pom.xml and testsuite/integration/pom.xml are triggered. These are:
- testsuite/pom.xml's integration.only profile
This just runs the integration module, and skips the spec and domain modules
- testsuite/integration/pom.xml's integration.smoke.tests profile
This just runs just the smoke tests
These profiles are 'activeByDefault' so they will always be run unless another profile is explicitly enabled.
2) All tests
Run by: mvn install -DallTests OR mvn install -DallTests=true
Once this property is present, the 'activeByDefault' profiles are no longer triggered, instead it triggers:
- testsuite/pom.xml's all.tests profile
This runs the integration,spec and domain modules
- testsuite/integration/pom.xml's integration.all.tests profile
This runs all the tests in two surefire executions. The first with all the smoke and integration tests. The second with the 'special' tests.
**QE profiles**
DISCLAIMER: This might not be exactly what QE want but I'll cover that with Ondra later.
1) testsuite/pom.xml
This contains the following profiles to control which testsuite modules are run
qe.integration - which just runs the integration module tests
qe.spec - which just runs the spec module tests
qe.domain - which just runs the domain module tests
So for example to run both the integration and spec module tests, but not the domain tests, you could do:
mvn install -Pqe.integration,qe.spec
For the integration tests this will by default run just the smoke tests (due the the 'activeByDefault' integration.smoke.tests profile.
2) testsuite/integration/pom.xml
This contains the following profiles to control subsets of tests:
qe.integration.smoke - which just runs the smoke tests
qe.integration.integration - which just runs the integration tests
qe.integration.special - which just runs the 'special' tests
So if -Pqe.integration is specified to run testsuite/integration tests you can control that. For example:
This runs just the testsuite/integration 'integration' tests
mvn install -Pqe.integration,qe.integration.integration
This runs just the testsuite/integration 'special' tests
mvn install -Pqe.integration,qe.integration.special
This runs the testsuite/integration smoke and 'special' tests
mvn install -Pqe.integration,qe.integration.smoke,qe.integration.special
Or you can use the integration.all.tests profile to run all the tests in the integration module (similar to using -DallTests but ONLY running tests from testsuite/integration and not from testsuite/spec or testsuite/domain)
mvn install -Pqe.integration,integration.all.tests
Then QE can create scripts to trigger combinations of the above as they see fit.
On 3 Nov 2011, at 16:20, Kabir Khan wrote:
> ok, will do (tomorrow probably)
> On 3 Nov 2011, at 15:02, Shelly McGowan wrote:
>
>>
>>
>> Kabir,
>>
>> Communicate your ideas to Ondra (ozizka). He can help you.
>>
>> Shelly
>>
>>
>> ----- Original Message -----
>> From: "Kabir Khan" <kkhan at redhat.com>
>> To: "thomas diesler" <thomas.diesler at jboss.com>, smcgowan at redhat.com
>> Cc: jboss-as7-dev at lists.jboss.org
>> Sent: Thursday, November 3, 2011 10:45:36 AM
>> Subject: Re: [jboss-as7-dev] Test Suite Changes
>>
>> I still contend that it should be able to do this differently.
>>
>> Simple mvn install (-DallTest=true) for developers.
>>
>> Then qe should be able to use separate qe profiles/the antrun plugin/whatever with some trickery in the scripts qe uses to do your stuff. I have a few ideas which I can try out if I have some spare cycles and I am sure Paul Gier could help.
>>
>> Sent from my Android phone using TouchDown (www.nitrodesk.com)
>>
>>
>> -----Original Message-----
>> From: Shelly McGowan [smcgowan at redhat.com]
>> Received: Thursday, 03 Nov 2011, 14:35
>> To: Thomas Diesler [thomas.diesler at jboss.com]
>> CC: jboss-as7-dev at lists.jboss.org
>> Subject: Re: [jboss-as7-dev] Test Suite Changes
>>
>>
>>
>>
>>> From August:
>>
>> http://lists.jboss.org/pipermail/jboss-as7-dev/2011-August/003452.html
>>
>> Shelly
>>
>>
>> ----- Original Message -----
>> From: "Shelly McGowan" <smcgowan at redhat.com>
>> To: "Thomas Diesler" <thomas.diesler at jboss.com>
>> Cc: jboss-as7-dev at lists.jboss.org, "Darran Lofthouse" <darran.lofthouse at jboss.com>
>> Sent: Thursday, November 3, 2011 10:24:46 AM
>> Subject: Re: [jboss-as7-dev] Test Suite Changes
>>
>>
>>
>> So, the changes to the test suite were based on requirements. The issues as I see it is that there is no
>> concensus (making everyone happy) which, in turn, has caused additional changes to the initial pull request[s].
>>
>> It was anticipated in the requirements discussions, though, that the integration tests would be used for test purposes
>> other than during a build of AS7.
>>
>> Thomas, have you looked at the requirements document links mentioned below?
>>
>> As far as bugs, fair enough. They're being addressed as they are uncovered. JIRAs should be filed.
>>
>>
>>
>> Shelly
>>
>>
>>
>> ----- Original Message -----
>> From: "Thomas Diesler" <thomas.diesler at jboss.com>
>> To: "Darran Lofthouse" <darran.lofthouse at jboss.com>
>> Cc: jboss-as7-dev at lists.jboss.org
>> Sent: Thursday, November 3, 2011 9:56:28 AM
>> Subject: Re: [jboss-as7-dev] Test Suite Changes
>>
>> Yes, I would also like to see a document that describes how the
>> testsuite is (more importantly should be) organised and how to execute
>> individual test sections. The document should start with a list of
>> requirements.
>>
>> The argument of "integration-tests.sh makes it easier" is not valid
>> IMHO. It is a diversion of an established standard and quite the
>> contrary from "makes it easier" IMHO. In short, start with requirements
>> and show whether and how they are met and where work still needs to be
>> done.
>>
>> Refactoring the test suite into a clean structure, such that standard
>> mvn calls can be used to execute what we need would be the preferred
>> solution IMHO.
>>
>> -thomas
>>
>> On 11/03/2011 12:40 PM, Darran Lofthouse wrote:
>>> It would be nice if we could agree which parameters should actually work
>>> and what they mean - the one that caught me out recently was when
>>> -DskipTests=true was broken.
>>>
>>> Regards,
>>> Darran Lofthouse.
>>>
>>>
>>>
>>> On 11/03/2011 11:34 AM, Kabir Khan wrote:
>>>> Smoke tests are running by default now which is nice. However, a './build.sh clean install -DallTests=true' no longer runs all the tests, while they did last week. This is starting to get very confusing, all I want to do is to process some pull requests and run the tests.
>>>>
>>>> If this really needs to keep changing all the time, a this-is-how-we-run-the-tests-today.txt file in the root folder would help :-)
>>>>
>>>> On 27 Oct 2011, at 17:20, Ondrej Zizka wrote:
>>>>
>>>>> Ok, seems like QA will have to carry the burden of disabling smoke tests whenever they want to run anything else - because Maven's profile activation logic really sucks.
>>>>> Fixing.
>>>>>
>>>>> Ondra
>>>>>
>>>>>
>>>>>
>>>>> On Thu, 2011-10-27 at 11:36 +0100, Kabir Khan wrote:
>>>>>> The smoke tests must be run as part of a standard build. Please fix.
>>>>>>
>>>>>> If QE have other requirements, which are not possible to combine with the above requirement maybe they should look into using alternate poms or the antrun plugin or something.
>>>>>> On 27 Oct 2011, at 09:48, Thomas Diesler wrote:
>>>>>>
>>>>>>> Currently, the smoke tests don't run as part of the build. Is this really what you want?
>>>>>>>
>>>>>>>
>>>>>> https://hudson.qa.jboss.com/hudson/view/JBossOSGi/job/jboss-as7-master/389/
>>>>>>
>>>>>>> -thomas
>>>>>>>
>>>>>>> On 10/27/2011 06:17 AM, Stuart Douglas wrote:
>>>>>>>> I just just merged some changes to the test suite that QE requires for EAP (see below for a link to their requirements). As a result the manner in which the tests are run has changed slightly.
>>>>>>>>
>>>>>>>> Due to limitations in maven it is no longer possible to activate all tests using mvn install -DallTests, instead you should use either:
>>>>>>>>
>>>>>>>> ./build.sh install -DallTests - To do a full build with all tests
>>>>>>>> ./integration-tests.sh -DallTests - To just run all integration tests
>>>>>>>>
>>>>>>>> Running individual tests from an IDE or via mvn test -Dtest=MyTest should work as normal.
>>>>>>>>
>>>>>>>> Ondej Zizka (in CC) has been the person working on these changes, so if you have any questions or comments he is the person to talk to.
>>>>>>>>
>>>>>>>> Stuart
>>>>>>>>
>>>>>>>> *
>>>>>> https://docspace.corp.redhat.com/docs/DOC-74146
>>>>>>
>>>>>>>> *
>>>>>> https://docspace.corp.redhat.com/docs/DOC-69049
>>>>>>
>>>>>>>> *
>>>>>> https://github.com/jbossas/jboss-as/pull/491
>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> jboss-as7-dev mailing list
>>>>>>>>
>>>>>>>>
>>>>>> jboss-as7-dev at lists.jboss.org
>>>>>>
>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>>>
>>>>>>> --
>>>>>>> xxxxxxxxxxxxxxxxxxxxxxxxxxxx
>>>>>>> Thomas Diesler
>>>>>>> JBoss OSGi Lead
>>>>>>> JBoss, a division of Red Hat
>>>>>>> xxxxxxxxxxxxxxxxxxxxxxxxxxxx
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> jboss-as7-dev mailing list
>>>>>>>
>>>>>> jboss-as7-dev at lists.jboss.org
>>>>>>
>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> jboss-as7-dev mailing list
>>>>>>
>>>>>> jboss-as7-dev at lists.jboss.org
>>>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>>
>>>> _______________________________________________
>>>> jboss-as7-dev mailing list
>>>> jboss-as7-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>> _______________________________________________
>>> jboss-as7-dev mailing list
>>> jboss-as7-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>
>> --
>> xxxxxxxxxxxxxxxxxxxxxxxxxxxx
>> Thomas Diesler
>> JBoss OSGi Lead
>> JBoss, a division of Red Hat
>> xxxxxxxxxxxxxxxxxxxxxxxxxxxx
>>
>> _______________________________________________
>> jboss-as7-dev mailing list
>> jboss-as7-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>> _______________________________________________
>> jboss-as7-dev mailing list
>> jboss-as7-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>
More information about the jboss-as7-dev
mailing list