[jboss-as7-dev] Testsuite - little test grouping improvement for AS 8

[Jason Greene]

I agree with everything you said Brian. 

On Nov 29, 2012, at 8:22 AM, Brian Stansberry <brian.stansberry at redhat.com> wrote:

> I'm OK with this, actually think it's a good idea, but it's not a case 
> of "can only help, without doing any harm." The harm is diverting 
> resources into categorizing tests. The help is that it makes it much 
> easier to perform useful categorizations of tests; the harm comes when 
> that apparent ease leads to unproductive categorization and we die a 
> death by a thousand cuts.
> 
> Regarding the details:
> 
> Slow -- is there a way to say "exclude the @Slow tests"? If not, this 
> isn't useful. Even then I have my doubts about usefulness. Why would 
> someone run all tests except the @Slow ones, when that will still take a 
> pretty long time and lightning can run them all?
> 
> Negative -- this triggers my "unproductive categorization" feelers. I 
> don't see myself spending energy trying to apply this to tests I write, 
> I don't see myself monitoring application of this annotation in patches 
> I review, and I'd be shocked if I'm unique in those regards. If it's not 
> applied to a very high % of the relevant tests, what use is it?
> 
> Smoke -- I think this could be useful because maintaining tests 
> separately from other related tests in basic seems unproductive. We 
> likely have some tests in smoke that shouldn't be and some in basic that 
> could be smoke tests, and this situation persists just because moving 
> things around is too much work.
> 
> But it's important to keep in mind the purpose of the smoke tests -- to 
> be run on every build, without burdening the developer with the need to 
> wait for the full -DallTests run. However, any patch is getting 
> automatically tested by lightning with -DallTests at least once before 
> getting merged. That reduces the value of smoke tests, to the point 
> where I'm starting to doubt if they have any real value at all. So IMHO 
> we shouldn't let the ease of use of a @Smoke annotation lead to a 
> proliferation of smoke tests that just slow down developer builds.
> 
> 
> On 11/29/12 12:01 AM, Ondrej Zizka wrote:
>> Hello all,
>> 
>> I'd like to discuss a small improvement of test grouping.
>> Please let me know what do you think about it, any pros/cons comments,
>> and additions, are welcome.
>> 
>> First, the TLDR version:
>> 
>> *The change consists of adding a few marking interfaces in
>> testsuite-shared and a tagging some tests using these in an annotation.*
>> *Nothing else, everything will work as before. People not interested in
>> this won't notice any change.*
>> 
>> Tests have some cross-cutting concerns,  which can't be covered by a
>> package-based tree structure.
>> For example, a set of tests for Common Criteria, or for Management. This
>> request comes mainly from QA.
>> Also, we are bound by technical solutions coming from Maven's and
>> Arquillian's limitations.
>> 
>> I'd like to leverage JUnit's @Category to tag the tests, to give the
>> testsuite users possibility to run such groups of tests from different
>> packages and modules,
>> just by stating e.g. -Dtest.group=CommonCriteria, instead of identifying
>> and maintaining lists of packages and classes.
>> Having an additional way to freely create groups without changing the
>> package structure can only help, without doing any harm.
>> 
>> 
>> 
>> Now the whole story. It's also at
>> https://community.jboss.org/wiki/AS8TestsuiteProposals so feel free to
>> add your notes there.
>> 
>> *Motivation*
>> The tests are categorized into modules and then further to packages.
>> This forms a tree structure, in which we follow the most important
>> distinctive feature/purpose of the test.
>> There are cases the tests need to be categorized in multiple ways. E.g.
>> security or management tests. It's more practical to keep them separated
>> by functionality tested.
>> But that means that when the group is to be run, user needs to provide
>> (and maintain) a long list of packages to run.
>> 
>> There's a solution: JUnit's @Category.  See AS7-5844
>> <https://issues.jboss.org/browse/AS7-5844>. Using that, the tests may be
>> tagged, and special groups can be created, instead of creating and
>> maintaining list of packages or even classes.
>> It's not supposed to replace package-based fine-grained division, but to
>> complement it.
>> 
>> Another example where current model has some flaws are smoke tests:
>> Separating the tests physically to different module is artificial and
>> not much semantic: If any tests starts being considered as smoke test,
>> what we do? Move it to different module and change package name. Does
>> that make sense? Wouldn't it be better to just add
>> `@Category(Smoke.class)` ?
>> 
>> It also collides with the technical solution of separating tests by type
>> of server setup (multinode, ...). If we decided to include a multinode
>> test as smoke, what would we do? Move it to smoke module and create
>> another surefire execution? Doesn't make much sense. Just adding the
>> `Smoke` category and filtering the tests to be run using categories is
>> way better.
>> 
>> Last but not least: These categories are /totally optional/, no one has
>> to go through the tests and add them.
>> It's a tool for use cases when this additional categorization is
>> necessary, like, security-related tests, it's subset Common Criteria
>> tests, JPA-related tests, "performance smoke tests", etc.
>> 
>> Other tests would remain untouched. The packages will remain the main
>> manner of categorization.
>> 
>> 
>>      *Who needs it?*
>> 
>> Mostly QA - when developing tests harness, when testing under various
>> conditions (different DB, IPv6, Common Criteria, security policy, ...)
>> 
>> 
>>      *What will change?*
>> 
>> The only change needed is
>> 
>>  * to define the groups (as empty interfaces)
>>  * to add the @Category( <GroupInterface>.class ) to test's class (and
>>    add it to Arquillian's deployment if necessary).
>> 
>> Purpose of each group will be documented in the interface's javadoc.
>> 
>> 
>>      *Tags (groups, if you like)*
>> 
>> So far, these are considered:
>> 
>> 
>>          *- CommonCriteria *
>>          *- Management*
>>          *- Security*
>> 
>> Stuart's note: It would be nice if the annotation could be applied at a
>> package level, as security tests are already isolated into individual
>> packages
>> 
>> 
>>          *- Slow*
>> 
>> Needs a definition of what's "slow", possibly in terms of comparison to
>> some quick test.
>> 
>> 
>>          *- Negative*
>> 
>> This one is different kind: QA wants to mark the negative tests to track
>> where they have some and where they don't.
>> So this is rather supposed for quality engineering tools.
>> 
>> 
>>          *- Smoke (maybe)*
>> 
>> Stuart's note: Unless we actually move the smoke tests into
>> testsuite/integration/basic then this is pointless.
>> 
>> There are adv and disadv for moving tests into .../basic:
>> + It would make adding/removing a test to the smoke group easier.
>> + It would allow marking also tests in other modules as smoke.
>> *  Determining whether a test is smoke test would need to go and look.
>> But does anyone need to know it when not actually managing the set of
>> smoke tests, in which case they would just look for the usages of the
>> Smoke type?
>> -  To get the current set, you'd need to search for Smoke type
>> appearances, or to run the smoke tests.  On a related note, I have a
>> feature request in Surefire to dry-run tests (just list which would run).
>> 
>> 
>> Regards,
>> Ondra
>> 
>> 
>> _______________________________________________
>> jboss-as7-dev mailing list
>> jboss-as7-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>> 
> 
> 
> -- 
> Brian Stansberry
> Principal Software Engineer
> JBoss by Red Hat
> _______________________________________________
> jboss-as7-dev mailing list
> jboss-as7-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev