[jboss-as7-dev] Testsuite - little test grouping improvement for AS 8

[Brian Stansberry]

I'm OK with this, actually think it's a good idea, but it's not a case 
of "can only help, without doing any harm." The harm is diverting 
resources into categorizing tests. The help is that it makes it much 
easier to perform useful categorizations of tests; the harm comes when 
that apparent ease leads to unproductive categorization and we die a 
death by a thousand cuts.

Regarding the details:

Slow -- is there a way to say "exclude the @Slow tests"? If not, this 
isn't useful. Even then I have my doubts about usefulness. Why would 
someone run all tests except the @Slow ones, when that will still take a 
pretty long time and lightning can run them all?

Negative -- this triggers my "unproductive categorization" feelers. I 
don't see myself spending energy trying to apply this to tests I write, 
I don't see myself monitoring application of this annotation in patches 
I review, and I'd be shocked if I'm unique in those regards. If it's not 
applied to a very high % of the relevant tests, what use is it?

Smoke -- I think this could be useful because maintaining tests 
separately from other related tests in basic seems unproductive. We 
likely have some tests in smoke that shouldn't be and some in basic that 
could be smoke tests, and this situation persists just because moving 
things around is too much work.

But it's important to keep in mind the purpose of the smoke tests -- to 
be run on every build, without burdening the developer with the need to 
wait for the full -DallTests run. However, any patch is getting 
automatically tested by lightning with -DallTests at least once before 
getting merged. That reduces the value of smoke tests, to the point 
where I'm starting to doubt if they have any real value at all. So IMHO 
we shouldn't let the ease of use of a @Smoke annotation lead to a 
proliferation of smoke tests that just slow down developer builds.


On 11/29/12 12:01 AM, Ondrej Zizka wrote:
> Hello all,
>
> I'd like to discuss a small improvement of test grouping.
> Please let me know what do you think about it, any pros/cons comments,
> and additions, are welcome.
>
> First, the TLDR version:
>
> *The change consists of adding a few marking interfaces in
> testsuite-shared and a tagging some tests using these in an annotation.*
> *Nothing else, everything will work as before. People not interested in
> this won't notice any change.*
>
> Tests have some cross-cutting concerns,  which can't be covered by a
> package-based tree structure.
> For example, a set of tests for Common Criteria, or for Management. This
> request comes mainly from QA.
> Also, we are bound by technical solutions coming from Maven's and
> Arquillian's limitations.
>
> I'd like to leverage JUnit's @Category to tag the tests, to give the
> testsuite users possibility to run such groups of tests from different
> packages and modules,
> just by stating e.g. -Dtest.group=CommonCriteria, instead of identifying
> and maintaining lists of packages and classes.
> Having an additional way to freely create groups without changing the
> package structure can only help, without doing any harm.
>
>
>
> Now the whole story. It's also at
> https://community.jboss.org/wiki/AS8TestsuiteProposals so feel free to
> add your notes there.
>
> *Motivation*
> The tests are categorized into modules and then further to packages.
> This forms a tree structure, in which we follow the most important
> distinctive feature/purpose of the test.
> There are cases the tests need to be categorized in multiple ways. E.g.
> security or management tests. It's more practical to keep them separated
> by functionality tested.
> But that means that when the group is to be run, user needs to provide
> (and maintain) a long list of packages to run.
>
> There's a solution: JUnit's @Category.  See AS7-5844
> <https://issues.jboss.org/browse/AS7-5844>. Using that, the tests may be
> tagged, and special groups can be created, instead of creating and
> maintaining list of packages or even classes.
> It's not supposed to replace package-based fine-grained division, but to
> complement it.
>
> Another example where current model has some flaws are smoke tests:
> Separating the tests physically to different module is artificial and
> not much semantic: If any tests starts being considered as smoke test,
> what we do? Move it to different module and change package name. Does
> that make sense? Wouldn't it be better to just add
> `@Category(Smoke.class)` ?
>
> It also collides with the technical solution of separating tests by type
> of server setup (multinode, ...). If we decided to include a multinode
> test as smoke, what would we do? Move it to smoke module and create
> another surefire execution? Doesn't make much sense. Just adding the
> `Smoke` category and filtering the tests to be run using categories is
> way better.
>
> Last but not least: These categories are /totally optional/, no one has
> to go through the tests and add them.
> It's a tool for use cases when this additional categorization is
> necessary, like, security-related tests, it's subset Common Criteria
> tests, JPA-related tests, "performance smoke tests", etc.
>
> Other tests would remain untouched. The packages will remain the main
> manner of categorization.
>
>
>       *Who needs it?*
>
> Mostly QA - when developing tests harness, when testing under various
> conditions (different DB, IPv6, Common Criteria, security policy, ...)
>
>
>       *What will change?*
>
> The only change needed is
>
>   * to define the groups (as empty interfaces)
>   * to add the @Category( <GroupInterface>.class ) to test's class (and
>     add it to Arquillian's deployment if necessary).
>
> Purpose of each group will be documented in the interface's javadoc.
>
>
>       *Tags (groups, if you like)*
>
> So far, these are considered:
>
>
>           *- CommonCriteria *
>           *- Management*
>           *- Security*
>
> Stuart's note: It would be nice if the annotation could be applied at a
> package level, as security tests are already isolated into individual
> packages
>
>
>           *- Slow*
>
> Needs a definition of what's "slow", possibly in terms of comparison to
> some quick test.
>
>
>           *- Negative*
>
> This one is different kind: QA wants to mark the negative tests to track
> where they have some and where they don't.
> So this is rather supposed for quality engineering tools.
>
>
>           *- Smoke (maybe)*
>
> Stuart's note: Unless we actually move the smoke tests into
> testsuite/integration/basic then this is pointless.
>
> There are adv and disadv for moving tests into .../basic:
> + It would make adding/removing a test to the smoke group easier.
> + It would allow marking also tests in other modules as smoke.
> *  Determining whether a test is smoke test would need to go and look.
> But does anyone need to know it when not actually managing the set of
> smoke tests, in which case they would just look for the usages of the
> Smoke type?
> -  To get the current set, you'd need to search for Smoke type
> appearances, or to run the smoke tests.  On a related note, I have a
> feature request in Surefire to dry-run tests (just list which would run).
>
>
> Regards,
> Ondra
>
>
> _______________________________________________
> jboss-as7-dev mailing list
> jboss-as7-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>


-- 
Brian Stansberry
Principal Software Engineer
JBoss by Red Hat