[jboss-as7-dev] Testsuite - little test grouping improvement for AS 8

[Ondrej Zizka]

Hello all,

I'd like to discuss a small improvement of test grouping.
Please let me know what do you think about it, any pros/cons comments,
and additions, are welcome.

First, the TLDR version:

The change consists of adding a few marking interfaces in
testsuite-shared and a tagging some tests using these in an annotation.
Nothing else, everything will work as before. People not interested in
this won't notice any change.

Tests have some cross-cutting concerns,  which can't be covered by a
package-based tree structure.
For example, a set of tests for Common Criteria, or for Management. This
request comes mainly from QA.
Also, we are bound by technical solutions coming from Maven's and
Arquillian's limitations.

I'd like to leverage JUnit's @Category to tag the tests, to give the
testsuite users possibility to run such groups of tests from different
packages and modules,
just by stating e.g. -Dtest.group=CommonCriteria, instead of identifying
and maintaining lists of packages and classes.
Having an additional way to freely create groups without changing the
package structure can only help, without doing any harm.



Now the whole story. It's also at
https://community.jboss.org/wiki/AS8TestsuiteProposals so feel free to
add your notes there.

Motivation
The tests are categorized into modules and then further to packages.
This forms a tree structure, in which we follow the most important
distinctive feature/purpose of the test.
There are cases the tests need to be categorized in multiple ways. E.g.
security or management tests. It's more practical to keep them separated
by functionality tested.
But that means that when the group is to be run, user needs to provide
(and maintain) a long list of packages to run.

There's a solution: JUnit's @Category.  See AS7-5844. Using that, the
tests may be tagged, and special groups can be created, instead of
creating and maintaining list of packages or even classes.
It's not supposed to replace package-based fine-grained division, but to
complement it.

Another example where current model has some flaws are smoke tests:
Separating the tests physically to different module is artificial and
not much semantic: If any tests starts being considered as smoke test,
what we do? Move it to different module and change package name. Does
that make sense? Wouldn't it be better to just add
`@Category(Smoke.class)` ?

It also collides with the technical solution of separating tests by type
of server setup (multinode, ...). If we decided to include a multinode
test as smoke, what would we do? Move it to smoke module and create
another surefire execution? Doesn't make much sense. Just adding the
`Smoke` category and filtering the tests to be run using categories is
way better.

Last but not least: These categories are totally optional, no one has to
go through the tests and add them.
It's a tool for use cases when this additional categorization is
necessary, like, security-related tests, it's subset Common Criteria
tests, JPA-related tests, "performance smoke tests", etc.

Other tests would remain untouched. The packages will remain the main
manner of categorization.


Who needs it?

Mostly QA - when developing tests harness, when testing under various
conditions (different DB, IPv6, Common Criteria, security policy, ...)


What will change?

The only change needed is 

      * to define the groups (as empty interfaces) 
      * to add the @Category( <GroupInterface>.class ) to test's class
        (and add it to Arquillian's deployment if necessary).

Purpose of each group will be documented in the interface's javadoc.


Tags (groups, if you like)

So far, these are considered:


- CommonCriteria 

- Management

- Security

Stuart's note: It would be nice if the annotation could be applied at a
package level, as security tests are already isolated into individual
packages


- Slow

Needs a definition of what's "slow", possibly in terms of comparison to
some quick test.


- Negative

This one is different kind: QA wants to mark the negative tests to track
where they have some and where they don't.
So this is rather supposed for quality engineering tools.


- Smoke (maybe)

Stuart's note: Unless we actually move the smoke tests into
testsuite/integration/basic then this is pointless.

There are adv and disadv for moving tests into .../basic:
+ It would make adding/removing a test to the smoke group easier.
+ It would allow marking also tests in other modules as smoke.
*  Determining whether a test is smoke test would need to go and look.
But does anyone need to know it when not actually managing the set of
smoke tests, in which case they would just look for the usages of the
Smoke type?
-  To get the current set, you'd need to search for Smoke type
appearances, or to run the smoke tests.  On a related note, I have a
feature request in Surefire to dry-run tests (just list which would
run).


Regards,
Ondra
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/jboss-as7-dev/attachments/20121129/b9ee3f49/attachment.html