Hello all,
I'd like to discuss a small improvement of test grouping.
Please let me know what do you think about it, any pros/cons comments, and additions, are welcome.
First, the TLDR version:
The change consists of adding a few marking interfaces in testsuite-shared and a tagging some tests using these in an annotation.
Nothing else, everything will work as before. People not interested in this won't notice any change.
Tests have some cross-cutting concerns, which can't be covered by a package-based tree structure.
For example, a set of tests for Common Criteria, or for Management. This request comes mainly from QA.
Also, we are bound by technical solutions coming from Maven's and Arquillian's limitations.
I'd like to leverage JUnit's @Category to tag the tests, to give the testsuite users possibility to run such groups of tests from different packages and modules,
just by stating e.g. -Dtest.group=CommonCriteria, instead of identifying and maintaining lists of packages and classes.
Having an additional way to freely create groups without changing the package structure can only help, without doing any harm.
Now the whole story. It's also at https://community.jboss.org/wiki/AS8TestsuiteProposals so feel free to add your notes there.
Motivation
The tests are categorized into modules and then further to packages.
This forms a tree structure, in which we follow the most important distinctive feature/purpose of the test.
There are cases the tests need to be categorized in multiple ways. E.g. security or management tests. It's more practical to keep them separated by functionality tested.
But that means that when the group is to be run, user needs to provide (and maintain) a long list of packages to run.
There's a solution: JUnit's @Category. See AS7-5844. Using that, the tests may be tagged, and special groups can be created, instead of creating and maintaining list of packages or even classes.
It's not supposed to replace package-based fine-grained division, but to complement it.
Another example where current model has some flaws are smoke tests: Separating the tests physically to different module is artificial and not much semantic: If any tests starts being considered as smoke test, what we do? Move it to different module and change package name. Does that make sense? Wouldn't it be better to just add `@Category(Smoke.class)` ?
It also collides with the technical solution of separating tests by type of server setup (multinode, ...). If we decided to include a multinode test as smoke, what would we do? Move it to smoke module and create another surefire execution? Doesn't make much sense. Just adding the `Smoke` category and filtering the tests to be run using categories is way better.
Last but not least: These categories are totally optional, no one has to go through the tests and add them.
It's a tool for use cases when this additional categorization is necessary, like, security-related tests, it's subset Common Criteria tests, JPA-related tests, "performance smoke tests", etc.
Other tests would remain untouched. The packages will remain the main manner of categorization.
Who needs it?
Mostly QA - when developing tests harness, when testing under various conditions (different DB, IPv6, Common Criteria, security policy, ...)
What will change?
The only change needed is
- to define the groups (as empty interfaces)
- to add the @Category( <GroupInterface>.class ) to test's class (and add it to Arquillian's deployment if necessary).
Purpose of each group will be documented in the interface's javadoc.
Tags (groups, if you like)
So far, these are considered:
- CommonCriteria
- Management
- Security
Stuart's note: It would be nice if the annotation could be applied at a package level, as security tests are already isolated into individual packages
- Slow
Needs a definition of what's "slow", possibly in terms of comparison to some quick test.
- Negative
This one is different kind: QA wants to mark the negative tests to track where they have some and where they don't.
So this is rather supposed for quality engineering tools.
- Smoke (maybe)
Stuart's note: Unless we actually move the smoke tests into testsuite/integration/basic then this is pointless.
There are adv and disadv for moving tests into .../basic:
+ It would make adding/removing a test to the smoke group easier.
+ It would allow marking also tests in other modules as smoke.
* Determining whether a test is smoke test would need to go and look. But does anyone need to know it when not actually managing the set of smoke tests, in which case they would just look for the usages of the Smoke type?
- To get the current set, you'd need to search for Smoke type appearances, or to run the smoke tests. On a related note, I have a feature request in Surefire to dry-run tests (just list which would run).
Regards,
Ondra