Ok, so it seems that there are three tags for which are accepted:
So I have limited the PR to just these, and applied the CommonCriteria
to the list I got from Peter Skopek.
The other two are to be applied later by Peter and Dominik Pospisil.
On Thu, 2012-11-29 at 08:22 -0600, Brian Stansberry wrote:
I'm OK with this, actually think it's a good idea, but
it's not a case
of "can only help, without doing any harm." The harm is diverting
resources into categorizing tests. The help is that it makes it much
easier to perform useful categorizations of tests; the harm comes when
that apparent ease leads to unproductive categorization and we die a
death by a thousand cuts.
Regarding the details:
Slow -- is there a way to say "exclude the @Slow tests"?
Even then I have my doubts about usefulness. Why would
someone run all tests except the @Slow ones, when that will still take a
pretty long time and lightning can run them all?
I'd consider it something between smoke and full run. Broadened
developer run. It was common suggestion for category. If no one
specifically asks for it now, well, let's skip this one.
Negative -- this triggers my "unproductive categorization"
don't see myself spending energy trying to apply this to tests I write,
I don't see myself monitoring application of this annotation in patches
I review, and I'd be shocked if I'm unique in those regards. If it's not
applied to a very high % of the relevant tests, what use is it?
It would be a tag for QE tools - for knowing what area is covered by
Smoke -- I think this could be useful because maintaining tests
separately from other related tests in basic seems unproductive. We
likely have some tests in smoke that shouldn't be and some in basic that
could be smoke tests, and this situation persists just because moving
things around is too much work.
But it's important to keep in mind the purpose of the smoke tests
be run on every build, without burdening the developer with the need to
wait for the full -DallTests run. However, any patch is getting
automatically tested by lightning with -DallTests at least once before
getting merged. That reduces the value of smoke tests, to the point
where I'm starting to doubt if they have any real value at all. So IMHO
we shouldn't let the ease of use of a @Smoke annotation lead to a
proliferation of smoke tests that just slow down developer builds.
On 11/29/12 12:01 AM, Ondrej Zizka wrote:
> Hello all,
> I'd like to discuss a small improvement of test grouping.
> Please let me know what do you think about it, any pros/cons comments,
> and additions, are welcome.
> First, the TLDR version:
> *The change consists of adding a few marking interfaces in
> testsuite-shared and a tagging some tests using these in an annotation.*
> *Nothing else, everything will work as before. People not interested in
> this won't notice any change.*
> Tests have some cross-cutting concerns, which can't be covered by a
> package-based tree structure.
> For example, a set of tests for Common Criteria, or for Management. This
> request comes mainly from QA.
> Also, we are bound by technical solutions coming from Maven's and
> Arquillian's limitations.
> I'd like to leverage JUnit's @Category to tag the tests, to give the
> testsuite users possibility to run such groups of tests from different
> packages and modules,
> just by stating e.g. -Dtest.group=CommonCriteria, instead of identifying
> and maintaining lists of packages and classes.
> Having an additional way to freely create groups without changing the
> package structure can only help, without doing any harm.
> Now the whole story. It's also at
so feel free to
> add your notes there.
> The tests are categorized into modules and then further to packages.
> This forms a tree structure, in which we follow the most important
> distinctive feature/purpose of the test.
> There are cases the tests need to be categorized in multiple ways. E.g.
> security or management tests. It's more practical to keep them separated
> by functionality tested.
> But that means that when the group is to be run, user needs to provide
> (and maintain) a long list of packages to run.
> There's a solution: JUnit's @Category. See AS7-5844
;. Using that, the tests may be
> tagged, and special groups can be created, instead of creating and
> maintaining list of packages or even classes.
> It's not supposed to replace package-based fine-grained division, but to
> complement it.
> Another example where current model has some flaws are smoke tests:
> Separating the tests physically to different module is artificial and
> not much semantic: If any tests starts being considered as smoke test,
> what we do? Move it to different module and change package name. Does
> that make sense? Wouldn't it be better to just add
> `(a)Category(Smoke.class)` ?
> It also collides with the technical solution of separating tests by type
> of server setup (multinode, ...). If we decided to include a multinode
> test as smoke, what would we do? Move it to smoke module and create
> another surefire execution? Doesn't make much sense. Just adding the
> `Smoke` category and filtering the tests to be run using categories is
> way better.
> Last but not least: These categories are /totally optional/, no one has
> to go through the tests and add them.
> It's a tool for use cases when this additional categorization is
> necessary, like, security-related tests, it's subset Common Criteria
> tests, JPA-related tests, "performance smoke tests", etc.
> Other tests would remain untouched. The packages will remain the main
> manner of categorization.
> *Who needs it?*
> Mostly QA - when developing tests harness, when testing under various
> conditions (different DB, IPv6, Common Criteria, security policy, ...)
> *What will change?*
> The only change needed is
> * to define the groups (as empty interfaces)
> * to add the @Category( <GroupInterface>.class ) to test's class (and
> add it to Arquillian's deployment if necessary).
> Purpose of each group will be documented in the interface's javadoc.
> *Tags (groups, if you like)*
> So far, these are considered:
> *- CommonCriteria *
> *- Management*
> *- Security*
> Stuart's note: It would be nice if the annotation could be applied at a
> package level, as security tests are already isolated into individual
> *- Slow*
> Needs a definition of what's "slow", possibly in terms of comparison
> some quick test.
> *- Negative*
> This one is different kind: QA wants to mark the negative tests to track
> where they have some and where they don't.
> So this is rather supposed for quality engineering tools.
> *- Smoke (maybe)*
> Stuart's note: Unless we actually move the smoke tests into
> testsuite/integration/basic then this is pointless.
> There are adv and disadv for moving tests into .../basic:
> + It would make adding/removing a test to the smoke group easier.
> + It would allow marking also tests in other modules as smoke.
> * Determining whether a test is smoke test would need to go and look.
> But does anyone need to know it when not actually managing the set of
> smoke tests, in which case they would just look for the usages of the
> Smoke type?
> - To get the current set, you'd need to search for Smoke type
> appearances, or to run the smoke tests. On a related note, I have a
> feature request in Surefire to dry-run tests (just list which would run).
> jboss-as7-dev mailing list