[jboss-as7-dev] Testsuite - little test grouping improvement for AS 8

Ondrej Zizka ozizka at redhat.com
Fri Nov 30 05:25:06 EST 2012 

Forgot to mention, more replies in-line.

On Fri, 2012-11-30 at 10:35 +0100, Ondrej Zizka wrote:

> Ok, so it seems that there are three tags which are accepted:
> 
>       * Management 
>       * Security 
>         
>               * CommonCriteria 
> 
> 
> So I have limited the PR to just these, and applied the CommonCriteria
> to the list I got from Peter Skopek.
> https://github.com/jbossas/jboss-as/pull/3550
> The other two are to be applied later by Peter and Dominik Pospisil.



> On Thu, 2012-11-29 at 08:22 -0600, Brian Stansberry wrote: 
> 
> > I'm OK with this, actually think it's a good idea, but it's not a case 
> > of "can only help, without doing any harm." The harm is diverting 
> > resources into categorizing tests. The help is that it makes it much 
> > easier to perform useful categorizations of tests; the harm comes when 
> > that apparent ease leads to unproductive categorization and we die a 
> > death by a thousand cuts.
> > 
> > Regarding the details:
> > 
> > Slow -- is there a way to say "exclude the @Slow tests"? 
> 
> Yes. 
> 
> > Even then I have my doubts about usefulness. Why would 
> > someone run all tests except the @Slow ones, when that will still take a 
> > pretty long time and lightning can run them all?
> 
> I'd consider it something between smoke and full run. Broadened
> developer run. It was common suggestion for category. If no one
> specifically asks for it now, well, let's skip this one.
> 
> 
> > Negative -- this triggers my "unproductive categorization" feelers. I 
> > don't see myself spending energy trying to apply this to tests I write, 
> > I don't see myself monitoring application of this annotation in patches 
> > I review, and I'd be shocked if I'm unique in those regards. If it's not 
> > applied to a very high % of the relevant tests, what use is it?
> 
> It would be a tag for QE tools - for knowing what area is covered by
> negative tests.
> 
> 
> > Smoke -- I think this could be useful because maintaining tests 
> > separately from other related tests in basic seems unproductive. We 
> > likely have some tests in smoke that shouldn't be and some in basic that 
> > could be smoke tests, and this situation persists just because moving 
> > things around is too much work.
> 
> +1. 
> 
> 
> 
> > But it's important to keep in mind the purpose of the smoke tests -- to 
> > be run on every build, without burdening the developer with the need to 
> > wait for the full -DallTests run. However, any patch is getting 
> > automatically tested by lightning with -DallTests at least once before 
> > getting merged. That reduces the value of smoke tests, to the point 
> > where I'm starting to doubt if they have any real value at all. So IMHO 
> > we shouldn't let the ease of use of a @Smoke annotation lead to a 
> > proliferation of smoke tests that just slow down developer builds.
> 
> 
> 
> 
> > On 11/29/12 12:01 AM, Ondrej Zizka wrote:
> > > Hello all,
> > >
> > > I'd like to discuss a small improvement of test grouping.
> > > Please let me know what do you think about it, any pros/cons comments,
> > > and additions, are welcome.
> > >
> > > First, the TLDR version:
> > >
> > > *The change consists of adding a few marking interfaces in
> > > testsuite-shared and a tagging some tests using these in an annotation.*
> > > *Nothing else, everything will work as before. People not interested in
> > > this won't notice any change.*
> > >
> > > Tests have some cross-cutting concerns,  which can't be covered by a
> > > package-based tree structure.
> > > For example, a set of tests for Common Criteria, or for Management. This
> > > request comes mainly from QA.
> > > Also, we are bound by technical solutions coming from Maven's and
> > > Arquillian's limitations.
> > >
> > > I'd like to leverage JUnit's @Category to tag the tests, to give the
> > > testsuite users possibility to run such groups of tests from different
> > > packages and modules,
> > > just by stating e.g. -Dtest.group=CommonCriteria, instead of identifying
> > > and maintaining lists of packages and classes.
> > > Having an additional way to freely create groups without changing the
> > > package structure can only help, without doing any harm.
> > >
> > >
> > >
> > > Now the whole story. It's also at
> > > https://community.jboss.org/wiki/AS8TestsuiteProposals so feel free to
> > > add your notes there.
> > >
> > > *Motivation*
> > > The tests are categorized into modules and then further to packages.
> > > This forms a tree structure, in which we follow the most important
> > > distinctive feature/purpose of the test.
> > > There are cases the tests need to be categorized in multiple ways. E.g.
> > > security or management tests. It's more practical to keep them separated
> > > by functionality tested.
> > > But that means that when the group is to be run, user needs to provide
> > > (and maintain) a long list of packages to run.
> > >
> > > There's a solution: JUnit's @Category.  See AS7-5844
> > > <https://issues.jboss.org/browse/AS7-5844>. Using that, the tests may be
> > > tagged, and special groups can be created, instead of creating and
> > > maintaining list of packages or even classes.
> > > It's not supposed to replace package-based fine-grained division, but to
> > > complement it.
> > >
> > > Another example where current model has some flaws are smoke tests:
> > > Separating the tests physically to different module is artificial and
> > > not much semantic: If any tests starts being considered as smoke test,
> > > what we do? Move it to different module and change package name. Does
> > > that make sense? Wouldn't it be better to just add
> > > `@Category(Smoke.class)` ?
> > >
> > > It also collides with the technical solution of separating tests by type
> > > of server setup (multinode, ...). If we decided to include a multinode
> > > test as smoke, what would we do? Move it to smoke module and create
> > > another surefire execution? Doesn't make much sense. Just adding the
> > > `Smoke` category and filtering the tests to be run using categories is
> > > way better.
> > >
> > > Last but not least: These categories are /totally optional/, no one has
> > > to go through the tests and add them.
> > > It's a tool for use cases when this additional categorization is
> > > necessary, like, security-related tests, it's subset Common Criteria
> > > tests, JPA-related tests, "performance smoke tests", etc.
> > >
> > > Other tests would remain untouched. The packages will remain the main
> > > manner of categorization.
> > >
> > >
> > >       *Who needs it?*
> > >
> > > Mostly QA - when developing tests harness, when testing under various
> > > conditions (different DB, IPv6, Common Criteria, security policy, ...)
> > >
> > >
> > >       *What will change?*
> > >
> > > The only change needed is
> > >
> > >   * to define the groups (as empty interfaces)
> > >   * to add the @Category( <GroupInterface>.class ) to test's class (and
> > >     add it to Arquillian's deployment if necessary).
> > >
> > > Purpose of each group will be documented in the interface's javadoc.
> > >
> > >
> > >       *Tags (groups, if you like)*
> > >
> > > So far, these are considered:
> > >
> > >
> > >           *- CommonCriteria *
> > >           *- Management*
> > >           *- Security*
> > >
> > > Stuart's note: It would be nice if the annotation could be applied at a
> > > package level, as security tests are already isolated into individual
> > > packages
> > >
> > >
> > >           *- Slow*
> > >
> > > Needs a definition of what's "slow", possibly in terms of comparison to
> > > some quick test.
> > >
> > >
> > >           *- Negative*
> > >
> > > This one is different kind: QA wants to mark the negative tests to track
> > > where they have some and where they don't.
> > > So this is rather supposed for quality engineering tools.
> > >
> > >
> > >           *- Smoke (maybe)*
> > >
> > > Stuart's note: Unless we actually move the smoke tests into
> > > testsuite/integration/basic then this is pointless.
> > >
> > > There are adv and disadv for moving tests into .../basic:
> > > + It would make adding/removing a test to the smoke group easier.
> > > + It would allow marking also tests in other modules as smoke.
> > > *  Determining whether a test is smoke test would need to go and look.
> > > But does anyone need to know it when not actually managing the set of
> > > smoke tests, in which case they would just look for the usages of the
> > > Smoke type?
> > > -  To get the current set, you'd need to search for Smoke type
> > > appearances, or to run the smoke tests.  On a related note, I have a
> > > feature request in Surefire to dry-run tests (just list which would run).
> > >
> > >
> > > Regards,
> > > Ondra
> > >
> > >
> > > _______________________________________________
> > > jboss-as7-dev mailing list
> > > jboss-as7-dev at lists.jboss.org
> > > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
> > >
> > 
> > 
> 
> 
> 
> _______________________________________________
> jboss-as7-dev mailing list
> jboss-as7-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/jboss-as7-dev/attachments/20121130/3f101336/attachment.html

More information about the jboss-as7-dev mailing list