<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; CHARSET=UTF-8">
<META NAME="GENERATOR" CONTENT="GtkHTML/4.6.0">
</HEAD>
<BODY>
Ok, so it seems that there are three tags for which are accepted:<BR>
<BR>
<UL>
<LI>Management
<LI>Security
<BLOCKQUOTE>
<LI>CommonCriteria
</BLOCKQUOTE>
</UL>
<BR>
So I have limited the PR to just these, and applied the CommonCriteria to the list I got from Peter Skopek.<BR>
<A HREF="https://github.com/jbossas/jboss-as/pull/3550">https://github.com/jbossas/jboss-as/pull/3550</A><BR>
The other two are to be applied later by Peter and Dominik Pospisil.<BR>
<BR>
Ondra<BR>
<BR>
<BR>
On Thu, 2012-11-29 at 08:22 -0600, Brian Stansberry wrote:
<BLOCKQUOTE TYPE=CITE>
<PRE>
I'm OK with this, actually think it's a good idea, but it's not a case
of "can only help, without doing any harm." The harm is diverting
resources into categorizing tests. The help is that it makes it much
easier to perform useful categorizations of tests; the harm comes when
that apparent ease leads to unproductive categorization and we die a
death by a thousand cuts.
Regarding the details:
Slow -- is there a way to say "exclude the @Slow tests"?
</PRE>
</BLOCKQUOTE>
Yes.
<BLOCKQUOTE TYPE=CITE>
<PRE>
Even then I have my doubts about usefulness. Why would
someone run all tests except the @Slow ones, when that will still take a
pretty long time and lightning can run them all?
</PRE>
</BLOCKQUOTE>
I'd consider it something between smoke and full run. Broadened developer run. It was common suggestion for category. If no one specifically asks for it now, well, let's skip this one.<BR>
<BR>
<BLOCKQUOTE TYPE=CITE>
<PRE>
Negative -- this triggers my "unproductive categorization" feelers. I
don't see myself spending energy trying to apply this to tests I write,
I don't see myself monitoring application of this annotation in patches
I review, and I'd be shocked if I'm unique in those regards. If it's not
applied to a very high % of the relevant tests, what use is it?
</PRE>
</BLOCKQUOTE>
It would be a tag for QE tools - for knowing what area is covered by negative tests.<BR>
<BR>
<BLOCKQUOTE TYPE=CITE>
<PRE>
Smoke -- I think this could be useful because maintaining tests
separately from other related tests in basic seems unproductive. We
likely have some tests in smoke that shouldn't be and some in basic that
could be smoke tests, and this situation persists just because moving
things around is too much work.
</PRE>
</BLOCKQUOTE>
+1.
<PRE>
</PRE>
<BLOCKQUOTE TYPE=CITE>
<PRE>
But it's important to keep in mind the purpose of the smoke tests -- to
be run on every build, without burdening the developer with the need to
wait for the full -DallTests run. However, any patch is getting
automatically tested by lightning with -DallTests at least once before
getting merged. That reduces the value of smoke tests, to the point
where I'm starting to doubt if they have any real value at all. So IMHO
we shouldn't let the ease of use of a @Smoke annotation lead to a
proliferation of smoke tests that just slow down developer builds.
</PRE>
</BLOCKQUOTE>
<BR>
<BR>
<BLOCKQUOTE TYPE=CITE>
<PRE>
On 11/29/12 12:01 AM, Ondrej Zizka wrote:
<FONT COLOR="#737373">> Hello all,</FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">> I'd like to discuss a small improvement of test grouping.</FONT>
<FONT COLOR="#737373">> Please let me know what do you think about it, any pros/cons comments,</FONT>
<FONT COLOR="#737373">> and additions, are welcome.</FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">> First, the TLDR version:</FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">> *The change consists of adding a few marking interfaces in</FONT>
<FONT COLOR="#737373">> testsuite-shared and a tagging some tests using these in an annotation.*</FONT>
<FONT COLOR="#737373">> *Nothing else, everything will work as before. People not interested in</FONT>
<FONT COLOR="#737373">> this won't notice any change.*</FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">> Tests have some cross-cutting concerns, which can't be covered by a</FONT>
<FONT COLOR="#737373">> package-based tree structure.</FONT>
<FONT COLOR="#737373">> For example, a set of tests for Common Criteria, or for Management. This</FONT>
<FONT COLOR="#737373">> request comes mainly from QA.</FONT>
<FONT COLOR="#737373">> Also, we are bound by technical solutions coming from Maven's and</FONT>
<FONT COLOR="#737373">> Arquillian's limitations.</FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">> I'd like to leverage JUnit's @Category to tag the tests, to give the</FONT>
<FONT COLOR="#737373">> testsuite users possibility to run such groups of tests from different</FONT>
<FONT COLOR="#737373">> packages and modules,</FONT>
<FONT COLOR="#737373">> just by stating e.g. -Dtest.group=CommonCriteria, instead of identifying</FONT>
<FONT COLOR="#737373">> and maintaining lists of packages and classes.</FONT>
<FONT COLOR="#737373">> Having an additional way to freely create groups without changing the</FONT>
<FONT COLOR="#737373">> package structure can only help, without doing any harm.</FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">> Now the whole story. It's also at</FONT>
<FONT COLOR="#737373">> <A HREF="https://community.jboss.org/wiki/AS8TestsuiteProposals">https://community.jboss.org/wiki/AS8TestsuiteProposals</A> so feel free to</FONT>
<FONT COLOR="#737373">> add your notes there.</FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">> *Motivation*</FONT>
<FONT COLOR="#737373">> The tests are categorized into modules and then further to packages.</FONT>
<FONT COLOR="#737373">> This forms a tree structure, in which we follow the most important</FONT>
<FONT COLOR="#737373">> distinctive feature/purpose of the test.</FONT>
<FONT COLOR="#737373">> There are cases the tests need to be categorized in multiple ways. E.g.</FONT>
<FONT COLOR="#737373">> security or management tests. It's more practical to keep them separated</FONT>
<FONT COLOR="#737373">> by functionality tested.</FONT>
<FONT COLOR="#737373">> But that means that when the group is to be run, user needs to provide</FONT>
<FONT COLOR="#737373">> (and maintain) a long list of packages to run.</FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">> There's a solution: JUnit's @Category. See AS7-5844</FONT>
<FONT COLOR="#737373">> <<A HREF="https://issues.jboss.org/browse/AS7-5844">https://issues.jboss.org/browse/AS7-5844</A>>. Using that, the tests may be</FONT>
<FONT COLOR="#737373">> tagged, and special groups can be created, instead of creating and</FONT>
<FONT COLOR="#737373">> maintaining list of packages or even classes.</FONT>
<FONT COLOR="#737373">> It's not supposed to replace package-based fine-grained division, but to</FONT>
<FONT COLOR="#737373">> complement it.</FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">> Another example where current model has some flaws are smoke tests:</FONT>
<FONT COLOR="#737373">> Separating the tests physically to different module is artificial and</FONT>
<FONT COLOR="#737373">> not much semantic: If any tests starts being considered as smoke test,</FONT>
<FONT COLOR="#737373">> what we do? Move it to different module and change package name. Does</FONT>
<FONT COLOR="#737373">> that make sense? Wouldn't it be better to just add</FONT>
<FONT COLOR="#737373">> `@Category(Smoke.class)` ?</FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">> It also collides with the technical solution of separating tests by type</FONT>
<FONT COLOR="#737373">> of server setup (multinode, ...). If we decided to include a multinode</FONT>
<FONT COLOR="#737373">> test as smoke, what would we do? Move it to smoke module and create</FONT>
<FONT COLOR="#737373">> another surefire execution? Doesn't make much sense. Just adding the</FONT>
<FONT COLOR="#737373">> `Smoke` category and filtering the tests to be run using categories is</FONT>
<FONT COLOR="#737373">> way better.</FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">> Last but not least: These categories are /totally optional/, no one has</FONT>
<FONT COLOR="#737373">> to go through the tests and add them.</FONT>
<FONT COLOR="#737373">> It's a tool for use cases when this additional categorization is</FONT>
<FONT COLOR="#737373">> necessary, like, security-related tests, it's subset Common Criteria</FONT>
<FONT COLOR="#737373">> tests, JPA-related tests, "performance smoke tests", etc.</FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">> Other tests would remain untouched. The packages will remain the main</FONT>
<FONT COLOR="#737373">> manner of categorization.</FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">> *Who needs it?*</FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">> Mostly QA - when developing tests harness, when testing under various</FONT>
<FONT COLOR="#737373">> conditions (different DB, IPv6, Common Criteria, security policy, ...)</FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">> *What will change?*</FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">> The only change needed is</FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">> * to define the groups (as empty interfaces)</FONT>
<FONT COLOR="#737373">> * to add the @Category( <GroupInterface>.class ) to test's class (and</FONT>
<FONT COLOR="#737373">> add it to Arquillian's deployment if necessary).</FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">> Purpose of each group will be documented in the interface's javadoc.</FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">> *Tags (groups, if you like)*</FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">> So far, these are considered:</FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">> *- CommonCriteria *</FONT>
<FONT COLOR="#737373">> *- Management*</FONT>
<FONT COLOR="#737373">> *- Security*</FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">> Stuart's note: It would be nice if the annotation could be applied at a</FONT>
<FONT COLOR="#737373">> package level, as security tests are already isolated into individual</FONT>
<FONT COLOR="#737373">> packages</FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">> *- Slow*</FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">> Needs a definition of what's "slow", possibly in terms of comparison to</FONT>
<FONT COLOR="#737373">> some quick test.</FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">> *- Negative*</FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">> This one is different kind: QA wants to mark the negative tests to track</FONT>
<FONT COLOR="#737373">> where they have some and where they don't.</FONT>
<FONT COLOR="#737373">> So this is rather supposed for quality engineering tools.</FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">> *- Smoke (maybe)*</FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">> Stuart's note: Unless we actually move the smoke tests into</FONT>
<FONT COLOR="#737373">> testsuite/integration/basic then this is pointless.</FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">> There are adv and disadv for moving tests into .../basic:</FONT>
<FONT COLOR="#737373">> + It would make adding/removing a test to the smoke group easier.</FONT>
<FONT COLOR="#737373">> + It would allow marking also tests in other modules as smoke.</FONT>
<FONT COLOR="#737373">> * Determining whether a test is smoke test would need to go and look.</FONT>
<FONT COLOR="#737373">> But does anyone need to know it when not actually managing the set of</FONT>
<FONT COLOR="#737373">> smoke tests, in which case they would just look for the usages of the</FONT>
<FONT COLOR="#737373">> Smoke type?</FONT>
<FONT COLOR="#737373">> - To get the current set, you'd need to search for Smoke type</FONT>
<FONT COLOR="#737373">> appearances, or to run the smoke tests. On a related note, I have a</FONT>
<FONT COLOR="#737373">> feature request in Surefire to dry-run tests (just list which would run).</FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">> Regards,</FONT>
<FONT COLOR="#737373">> Ondra</FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">></FONT>
<FONT COLOR="#737373">> _______________________________________________</FONT>
<FONT COLOR="#737373">> jboss-as7-dev mailing list</FONT>
<FONT COLOR="#737373">> <A HREF="mailto:jboss-as7-dev@lists.jboss.org">jboss-as7-dev@lists.jboss.org</A></FONT>
<FONT COLOR="#737373">> <A HREF="https://lists.jboss.org/mailman/listinfo/jboss-as7-dev">https://lists.jboss.org/mailman/listinfo/jboss-as7-dev</A></FONT>
<FONT COLOR="#737373">></FONT>
</PRE>
</BLOCKQUOTE>
<BR>
</BODY>
</HTML>