[jbosscache-dev] Maven TestNG rewrite complete for JBoss Cache and POJO Cache

[Jason T. Greene]

Hello Everyone,

All 2516 JBoss Cache and POJO Cache tests have been ported from JUnit to 
TestNG. Since this involved some significant changes I took the 
opportunity to remove all compiler warnings, which notably reduces the 
warning noise. I have introduced primary test groups (explained below) 
and have assigned the relevant tests.

I also have refactored the maven build to handle what I refer to as 
permutations. Essentially it allows us to rerun subsections of tests 
using different configurations. This will allow us to test things like 
different jgroups protocol stacks using the same (but relevant) tests.

There is one small outstanding issue, which is the unreliable and buggy 
nature of system output redirection (one of the many maven surefire 
bugs). So currently when you execute a testrun all sys out will go to 
your terminal.


Finally, I have written a detailed overview of the maven + testng 
testing mechanics, and have included it in the project README:
http://anonsvn.jboss.org/repos/jbosscache/core/trunk/README-Maven.txt

I have also attached the additional overview section below.

Thanks,
-Jason


Testing
=======

Tests are written against the TestNG testing framework. Each test should 
belong to one or more group. The group acts as a filter, and is used to 
select which tests are ran as part of the maven test lifecycle. There 
are 3 groups that are currently in use, but there is not formal, you can 
make up any group name if you like.

Current Groups
--------------
* functional - Tests which test the general functionality of JBoss Cache
* jgroups - Tests which need to send data on a JGroups Channel
* transaction - Tests which use a transaction manager

It should be noted that every test should at least be in the functional 
group, since this is the default test group that is executed by maven, 
and the one that is required to prepare a release.

Executing the default test run
------------------------------
The default run executes all tests in the functional group. To just run 
the tests with txt and xml output the command is:

mvn test

Alternatively, you can execute the tests AND generate a report with:

mvn surefire-report:report

If you already have ran a test cycle, and you want to generate a report 
off the current reports, then you use the report-only goal, ike so:

mvn surefire-report:report-only

Executing different groups
--------------------------
A group can be executed (using the default configuration) by simply 
using the groups property like so:

mvn -Dgroups=jgroups surefire-report:report

Mutiple groups can also be executed, although if a test is in more than 
of the selected groups, it is executed only once:

mvn -Dgroups=jgroups,transaction surefire-report:report

Executing a single test
-----------------------
A single test can be executed using the test property. The value is the 
short name (not the fully qualified package name) of the test.

mvn -Dtest=FqnTest

Skipping the test run
---------------------
It is sometimes desirable to install the jboss cache package in your 
local repository without performing a full test run. To do this, simply 
use the maven.test.skip.exec property:

mvn -Dmaven.test.skip.exec=true install

Again, this is just a shortcut for local use. It SHOULD NEVER BE USED 
when releasing. Also, make sure "exec" is included in the property, if 
not the tests will not be built, which will prevent a test jar being 
produced (POJO Cache needs the Core Cache test jar).

Permutations
------------
We use the term permutation to describe a group execution against a 
particular config. This allows us to test a variety of environments and 
configurations without rewriting the same basic test over and over 
again. For example, the jgroups-tcp permutation executes the jgroups 
group using the TCP config. Each permutation requires a maven profile 
which defines the various options, environmental variables, etc. The 
command to run the jgroups-tcp permutatin is:

mvn -Pjgroups-tcp surefire-report:report

Each permutation uses its own report directory, and its own html output 
file name. This allows you to execute multiple permutations without 
wiping the results from the previous run. Note that due to the way maven 
  operates, only one permutation can be executed per mvn command. So 
automating multiple runs requires shell scripting, or some other 
execution framework to make multiple called to maven.