[jbosscache-dev] Re: Maven TestNG rewrite complete for JBoss Cache and POJO Cache

[Manik Surtani]

Brilliant - thanks for taking care of this!

On 12 Sep 2007, at 06:49, Jason T. Greene wrote:

> Hello Everyone,
>
> All 2516 JBoss Cache and POJO Cache tests have been ported from  
> JUnit to TestNG. Since this involved some significant changes I  
> took the opportunity to remove all compiler warnings, which notably  
> reduces the warning noise. I have introduced primary test groups  
> (explained below) and have assigned the relevant tests.
>
> I also have refactored the maven build to handle what I refer to as  
> permutations. Essentially it allows us to rerun subsections of  
> tests using different configurations. This will allow us to test  
> things like different jgroups protocol stacks using the same (but  
> relevant) tests.
>
> There is one small outstanding issue, which is the unreliable and  
> buggy nature of system output redirection (one of the many maven  
> surefire bugs). So currently when you execute a testrun all sys out  
> will go to your terminal.
>
>
> Finally, I have written a detailed overview of the maven + testng  
> testing mechanics, and have included it in the project README:
> http://anonsvn.jboss.org/repos/jbosscache/core/trunk/README-Maven.txt
>
> I have also attached the additional overview section below.
>
> Thanks,
> -Jason
>
>
> Testing
> =======
>
> Tests are written against the TestNG testing framework. Each test  
> should belong to one or more group. The group acts as a filter, and  
> is used to select which tests are ran as part of the maven test  
> lifecycle. There are 3 groups that are currently in use, but there  
> is not formal, you can make up any group name if you like.
>
> Current Groups
> --------------
> * functional - Tests which test the general functionality of JBoss  
> Cache
> * jgroups - Tests which need to send data on a JGroups Channel
> * transaction - Tests which use a transaction manager
>
> It should be noted that every test should at least be in the  
> functional group, since this is the default test group that is  
> executed by maven, and the one that is required to prepare a release.
>
> Executing the default test run
> ------------------------------
> The default run executes all tests in the functional group. To just  
> run the tests with txt and xml output the command is:
>
> mvn test
>
> Alternatively, you can execute the tests AND generate a report with:
>
> mvn surefire-report:report
>
> If you already have ran a test cycle, and you want to generate a  
> report off the current reports, then you use the report-only goal,  
> ike so:
>
> mvn surefire-report:report-only
>
> Executing different groups
> --------------------------
> A group can be executed (using the default configuration) by simply  
> using the groups property like so:
>
> mvn -Dgroups=jgroups surefire-report:report
>
> Mutiple groups can also be executed, although if a test is in more  
> than of the selected groups, it is executed only once:
>
> mvn -Dgroups=jgroups,transaction surefire-report:report
>
> Executing a single test
> -----------------------
> A single test can be executed using the test property. The value is  
> the short name (not the fully qualified package name) of the test.
>
> mvn -Dtest=FqnTest
>
> Skipping the test run
> ---------------------
> It is sometimes desirable to install the jboss cache package in  
> your local repository without performing a full test run. To do  
> this, simply use the maven.test.skip.exec property:
>
> mvn -Dmaven.test.skip.exec=true install
>
> Again, this is just a shortcut for local use. It SHOULD NEVER BE  
> USED when releasing. Also, make sure "exec" is included in the  
> property, if not the tests will not be built, which will prevent a  
> test jar being produced (POJO Cache needs the Core Cache test jar).
>
> Permutations
> ------------
> We use the term permutation to describe a group execution against a  
> particular config. This allows us to test a variety of environments  
> and configurations without rewriting the same basic test over and  
> over again. For example, the jgroups-tcp permutation executes the  
> jgroups group using the TCP config. Each permutation requires a  
> maven profile which defines the various options, environmental  
> variables, etc. The command to run the jgroups-tcp permutatin is:
>
> mvn -Pjgroups-tcp surefire-report:report
>
> Each permutation uses its own report directory, and its own html  
> output file name. This allows you to execute multiple permutations  
> without wiping the results from the previous run. Note that due to  
> the way maven  operates, only one permutation can be executed per  
> mvn command. So automating multiple runs requires shell scripting,  
> or some other execution framework to make multiple called to maven.
>
>
>

--
Manik Surtani

Lead, JBoss Cache
JBoss, a division of Red Hat