[jbosscache-dev] Re: Maven TestNG rewrite complete for JBoss Cache and POJO Cache

Hello Everyone, All 2516 JBoss Cache and POJO Cache tests have been ported from JUnit to TestNG. Since this involved some significant changes I took the opportunity to remove all compiler warnings, which notably reduces the warning noise. I have introduced primary test groups (explained below) and have assigned the relevant tests. I also have refactored the maven build to handle what I refer to as permutations. Essentially it allows us to rerun subsections of tests using different configurations. This will allow us to test things like different jgroups protocol stacks using the same (but relevant) tests. There is one small outstanding issue, which is the unreliable and buggy nature of system output redirection (one of the many maven surefire bugs). So currently when you execute a testrun all sys out will go to your terminal. Finally, I have written a detailed overview of the maven + testng testing mechanics, and have included it in the project README: http://anonsvn.jboss.org/repos/jbosscache/core/trunk/README-Maven.txt I have also attached the additional overview section below. Thanks, -Jason Testing ======= Tests are written against the TestNG testing framework. Each test should belong to one or more group. The group acts as a filter, and is used to select which tests are ran as part of the maven test lifecycle. There are 3 groups that are currently in use, but there is not formal, you can make up any group name if you like. Current Groups -------------- * functional - Tests which test the general functionality of JBoss Cache * jgroups - Tests which need to send data on a JGroups Channel * transaction - Tests which use a transaction manager It should be noted that every test should at least be in the functional group, since this is the default test group that is executed by maven, and the one that is required to prepare a release. Executing the default test run ------------------------------ The default run executes all tests in the functional group. To just run the tests with txt and xml output the command is: mvn test Alternatively, you can execute the tests AND generate a report with: mvn surefire-report:report If you already have ran a test cycle, and you want to generate a report off the current reports, then you use the report-only goal, ike so: mvn surefire-report:report-only Executing different groups -------------------------- A group can be executed (using the default configuration) by simply using the groups property like so: mvn -Dgroups=jgroups surefire-report:report Mutiple groups can also be executed, although if a test is in more than of the selected groups, it is executed only once: mvn -Dgroups=jgroups,transaction surefire-report:report Executing a single test ----------------------- A single test can be executed using the test property. The value is the short name (not the fully qualified package name) of the test. mvn -Dtest=FqnTest Skipping the test run --------------------- It is sometimes desirable to install the jboss cache package in your local repository without performing a full test run. To do this, simply use the maven.test.skip.exec property: mvn -Dmaven.test.skip.exec=true install Again, this is just a shortcut for local use. It SHOULD NEVER BE USED when releasing. Also, make sure "exec" is included in the property, if not the tests will not be built, which will prevent a test jar being produced (POJO Cache needs the Core Cache test jar). Permutations ------------ We use the term permutation to describe a group execution against a particular config. This allows us to test a variety of environments and configurations without rewriting the same basic test over and over again. For example, the jgroups-tcp permutation executes the jgroups group using the TCP config. Each permutation requires a maven profile which defines the various options, environmental variables, etc. The command to run the jgroups-tcp permutatin is: mvn -Pjgroups-tcp surefire-report:report Each permutation uses its own report directory, and its own html output file name. This allows you to execute multiple permutations without wiping the results from the previous run. Note that due to the way maven operates, only one permutation can be executed per mvn command. So automating multiple runs requires shell scripting, or some other execution framework to make multiple called to maven.