[jbossts-issues] [JBoss JIRA] (JBTM-2419) Consider reducing the number of performance regression tests ran against a PR

Consider reducing the number of performance regression tests ran against a PR ----------------------------------------------------------------------------- Key: JBTM-2419 URL: https://issues.redhat.com/browse/JBTM-2419 Project: JBoss Transaction Manager Issue Type: Task Components: Testing Affects Versions: 5.1.1 Reporter: Michael Musgrove Priority: Minor Fix For: 5.next We run our JMH tests multiple times in order to estimate run-to-run variance. The [JMH docs|http://hg.openjdk.java.net/code-tools/jmh/file/tip/jmh-samples/src/m...] state: bq. JVMs are complex systems, and the non-determinism is inherent for them. This requires us to always account the run-to-run variance as the one of the effects in our experiments. Forking aggregates the results across several JVM launches. Currently we set the number of forks to 10. We run each test 6 times with each run lasting 1 min. Since we have 10 tests this means it takes 10 * 10 * 6 minutes for the benchmarks to complete. In the future we want to keep adding new performance tests and the time taken to run a PR will become prohibitive. I would like to propose one of the following solutions: - reduce the number of forks; - run a subset of performance tests on each PR and then run them all once a week or so on the main build My preference is to reduce the number of forks to 3. Note that the PR requester has the option to disable performance tests for his PR (by including !PERF in the comment section).