[jbossts-issues] [JBoss JIRA] (JBTM-2419) Consider reducing the number of JMH perf test forks on PRs

[Michael Musgrove (JIRA)]

Michael Musgrove created JBTM-2419:
--------------------------------------

             Summary: Consider reducing the number of JMH perf test forks on PRs
                 Key: JBTM-2419
                 URL: https://issues.jboss.org/browse/JBTM-2419
             Project: JBoss Transaction Manager
          Issue Type: Task
          Components: Testing
    Affects Versions: 5.1.1
            Reporter: Michael Musgrove
            Assignee: Michael Musgrove
             Fix For: 5.later


We run our JMH tests multiple times in order to estimate run-to-run variance. The [JMH docs|http://hg.openjdk.java.net/code-tools/jmh/file/tip/jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_13_RunToRun.java] state:
bq. JVMs are complex systems, and the non-determinism is inherent for them. This requires us to always account the run-to-run variance as the one of the effects in our experiments. Forking aggregates the results across several JVM launches.

Currently we set the number of forks to 10. We run each test 6 times with each run lasting 1 min. Since we have 10 tests this means it takes 10 * 10 * 6 minutes for the benchmarks to complete. In the future we want to keep adding new performance tests and the time taken to run a PR will become prohibitive. I would like to propose one of the following solutions:

- reduce the number of forks;
- run a subset of performance tests on each PR and then run them all once a week or so on the main build

My preference is to reduce the number of forks to 3. Note that the PR requester has the option to disable performance tests for his PR (by including !PERF in the comment section). 




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)