[JBoss JIRA] (JBTM-2419) Reduce the number of performance regression tests that run against a PR
by Michael Musgrove (Jira)
[ https://issues.redhat.com/browse/JBTM-2419?page=com.atlassian.jira.plugin... ]
Michael Musgrove updated JBTM-2419:
-----------------------------------
Summary: Reduce the number of performance regression tests that run against a PR (was: Consider reducing the number of performance regression tests ran against a PR)
> Reduce the number of performance regression tests that run against a PR
> -----------------------------------------------------------------------
>
> Key: JBTM-2419
> URL: https://issues.redhat.com/browse/JBTM-2419
> Project: JBoss Transaction Manager
> Issue Type: Task
> Components: Testing
> Affects Versions: 5.1.1
> Reporter: Michael Musgrove
> Priority: Minor
> Fix For: 5.next
>
>
> We run our JMH tests multiple times in order to estimate run-to-run variance. The [JMH docs|http://hg.openjdk.java.net/code-tools/jmh/file/tip/jmh-samples/src/m...] state:
> bq. JVMs are complex systems, and the non-determinism is inherent for them. This requires us to always account the run-to-run variance as the one of the effects in our experiments. Forking aggregates the results across several JVM launches.
> Currently we set the number of forks to 10. We run each test 6 times with each run lasting 1 min. Since we have 10 tests this means it takes 10 * 10 * 6 minutes for the benchmarks to complete. In the future we want to keep adding new performance tests and the time taken to run a PR will become prohibitive. I would like to propose one of the following solutions:
> - reduce the number of forks;
> - run a subset of performance tests on each PR and then run them all once a week or so on the main build
> My preference is to reduce the number of forks to 3. Note that the PR requester has the option to disable performance tests for his PR (by including !PERF in the comment section).
--
This message was sent by Atlassian Jira
(v7.13.8#713008)