[JBoss JIRA] (JBTM-3235) Performance regression tests should consult the "Score Error" before reporting regressions
by Michael Musgrove (Jira)
[ https://issues.redhat.com/browse/JBTM-3235?page=com.atlassian.jira.plugin... ]
Michael Musgrove updated JBTM-3235:
-----------------------------------
Description:
We perform [microbenchmark performance testing of JTA|https://github.com/jbosstm/performance/tree/master/narayana/ArjunaJTA...] and we have Jenkins CI jobs (btny-pulls-performance and narayana-performance-version-comparison) that do regression testing against different versions of the product.
These tests use [JMH|https://openjdk.java.net/projects/code-tools/jmh/] to perform the test runs. The results of each run are reported as a CSV file in the following example format:
{code}
"Version","Benchmark","Mode","Threads","Samples","Score","Score Error (99.9%)","Unit"
5.5.32.Final,"io.narayana.perf.product.NarayanaComparison.test","thrpt",240,15,217253.976024,8720.877211,"ops/s"
5.9.0.Final,"io.narayana.perf.product.NarayanaComparison.test","thrpt",240,15,216552.983213,6586.482104,"ops/s"
{code}
This example shows a comparison between versions 5.5.32.Final and 5.9.0.Final. The data shows that
we measured using a mode called "thrpt" (ie transactions per second or TPS) and the Score shows the TPS was 217253 vers
us 216552. There is also a Score Error which I understand indicates the upper and lower error bars for the measurement.
None of our jobs take account of the error bars when reporting regressions. What we need to do is look at the Score Error to see if the runs are within tolerance before reporting a regression.
An example of how to manually run a regression test is as follows:
{code}
git clone https://github.com/jbosstm/performance # clone the performance git repo
cd performance
export JMHARGS="-t 240 -r 30 -f 3 -wi 5 -i 5" # tune the test using JMH config params
# and then do a performance run:
export version=5.5.32.Final # set the target narayana version
./build.sh -f narayana/ArjunaJTA/jta/pom.xml clean install -DskipTests -Dnarayana.version=$version
java -classpath narayana/ArjunaJTA/jta/target/classes/ -Dcom.arjuna.ats.arjuna.common.propertiesFile=jbossts-properties.xml -jar narayana/ArjunaJTA/jta/target/benchmarks.jar io.narayana.perf.product.NarayanaComparison.* $JMHARGS -rf csv -rff io.narayana.perf.product.NarayanaComparison-$version.csv
{code}
The results are in the named csv file.
Then do the same for the second version
was:
We perform [microbenchmark performance testing of JTA|https://github.com/jbosstm/performance/tree/master/narayana/ArjunaJTA...] and we have Jenkins CI jobs (btny-pulls-performance and narayana-performance-version-comparison) that do regression testing against different versions of the product.
These tests use [JMH|https://openjdk.java.net/projects/code-tools/jmh/] to perform the test runs. The results of each run are reported as a CSV file in the following example format:
"Version","Benchmark","Mode","Threads","Samples","Score","Score Error (99.9%)","Unit"
5.5.32.Final,"io.narayana.perf.product.NarayanaComparison.test","thrpt",240,15,217253.976024,8720.877211,"ops/s"
5.9.0.Final,"io.narayana.perf.product.NarayanaComparison.test","thrpt",240,15,216552.983213,6586.482104,"ops/s"
This example shows a comparison between versions 5.5.32.Final and 5.9.0.Final. The data shows that
we measured using a mode called "thrpt" (ie transactions per second or TPS) and the Score shows the TPS was 217253 vers
us 216552. There is also a Score Error which I understand indicates the upper and lower error bars for the measurement.
None of our jobs take account of the error bars when reporting regressions. What we need to do is look at the Score Error to see if the runs are within tolerance before reporting a regression.
An example of how to manually run a regression test is as follows:
{code}
git clone https://github.com/jbosstm/performance # clone the performance git repo
cd performance
export JMHARGS="-t 240 -r 30 -f 3 -wi 5 -i 5" # tune the test using JMH config params
# and then do a performance run:
export version=5.5.32.Final # set the target narayana version
./build.sh -f narayana/ArjunaJTA/jta/pom.xml clean install -DskipTests -Dnarayana.version=$version
java -classpath narayana/ArjunaJTA/jta/target/classes/ -Dcom.arjuna.ats.arjuna.common.propertiesFile=jbossts-properties.xml -jar narayana/ArjunaJTA/jta/target/benchmarks.jar io.narayana.perf.product.NarayanaComparison.* $JMHARGS -rf csv -rff io.narayana.perf.product.NarayanaComparison-$version.csv
{code}
The results are in the named csv file.
Then do the same for the second version
> Performance regression tests should consult the "Score Error" before reporting regressions
> ------------------------------------------------------------------------------------------
>
> Key: JBTM-3235
> URL: https://issues.redhat.com/browse/JBTM-3235
> Project: JBoss Transaction Manager
> Issue Type: Task
> Components: Performance Testing
> Affects Versions: 5.10.1.Final
> Reporter: Michael Musgrove
> Assignee: Michael Musgrove
> Priority: Optional
> Fix For: 5.next
>
>
> We perform [microbenchmark performance testing of JTA|https://github.com/jbosstm/performance/tree/master/narayana/ArjunaJTA...] and we have Jenkins CI jobs (btny-pulls-performance and narayana-performance-version-comparison) that do regression testing against different versions of the product.
> These tests use [JMH|https://openjdk.java.net/projects/code-tools/jmh/] to perform the test runs. The results of each run are reported as a CSV file in the following example format:
> {code}
> "Version","Benchmark","Mode","Threads","Samples","Score","Score Error (99.9%)","Unit"
> 5.5.32.Final,"io.narayana.perf.product.NarayanaComparison.test","thrpt",240,15,217253.976024,8720.877211,"ops/s"
> 5.9.0.Final,"io.narayana.perf.product.NarayanaComparison.test","thrpt",240,15,216552.983213,6586.482104,"ops/s"
> {code}
> This example shows a comparison between versions 5.5.32.Final and 5.9.0.Final. The data shows that
> we measured using a mode called "thrpt" (ie transactions per second or TPS) and the Score shows the TPS was 217253 vers
> us 216552. There is also a Score Error which I understand indicates the upper and lower error bars for the measurement.
> None of our jobs take account of the error bars when reporting regressions. What we need to do is look at the Score Error to see if the runs are within tolerance before reporting a regression.
> An example of how to manually run a regression test is as follows:
> {code}
> git clone https://github.com/jbosstm/performance # clone the performance git repo
> cd performance
> export JMHARGS="-t 240 -r 30 -f 3 -wi 5 -i 5" # tune the test using JMH config params
> # and then do a performance run:
> export version=5.5.32.Final # set the target narayana version
> ./build.sh -f narayana/ArjunaJTA/jta/pom.xml clean install -DskipTests -Dnarayana.version=$version
> java -classpath narayana/ArjunaJTA/jta/target/classes/ -Dcom.arjuna.ats.arjuna.common.propertiesFile=jbossts-properties.xml -jar narayana/ArjunaJTA/jta/target/benchmarks.jar io.narayana.perf.product.NarayanaComparison.* $JMHARGS -rf csv -rff io.narayana.perf.product.NarayanaComparison-$version.csv
> {code}
> The results are in the named csv file.
> Then do the same for the second version
--
This message was sent by Atlassian Jira
(v7.13.8#713008)