December 2019 - jbossts-issues - Jboss List Archives

[JBoss JIRA] (JBTM-3235) Performance regression tests should consult the "Score Error" before reporting regressions

by Michael Musgrove (Jira)

[ https://issues.redhat.com/browse/JBTM-3235?page=com.atlassian.jira.plugin... ] Michael Musgrove updated JBTM-3235: ----------------------------------- Description: We perform [microbenchmark performance testing of JTA|https://github.com/jbosstm/performance/tree/master/narayana/ArjunaJTA...] and we have Jenkins CI jobs (btny-pulls-performance and narayana-performance-version-comparison) that do regression testing against different versions of the product. These tests use [JMH|https://openjdk.java.net/projects/code-tools/jmh/] to perform the test runs. The results of each run are reported as a CSV file in the following example format: "Version","Benchmark","Mode","Threads","Samples","Score","Score Error (99.9%)","Unit" 5.5.32.Final,"io.narayana.perf.product.NarayanaComparison.test","thrpt",240,15,217253.976024,8720.877211,"ops/s" 5.9.0.Final,"io.narayana.perf.product.NarayanaComparison.test","thrpt",240,15,216552.983213,6586.482104,"ops/s" This example shows a comparison between versions 5.5.32.Final and 5.9.0.Final. The data shows that we measured using a mode called "thrpt" (ie transactions per second or TPS) and the Score shows the TPS was 217253 vers us 216552. There is also a Score Error which I understand indicates the upper and lower error bars for the measurement. None of our jobs take account of the error bars when reporting regressions. What we need to do is look at the Score Error to see if the runs are within tolerance before reporting a regression. An example of how to manually run a regression test is as follows: {code} git clone https://github.com/jbosstm/performance # clone the performance git repo cd performance export JMHARGS="-t 240 -r 30 -f 3 -wi 5 -i 5" # tune the test using JMH config params # and then do a performance run: export version=5.5.32.Final # set the target narayana version ./build.sh -f narayana/ArjunaJTA/jta/pom.xml clean install -DskipTests -Dnarayana.version=$version java -classpath narayana/ArjunaJTA/jta/target/classes/ -Dcom.arjuna.ats.arjuna.common.propertiesFile=jbossts-properties.xml -jar narayana/ArjunaJTA/jta/target/benchmarks.jar io.narayana.perf.product.NarayanaComparison.* $JMHARGS -rf csv -rff io.narayana.perf.product.NarayanaComparison-$version.csv {code} The results are in the named csv file. Then do the same for the second version was: We perform [microbenchmark performance testing of JTA|https://github.com/jbosstm/performance/tree/master/narayana/ArjunaJTA...] and we have Jenkins CI jobs (btny-pulls-performance and narayana-performance-version-comparison) that do regression testing against different versions of the product. These tests use [JMH|https://openjdk.java.net/projects/code-tools/jmh/] to perform the test runs. The results of each run are reported as a CSV file in the following example format: "Version","Benchmark","Mode","Threads","Samples","Score","Score Error (99.9%)","Unit" 5.5.32.Final,"io.narayana.perf.product.NarayanaComparison.test","thrpt",240,15,217253.976024,8720.877211,"ops/s" 5.9.0.Final,"io.narayana.perf.product.NarayanaComparison.test","thrpt",240,15,216552.983213,6586.482104,"ops/s" This example shows a comparison between versions 5.5.32.Final and 5.9.0.Final. The data shows that we measured using a mode called "thrpt" (ie transactions per second or TPS) and the Score shows the TPS was 217253 vers us 216552. There is also a Score Error which I understand indicates the upper and lower error bars for the measurement. None of our jobs take account of the error bars when reporting regressions. What we need to do is look at the Score Error to see if the runs are within tolerance before reporting a regression. git clone https://github.com/jbosstm/performance # clone the performance git repo cd performance export JMHARGS="-t 240 -r 30 -f 3 -wi 5 -i 5" # tune the test using JMH config params and then do a performance run: export version=5.5.32.Final # set the target narayana version ./build.sh -f narayana/ArjunaJTA/jta/pom.xml clean install -DskipTests -Dnarayana.version=$version java -classpath narayana/ArjunaJTA/jta/target/classes/ -Dcom.arjuna.ats.arjuna.common.propertiesFile=jbossts-properties.xml -jar narayana/ArjunaJTA/jta/target/benchmarks.jar io.narayana.perf.product.NarayanaComparison.* $JMHARGS -rf csv -rff io.narayana.perf.product.NarayanaComparison-$version.csv The results are in the named csv file. Then do the same for the second version > Performance regression tests should consult the "Score Error" before reporting regressions > ------------------------------------------------------------------------------------------ > > Key: JBTM-3235 > URL: https://issues.redhat.com/browse/JBTM-3235 > Project: JBoss Transaction Manager > Issue Type: Task > Components: Performance Testing > Affects Versions: 5.10.1.Final > Reporter: Michael Musgrove > Assignee: Michael Musgrove > Priority: Optional > Fix For: 5.next > > > We perform [microbenchmark performance testing of JTA|https://github.com/jbosstm/performance/tree/master/narayana/ArjunaJTA...] and we have Jenkins CI jobs (btny-pulls-performance and narayana-performance-version-comparison) that do regression testing against different versions of the product. > These tests use [JMH|https://openjdk.java.net/projects/code-tools/jmh/] to perform the test runs. The results of each run are reported as a CSV file in the following example format: > "Version","Benchmark","Mode","Threads","Samples","Score","Score Error (99.9%)","Unit" > 5.5.32.Final,"io.narayana.perf.product.NarayanaComparison.test","thrpt",240,15,217253.976024,8720.877211,"ops/s" > 5.9.0.Final,"io.narayana.perf.product.NarayanaComparison.test","thrpt",240,15,216552.983213,6586.482104,"ops/s" > This example shows a comparison between versions 5.5.32.Final and 5.9.0.Final. The data shows that > we measured using a mode called "thrpt" (ie transactions per second or TPS) and the Score shows the TPS was 217253 vers > us 216552. There is also a Score Error which I understand indicates the upper and lower error bars for the measurement. > None of our jobs take account of the error bars when reporting regressions. What we need to do is look at the Score Error to see if the runs are within tolerance before reporting a regression. > An example of how to manually run a regression test is as follows: > {code} > git clone https://github.com/jbosstm/performance # clone the performance git repo > cd performance > export JMHARGS="-t 240 -r 30 -f 3 -wi 5 -i 5" # tune the test using JMH config params > # and then do a performance run: > export version=5.5.32.Final # set the target narayana version > ./build.sh -f narayana/ArjunaJTA/jta/pom.xml clean install -DskipTests -Dnarayana.version=$version > java -classpath narayana/ArjunaJTA/jta/target/classes/ -Dcom.arjuna.ats.arjuna.common.propertiesFile=jbossts-properties.xml -jar narayana/ArjunaJTA/jta/target/benchmarks.jar io.narayana.perf.product.NarayanaComparison.* $JMHARGS -rf csv -rff io.narayana.perf.product.NarayanaComparison-$version.csv > {code} > The results are in the named csv file. > Then do the same for the second version -- This message was sent by Atlassian Jira (v7.13.8#713008)

4 years, 5 months

1
0
0 / 0

[JBoss JIRA] (JBTM-3235) Performance regression tests should consult the "Score Error" before reporting regressions

by Michael Musgrove (Jira)

Michael Musgrove created JBTM-3235: -------------------------------------- Summary: Performance regression tests should consult the "Score Error" before reporting regressions Key: JBTM-3235 URL: https://issues.redhat.com/browse/JBTM-3235 Project: JBoss Transaction Manager Issue Type: Task Components: Performance Testing Affects Versions: 5.10.1.Final Reporter: Michael Musgrove Assignee: Michael Musgrove Fix For: 5.next We perform [microbenchmark performance testing of JTA|https://github.com/jbosstm/performance/tree/master/narayana/ArjunaJTA...] and we have Jenkins CI jobs (btny-pulls-performance and narayana-performance-version-comparison) that do regression testing against different versions of the product. These tests use [JMH|https://openjdk.java.net/projects/code-tools/jmh/] to perform the test runs. The results of each run are reported as a CSV file in the following example format: "Version","Benchmark","Mode","Threads","Samples","Score","Score Error (99.9%)","Unit" 5.5.32.Final,"io.narayana.perf.product.NarayanaComparison.test","thrpt",240,15,217253.976024,8720.877211,"ops/s" 5.9.0.Final,"io.narayana.perf.product.NarayanaComparison.test","thrpt",240,15,216552.983213,6586.482104,"ops/s" This example shows a comparison between versions 5.5.32.Final and 5.9.0.Final. The data shows that we measured using a mode called "thrpt" (ie transactions per second or TPS) and the Score shows the TPS was 217253 vers us 216552. There is also a Score Error which I understand indicates the upper and lower error bars for the measurement. None of our jobs take account of the error bars when reporting regressions. What we need to do is look at the Score Error to see if the runs are within tolerance before reporting a regression. git clone https://github.com/jbosstm/performance # clone the performance git repo cd performance export JMHARGS="-t 240 -r 30 -f 3 -wi 5 -i 5" # tune the test using JMH config params and then do a performance run: export version=5.5.32.Final # set the target narayana version ./build.sh -f narayana/ArjunaJTA/jta/pom.xml clean install -DskipTests -Dnarayana.version=$version java -classpath narayana/ArjunaJTA/jta/target/classes/ -Dcom.arjuna.ats.arjuna.common.propertiesFile=jbossts-properties.xml -jar narayana/ArjunaJTA/jta/target/benchmarks.jar io.narayana.perf.product.NarayanaComparison.* $JMHARGS -rf csv -rff io.narayana.perf.product.NarayanaComparison-$version.csv The results are in the named csv file. Then do the same for the second version -- This message was sent by Atlassian Jira (v7.13.8#713008)

4 years, 5 months

1
0
0 / 0

[JBoss JIRA] (JBTM-3232) Conflicting JAX-RS paths in io.narayana.lra.coordinator.api.Coordinator

by Martin Stefanko (Jira)

[ https://issues.redhat.com/browse/JBTM-3232?page=com.atlassian.jira.plugin... ] Martin Stefanko commented on JBTM-3232: --------------------------------------- [~mmusgrov] I don't think that's the case. Please see https://issues.redhat.com/browse/JBTM-3234. > Conflicting JAX-RS paths in io.narayana.lra.coordinator.api.Coordinator > ----------------------------------------------------------------------- > > Key: JBTM-3232 > URL: https://issues.redhat.com/browse/JBTM-3232 > Project: JBoss Transaction Manager > Issue Type: Bug > Components: LRA > Affects Versions: 5.10.1.Final > Reporter: Martin Stefanko > Assignee: Martin Stefanko > Priority: Major > > In the JAX-RS resource io.narayana.lra.coordinator.api.Coordinator there are two conflicting JAX-RS paths: > {code:java} > @GET > @Path("{LraId}/status") > public Response getLRAStatus > {code} > and > {code:java} > @GET > @Path("{NestedLraId}/status") > public Response getNestedLRAStatus > {code} > which impose erroneous states for the nested LRAs. I would propose to prefix all endpoints for nested LRAs at the Coordinator class to be prefixed with `/nested`. -- This message was sent by Atlassian Jira (v7.13.8#713008)

4 years, 5 months

1
0
0 / 0

[JBoss JIRA] (JBTM-3232) Conflicting JAX-RS paths in io.narayana.lra.coordinator.api.Coordinator

by Michael Musgrove (Jira)

[ https://issues.redhat.com/browse/JBTM-3232?page=com.atlassian.jira.plugin... ] Michael Musgrove commented on JBTM-3232: ---------------------------------------- In fact the code should be identical with the other status endpoint so just delete it. The other endpoints with the NestedLraId param should not cause any issues so they can be left as they are. > Conflicting JAX-RS paths in io.narayana.lra.coordinator.api.Coordinator > ----------------------------------------------------------------------- > > Key: JBTM-3232 > URL: https://issues.redhat.com/browse/JBTM-3232 > Project: JBoss Transaction Manager > Issue Type: Bug > Components: LRA > Affects Versions: 5.10.1.Final > Reporter: Martin Stefanko > Assignee: Martin Stefanko > Priority: Major > > In the JAX-RS resource io.narayana.lra.coordinator.api.Coordinator there are two conflicting JAX-RS paths: > {code:java} > @GET > @Path("{LraId}/status") > public Response getLRAStatus > {code} > and > {code:java} > @GET > @Path("{NestedLraId}/status") > public Response getNestedLRAStatus > {code} > which impose erroneous states for the nested LRAs. I would propose to prefix all endpoints for nested LRAs at the Coordinator class to be prefixed with `/nested`. -- This message was sent by Atlassian Jira (v7.13.8#713008)

4 years, 5 months

1
0
0 / 0

[JBoss JIRA] (JBTM-3234) Coordinator#getNestedLRAStatus should return ParticipantStatus

by Martin Stefanko (Jira)

[ https://issues.redhat.com/browse/JBTM-3234?page=com.atlassian.jira.plugin... ] Martin Stefanko reassigned JBTM-3234: ------------------------------------- Assignee: Martin Stefanko (was: Michael Musgrove) > Coordinator#getNestedLRAStatus should return ParticipantStatus > -------------------------------------------------------------- > > Key: JBTM-3234 > URL: https://issues.redhat.com/browse/JBTM-3234 > Project: JBoss Transaction Manager > Issue Type: Bug > Components: LRA > Affects Versions: 5.10.1.Final > Reporter: Martin Stefanko > Assignee: Martin Stefanko > Priority: Major > > the method Coordinator#getNestedLRAStatus is used in LRA processing as normal `@Status` callback however currently it returns the status of the nested LRA which is of type LRAStatus and not ParticipantStatus which is the correct return type of `@Status` methods according to the specification. -- This message was sent by Atlassian Jira (v7.13.8#713008)

4 years, 5 months

1
0
0 / 0

[JBoss JIRA] (JBTM-3234) Coordinator#getNestedLRAStatus should return ParticipantStatus

by Martin Stefanko (Jira)

Martin Stefanko created JBTM-3234: ------------------------------------- Summary: Coordinator#getNestedLRAStatus should return ParticipantStatus Key: JBTM-3234 URL: https://issues.redhat.com/browse/JBTM-3234 Project: JBoss Transaction Manager Issue Type: Bug Components: LRA Affects Versions: 5.10.1.Final Reporter: Martin Stefanko Assignee: Michael Musgrove the method Coordinator#getNestedLRAStatus is used in LRA processing as normal `@Status` callback however currently it returns the status of the nested LRA which is of type LRAStatus and not ParticipantStatus which is the correct return type of `@Status` methods according to the specification. -- This message was sent by Atlassian Jira (v7.13.8#713008)

4 years, 5 months

1
0
0 / 0

[JBoss JIRA] (JBTM-3232) Conflicting JAX-RS paths in io.narayana.lra.coordinator.api.Coordinator

by Michael Musgrove (Jira)

[ https://issues.redhat.com/browse/JBTM-3232?page=com.atlassian.jira.plugin... ] Michael Musgrove commented on JBTM-3232: ---------------------------------------- Just delete the nested path (and move the code into the other endpoint). > Conflicting JAX-RS paths in io.narayana.lra.coordinator.api.Coordinator > ----------------------------------------------------------------------- > > Key: JBTM-3232 > URL: https://issues.redhat.com/browse/JBTM-3232 > Project: JBoss Transaction Manager > Issue Type: Bug > Components: LRA > Affects Versions: 5.10.1.Final > Reporter: Martin Stefanko > Assignee: Martin Stefanko > Priority: Major > > In the JAX-RS resource io.narayana.lra.coordinator.api.Coordinator there are two conflicting JAX-RS paths: > {code:java} > @GET > @Path("{LraId}/status") > public Response getLRAStatus > {code} > and > {code:java} > @GET > @Path("{NestedLraId}/status") > public Response getNestedLRAStatus > {code} > which impose erroneous states for the nested LRAs. I would propose to prefix all endpoints for nested LRAs at the Coordinator class to be prefixed with `/nested`. -- This message was sent by Atlassian Jira (v7.13.8#713008)

4 years, 5 months

1
0
0 / 0

[JBoss JIRA] (JBTM-3233) Port the JTA recovery quickstart to run with the Quarkus Narayana JTA extension

by Michael Musgrove (Jira)

Michael Musgrove created JBTM-3233: -------------------------------------- Summary: Port the JTA recovery quickstart to run with the Quarkus Narayana JTA extension Key: JBTM-3233 URL: https://issues.redhat.com/browse/JBTM-3233 Project: JBoss Transaction Manager Issue Type: Task Components: Quickstarts Affects Versions: 5.10.1.Final Reporter: Michael Musgrove Fix For: 5.next The [JTA recovery quickstart|https://github.com/jbosstm/quickstart/tree/master/ArjunaJTA/re...] with maven artefact id: org.jboss.narayana.quickstart.jta:recovery looks like it should run fine with the [Quarkus Narayana JTA extension|https://github.com/quarkusio/quarkus/tree/master/extensions/nar...]. This task is to check what we need to do with the quickstart to run it with the extension. Maybe include it as one of the [Quarkus quickstart examples|https://github.com/quarkusio/quarkus-quickstarts]. -- This message was sent by Atlassian Jira (v7.13.8#713008)

4 years, 5 months

1
0
0 / 0

[JBoss JIRA] (JBTM-3232) Conflicting JAX-RS paths in io.narayana.lra.coordinator.api.Coordinator

by Martin Stefanko (Jira)

Martin Stefanko created JBTM-3232: ------------------------------------- Summary: Conflicting JAX-RS paths in io.narayana.lra.coordinator.api.Coordinator Key: JBTM-3232 URL: https://issues.redhat.com/browse/JBTM-3232 Project: JBoss Transaction Manager Issue Type: Bug Components: LRA Affects Versions: 5.10.1.Final Reporter: Martin Stefanko Assignee: Martin Stefanko In the JAX-RS resource io.narayana.lra.coordinator.api.Coordinator there are two conflicting JAX-RS paths: {code:java} @GET @Path("{LraId}/status") public Response getLRAStatus {code} and {code:java} @GET @Path("{NestedLraId}/status") public Response getNestedLRAStatus {code} which impose erroneous states for the nested LRAs. I would propose to prefix all endpoints for nested LRAs at the Coordinator class to be prefixed with `/nested`. -- This message was sent by Atlassian Jira (v7.13.8#713008)

4 years, 5 months

1
0
0 / 0

[JBoss JIRA] (JBTM-3231) LRA recovery test fails after restart on JDK11

by Thomas Jenkinson (Jira)

[ https://issues.redhat.com/browse/JBTM-3231?page=com.atlassian.jira.plugin... ] Thomas Jenkinson commented on JBTM-3231: ---------------------------------------- There are some examples where this might not be JDK 11 specific: https://ci-master-jenkins-csb-narayana.cloud.paas.psi.redhat.com/job/nara... https://ci-master-jenkins-csb-narayana.cloud.paas.psi.redhat.com/job/nara... Although the error message is not identical > LRA recovery test fails after restart on JDK11 > ---------------------------------------------- > > Key: JBTM-3231 > URL: https://issues.redhat.com/browse/JBTM-3231 > Project: JBoss Transaction Manager > Issue Type: Bug > Components: LRA, Testing > Affects Versions: 5.10.1.Final > Reporter: Michael Musgrove > Assignee: Michael Musgrove > Priority: Major > Fix For: 5.next > > > CI testing: > LRACoordinatorTestCase=testRecovery2 fails to restart the container after crashing via byteman on JDK11. Seems to be intermittent on Amsterdam CI and more consistent on Upshift, for example: > http://narayanaci1.eng.hst.ams2.redhat.com/job/narayana/552/ > https://ci-master-jenkins-csb-narayana.cloud.paas.psi.redhat.com/job/nara... -- This message was sent by Atlassian Jira (v7.13.8#713008)

4 years, 5 months

1
0
0 / 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

jbossts-issues December 2019