[ https://issues.jboss.org/browse/JBTM-1453?page=com.atlassian.jira.plugin.... ] Michael Musgrove edited comment on JBTM-1453 at 6/22/14 7:35 PM: ----------------------------------------------------------------- I've fixed the "original requirement as stated in the JIRA" and set up a regular job for detecting regressions: view/Narayana+BlackTie/job/narayana-performance. The job is tied to haverstraw and runs as part of the QA test suite. A test passes if it completes within a 10% variance from the previous successful run. TODO: Tom's extra requirement (RTS and XTS) involves new work. I have some tests for RTS that I can probably integrate into the test suite without too much trouble. I will discuss how to add the XTS requirement with Gytis. TODO: The job includes axes for JTS testing with JacORB and SunOrb but those tests currently do not report perf regressions via org.jboss.jbossts.qa.Utils.PerformanceProfileStore#checkVariance(String performanceName, float operationDuration)) so I have commented those axes for now. was (Author: mmusgrov): I've fixed the "original requirement as stated in the JIRA" and set up a regular job for detecting regressions: view/Narayana+BlackTie/job/narayana-performance. The job is tied to haverstraw and runs as part of the QA test suite. A test passes if it completes within a 10% variance from the previous successful run. TODO: I also used Toms job config, JBTM-1453, for obtaining JTS figures using JacORB and SunOrb but the test suite does not store benchmarks for later comparison with previous jobs and failing the build if the variance exceeds the configurable threshold (controlled by the system property "org.jboss.jbossts.qa.Utils.PerformanceVariance"). NB success/failure is tested via org.jboss.jbossts.qa.Utils.PerformanceProfileStore#checkVariance(String performanceName, float operationDuration) TODO: Tom's extra requirement (RTS and XTS) involves new work. I have some tests for RTS that I can probably integrate into the test suite without too much trouble. I will discuss how to add the XTS requirement with Gytis. -- This message was sent by Atlassian JIRA (v6.2.6#6264)