[jbossts-issues] [JBoss JIRA] (JBTM-1453) Reinstate/review the numbers to detect performance regressions

[Michael Musgrove (JIRA)]

    [ https://issues.jboss.org/browse/JBTM-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12978413#comment-12978413 ] 

Michael Musgrove edited comment on JBTM-1453 at 6/22/14 7:35 PM:
-----------------------------------------------------------------

I've fixed the "original requirement as stated in the JIRA" and set up a regular job for detecting regressions: view/Narayana+BlackTie/job/narayana-performance.
 
The job is tied to haverstraw and runs as part of the QA test suite. A test passes if it completes within a 10% variance from the previous successful run.

TODO: Tom's extra requirement (RTS and XTS) involves new work. I have some tests for RTS that I can probably integrate into the test suite without too much trouble. I will discuss how to add the XTS requirement with Gytis.

TODO: The job includes axes for JTS testing with JacORB and SunOrb but those tests currently do not report perf regressions  via org.jboss.jbossts.qa.Utils.PerformanceProfileStore#checkVariance(String performanceName, float operationDuration)) so I have commented those axes for now.


was (Author: mmusgrov):
I've fixed the "original requirement as stated in the JIRA" and set up a regular job for detecting regressions: view/Narayana+BlackTie/job/narayana-performance.
 
The job is tied to haverstraw and runs as part of the QA test suite. A test passes if it completes within a 10% variance from the previous successful run.

TODO: I also used Toms job config, JBTM-1453, for obtaining JTS figures using JacORB and SunOrb but the test suite does not store benchmarks for later comparison with previous jobs and failing the build if the variance exceeds the configurable threshold (controlled by the system property "org.jboss.jbossts.qa.Utils.PerformanceVariance"). NB success/failure is tested via org.jboss.jbossts.qa.Utils.PerformanceProfileStore#checkVariance(String performanceName, float operationDuration)

TODO: Tom's extra requirement (RTS and XTS) involves new work. I have some tests for RTS that I can probably integrate into the test suite without too much trouble. I will discuss how to add the XTS requirement with Gytis.

> Reinstate/review the numbers to detect performance regressions
> --------------------------------------------------------------
>
>                 Key: JBTM-1453
>                 URL: https://issues.jboss.org/browse/JBTM-1453
>             Project: JBoss Transaction Manager
>          Issue Type: Task
>      Security Level: Public(Everyone can see) 
>          Components: Performance Testing
>            Reporter: Tom Jenkinson
>            Assignee: Michael Musgrove
>             Fix For: 5.0.3
>
>   Original Estimate: 3 days
>  Remaining Estimate: 3 days
>




--
This message was sent by Atlassian JIRA
(v6.2.6#6264)