[jbossts-issues] [JBoss JIRA] (JBTM-3137) Too much variance in the performance PR checker

[Mayank Kunwar (Jira)]

    [ https://issues.redhat.com/browse/JBTM-3137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13988689#comment-13988689 ] 

Mayank Kunwar commented on JBTM-3137:
-------------------------------------


For this issue I made two changes:

1. There is a article for making the benchmark more effective and using JMH in proper manner (best practices) so that we could get more accurate performance. Here is the article link https://www.researchgate.net/publication/333825812_What's_Wrong_With_My_Benchmark_Results_Studying_Bad_Practices_in_JMH_Benchmarks

Under this article, the author has provided a plugin for finding JMH related bugs. So we have to import this plugin on top of Spotbug plugin in IDE.

So, once I imported JMH spotbug plugin and applied it to the performance repo, it was reporting some bugs related to JMH usage. We were not using JMH effectively. So I fixed all these bugs, reported by the mentioned plugin.

Under the article that I mentioned above, it says that we have to restrict JVM optimization by using Blackhole (from JMH), so that we could evaluate performance of the Tests more effectively.

2. I did an experiment with a simple program, in order to evaluate more appropriate JMH argument, which will provide us with more accurate performance results. Maybe these values can be tuned, further more effectively. I think tuning these values is a tricky part.

Here is a simple program.

@Benchmark
    public boolean evaluateJMHParam(Blackhole bh) {
        for(int i=0; i<10;i++){
            bh.consume(Math.log(i));
        }
        return true;
    }

After iterating the JMH arguments, five times each, for different combinations of JMH arguments, got to a conclusion to use *JMH Arguments* as : -t $i -r 30 -f 3 -wi 5 -i 5
t - thread count
r - measurement run time
f - fork
wi - warmup iterations
i - iterations

In the above JMH arguments: the $i that is thread count, that will vary. Values for $i will be [1, 24, 240, 1600]


> Too much variance in the performance PR checker
> -----------------------------------------------
>
>                 Key: JBTM-3137
>                 URL: https://issues.redhat.com/browse/JBTM-3137
>             Project: JBoss Transaction Manager
>          Issue Type: Bug
>          Components: Performance Testing
>    Affects Versions: 5.9.5.Final
>            Reporter: Michael Musgrove
>            Assignee: Mayank Kunwar
>            Priority: Major
>             Fix For: 5.next
>
>
> The PERF axis of the PR checker job runs some tests to validate that the PR does not introduce any regressions but the axis is not useful since there is too much variance in the runs. For example PR https://github.com/jbosstm/narayana/pull/1437 ran 3 times each producing wildly different results.



--
This message was sent by Atlassian Jira
(v7.13.8#713008)