[
https://issues.redhat.com/browse/JBTM-3137?page=com.atlassian.jira.plugin...
]
Mayank Kunwar commented on JBTM-3137:
-------------------------------------
For this issue I made two changes:
1. There is a article for making the benchmark more effective and using JMH in proper
manner (best practices) so that we could get more accurate performance. Here is the
article link
https://www.researchgate.net/publication/333825812_What's_Wrong_With_...
Under this article, the author has provided a plugin for finding JMH related bugs. So we
have to import this plugin on top of Spotbug plugin in IDE.
So, once I imported JMH spotbug plugin and applied it to the performance repo, it was
reporting some bugs related to JMH usage. We were not using JMH effectively. So I fixed
all these bugs, reported by the mentioned plugin.
Under the article that I mentioned above, it says that we have to restrict JVM
optimization by using Blackhole (from JMH), so that we could evaluate performance of the
Tests more effectively.
2. I did an experiment with a simple program, in order to evaluate more appropriate JMH
argument, which will provide us with more accurate performance results. Maybe these values
can be tuned, further more effectively. I think tuning these values is a tricky part.
Here is a simple program.
@Benchmark
public boolean evaluateJMHParam(Blackhole bh) {
for(int i=0; i<10;i++){
bh.consume(Math.log(i));
}
return true;
}
After iterating the JMH arguments, five times each, for different combinations of JMH
arguments, got to a conclusion to use *JMH Arguments* as : -t $i -r 30 -f 3 -wi 5 -i 5
t - thread count
r - measurement run time
f - fork
wi - warmup iterations
i - iterations
In the above JMH arguments: the $i that is thread count, that will vary. Values for $i
will be [1, 24, 240, 1600]
Too much variance in the performance PR checker
-----------------------------------------------
Key: JBTM-3137
URL:
https://issues.redhat.com/browse/JBTM-3137
Project: JBoss Transaction Manager
Issue Type: Bug
Components: Performance Testing
Affects Versions: 5.9.5.Final
Reporter: Michael Musgrove
Assignee: Mayank Kunwar
Priority: Major
Fix For: 5.next
The PERF axis of the PR checker job runs some tests to validate that the PR does not
introduce any regressions but the axis is not useful since there is too much variance in
the runs. For example PR
https://github.com/jbosstm/narayana/pull/1437 ran 3 times each
producing wildly different results.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)