[infinispan-dev] very interesting performance results

[Dan Berindei]

On Fri, Jan 27, 2012 at 3:43 PM, Mircea Markus <mircea.markus at jboss.com> wrote:
> On 27 Jan 2012, at 13:31, Sanne Grinovero wrote:
>> My experiments where using the default JVM settings regarding compile
>> settings, with these others:
>>
>> -Xmx2G -Xms2G -XX:MaxPermSize=128M -XX:+HeapDumpOnOutOfMemoryError
>> -Xss512k -XX:HeapDumpPath=/tmp/java_heap
>> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
>> -Dlog4j.configuration=file:/opt/log4j.xml -XX:+PrintCompilation
>> -Xbatch -server -XX:+UseCompressedOops -XX:+UseLargePages
>> -XX:LargePageSizeInBytes=2m -XX:+AlwaysPreTouch
>>
>> And it's with these options that after 30 minutes of full-stress it's
>> still not finished warming up all of Infinispan+JGroups code.
>> After that I stated I was going to *experiment* with
>> -XX:CompileThreshold=10", to see if I could get it to compile in a
>> shorter time, just to save me from waiting too long in performance
>> tests. It doesn't seem to matter much, so I'm reverting it back to the
>> above values (default for this parameter for my VM is 10000).
>
> That's surprising, I'd say that in 30 mins of invocations all the critical paths are touched much more many times than 10000. E.g. the number of reads/sec is in thousands (20k on the cluster lab). Might be that this param is ignored or it collides with other -XX ?
>

PrintCompilation tells you which methods are compiled, but it doesn't
tell you which methods were inlined with it. So something like this
can (and probably does) happen:

1. ConcurrentSkipListMap.doPut is compiled
2. UNICAST2.down() (assuming JGroups 3.0.3.Final) calls
AckSenderWindow.add() -> ConcurrentSkipListMap.put -> doPut. This gets
compiled, and everything is inlined in it.
2. The conditions where ConcurrentSkipListMap.put is called change
dramatically - so the initial optimizations are no longer valid.
2. Eventually the timers and everything else that's using
ConcurrentSkipListMap call put() 10000 times and
ConcurrentSkipListMap.doPut is compiled again.

AFAIK oprofile will not report any inlined methods, so if
ConcurrentSkipListMap appears in the report it means that it still
called a fair amount of times. On the other hand, oprofile also
doesn't report interpreted methods - so if it appears as a separate
method in the oprofile report than it means it is compiled.

You should also try running the test without -Xbatch, apparently it's
only good for debugging the JVM:
http://stackoverflow.com/questions/3369791/java-vm-tuning-xbatch-and-xcomp

It shouldn't change what methods get compiled, but compilation should
be less expensive without it.

Cheers
Dan