How long are you warming up the VM? As mentioned in the other thread, I've discovered that even under high load it will take more than 15 minutes before all of Infinispan's code is running in compiled mode.

On 27 Jan 2012, at 10:59, Manik Surtani wrote: On 26 Jan 2012, at 23:04, Sanne Grinovero wrote: How long are you warming up the VM? As mentioned in the other thread, I've discovered that even under high load it will take more than 15 minutes before all of Infinispan's code is running in compiled mode. We should be able to tune this though?  IIRC the JVM will take in options related to after how many loops a method gets compiled. radargun doesn't support time-based warmup, but only operation based warmup. Not hard to implement though.

> > These PUT operations are doing a GET as well, correct? I'd love to see such graphs using SKIP_REMOTE_LOOKUP. it is configured with unsafe return values. With safe return, the values might get even better…

On 27 January 2012 11:07, Mircea Markus<mircea.markus(a)jboss.com> wrote: > > On 26 Jan 2012, at 23:04, Sanne Grinovero wrote: > >> Very nice! >> All my previous tests also confirm that there is a correlation between PUT and GET performance, when one increases the other goes down. >> >> These PUT operations are doing a GET as well, correct? I'd love to see such graphs using SKIP_REMOTE_LOOKUP. > it is configured with unsafe return values. With safe return, the values might get even better... >> How long are you warming up the VM? As mentioned in the other thread, I've discovered that even under high load it will take more than 15 minutes before all of Infinispan's code is running in compiled mode. > The warmup is 100k operations, doesn't seem too much. I'm now experimenting with -XX:CompileThreshold=10 , and it's fairly warmed up only after 100k Write operations and a million read operations. And that's all in the same VM! Maybe you could try RadarGun making sure that each VM runs at least a million operations in the warmup phase? Maybe it doesn't matter at all, but I'd measure rather than guess it. Also your test is different than mine; maybe a better strategy is to figure out what's your correct warmup by looking at the output of -XX:+PrintCompilation, and see how long it takes before it's relatively quiet.

On 27 Jan 2012, at 11:36, Bela Ban wrote: > 10 is an awfully small value; in my experience (I had a default of 100 > for my JGroups perf tests), this made the tests longer than with the > default (which is 10000 IIRC) ! default is 1500: http://java.sun.com/docs/books/performance/1st_edition/html/JPAppHotspot.... _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

My experiments where using the default JVM settings regarding compile settings, with these others: -Xmx2G -Xms2G -XX:MaxPermSize=128M -XX:+HeapDumpOnOutOfMemoryError -Xss512k -XX:HeapDumpPath=/tmp/java_heap -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1 -Dlog4j.configuration=file:/opt/log4j.xml -XX:+PrintCompilation -Xbatch -server -XX:+UseCompressedOops -XX:+UseLargePages -XX:LargePageSizeInBytes=2m -XX:+AlwaysPreTouch And it's with these options that after 30 minutes of full-stress it's still not finished warming up all of Infinispan+JGroups code. After that I stated I was going to *experiment* with -XX:CompileThreshold=10", to see if I could get it to compile in a shorter time, just to save me from waiting too long in performance tests. It doesn't seem to matter much, so I'm reverting it back to the above values (default for this parameter for my VM is 10000).

On 27 January 2012 11:07, Mircea Markus <mircea.markus(a)jboss.com&gt; wrote: > > On 26 Jan 2012, at 23:04, Sanne Grinovero wrote: > >> Very nice! >> All my previous tests also confirm that there is a correlation between PUT and GET performance, when one increases the other goes down. >> >> These PUT operations are doing a GET as well, correct? I'd love to see such graphs using SKIP_REMOTE_LOOKUP. > it is configured with unsafe return values. With safe return, the values might get even better... >> How long are you warming up the VM? As mentioned in the other thread, I've discovered that even under high load it will take more than 15 minutes before all of Infinispan's code is running in compiled mode. > The warmup is 100k operations, doesn't seem too much. I'm now experimenting with -XX:CompileThreshold=10 , and it's fairly warmed up only after 100k Write operations and a million read operations. And that's all in the same VM!

Maybe you could try RadarGun making sure that each VM runs at least a million operations in the warmup phase? Maybe it doesn't matter at all, but I'd measure rather than guess it. Also your test is different than mine; maybe a better strategy is to figure out what's your correct warmup by looking at the output of -XX:+PrintCompilation, and see how long it takes before it's relatively quiet.

On 1/26/12 6:58 PM, Mircea Markus wrote: > Hi, > > As discussed on another thread, I've hacked the remote get to only go to a > single node. The performance of put almost doubled, whilst the one of gets > decreased by about 10 percent. > The performance decrease for gets is understandable, but I can't really > understand why the write got so very fast. The OOB thread pool is not shared > between the writes and the reads... Are both reads and writes marked as OOB ? Then they share the same OOB thread pool !

> That makes me think that there might be a contention point between OOB and regular threads? Both regular and OOB message are added to the *same* receive window on the receiver side, so there are definitely contention points between them... -- Bela Ban Lead JGroups (http://www.jgroups.org) JBoss / Red Hat _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev