Hi Pedro ----- Original Message ----- | From: "Pedro Ruivo" <pedro(a)infinispan.org&gt; | To: infinispan-dev(a)lists.jboss.org | Sent: Wednesday, June 12, 2013 11:04:46 AM | Subject: Re: [infinispan-dev] New bundler performance | | Hi Radim, | | 1) have you tried to run your tests with the bundling disable in | jgroups? (UDP.enable_bundling=false) This attribute is deprecated and is not used anymore. Bundling cannot be switched off in configuration (as far as I know), only by setting the DONT_BUNDLE flag. I could try to disable bundling manually, I will do that. | 2) how many Stressor threads (assuming radargun) are executing | transactions in each node? Could you run with a high number of Stressor | threads like 16, 32, 64 and 128 (with and without bundling in JGroups)? This is non-transactional, I assume you're asking about operations 5 threads -> 4300 writes 10 threads -> 4800 writes 20 threads -> 4200 writes 40 threads -> 3700 writes | | About the reads, I don't know if this make sense, but if the | communication cost is higher the threads are blocked longer and will | perform less read operations in the same time interval. The number is computed as number of reads / (time spent in the reads / number of threads). Therefore, it should not be affected by the writes in the direct way from the test (the writes should affect the result in such way only that the node is processing something more or less load in that moment and has less or more time to respond to the read request). But OK, it could be affected in some complicated fashion, let's focus on the writes. Radim | | Cheers, | Pedro Ruivo | | On 06/12/2013 09:54 AM, Radim Vansa wrote: | > Hi, | > | > I was going through the commits (running tests on each of them) to seek the | > performance regression we've recently discovered and it seems that our | > test (replicated udp non-transactional stress test on 4 nodes) experiences | > a serious regression on the commit | > | > ISPN-2848 Use the new bundling mechanism from JGroups 3.3.0 | > (73da108cdcf9db4f3edbcd6dbda6938d6e45d148) | > | > The performance drops from about 7800 writes/s to 4800 writes/s, and from | > 1.5M reads/s to 1.2M reads/s (having slower reads in replicated mode is | > really odd). | > | > It seems that the bundler is not really as good as we hoped for - it may be | > the bottleneck. I have tried to create another bundler which shares the | > queue between 4 instances of TransportQueueBundler (so, 4 threads are | > actually sending the messages which go into one queue) and the performance | > mildly improved - to 5200 writes/s, but that's not enough. | > | > Radim | > | > Note: you may have seen my conversation with Pedro Ruivo on IRC about the | > bundler several days ago, in that time our configuration had old bundler. | > This was fixed, but as I have not built Infinispan properly (something got | > cached), I have not noticed the regression between these builds. | > | > ----------------------------------------------------------- | > Radim Vansa | > Quality Assurance Engineer | > JBoss Datagrid | > tel. +420532294559 ext. 62559 | > | > Red Hat Czech, s.r.o. | > Brno, Purkyňova 99/71, PSČ 612 45 | > Czech Republic | > | > | > _______________________________________________ | > infinispan-dev mailing list | > infinispan-dev(a)lists.jboss.org | > https://lists.jboss.org/mailman/listinfo/infinispan-dev | > | _______________________________________________ | infinispan-dev mailing list | infinispan-dev(a)lists.jboss.org | https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 12 Jun 2013, at 14:16, Bela Ban <bban(a)redhat.com&gt; wrote: Right now we've removed the DONT_BUNDLE flag for the sync calls, so putting the old bundler in place would require some code changes. Is there anything wrong with using DONT_BUNDLE on the sync calls with the new bundler? I'm thinking we should support both for now, as the new bundling needs to prove itself still. +1. or some jprofiler snapshots to check the get? Cheers, -- Mircea Markus Infinispan lead (www.infinispan.org)

On 12 Jun 2013, at 15:38, Bela Ban <bban(a)redhat.com&gt; wrote: The only thing I'm aware of that's needed to support old bundling is to add DONT_BUNDLE to sync messages, which seems pretty straight forward. I remember discussing that DONT_BUNDLE + sync messages + new bundler is suboptimal for some reason (batching?) - am I wrong saying that? We're currently holding the 5.3.0.CR2 release for this. I'm thinking to: - support old bundling as well - based on the when the jgroups perf problem is fixed we might release CR3 or include the fix in 5.3.1.Final Cheers, -- Mircea Markus Infinispan lead (www.infinispan.org)

Radim, can you please give Pedro's branch a try? https://github.com/infinispan/infinispan/pull/1896 If all good we'll release CR2 from that one. On 12 Jun 2013, at 16:02, Bela Ban <bban(a)redhat.com&gt; wrote: Cheers, -- Mircea Markus Infinispan lead (www.infinispan.org)

Hi Radim, On 06/13/2013 09:24 AM, Radim Vansa wrote: during this run, have you used the old bundler in JGroups? It was not changed by me but yes I think that is ok. See ISPN-3043 (https://issues.jboss.org/browse/ISPN-3043) Cheers, Pedro ps. it's holiday in Lisbon so I'll not be online all day.

| > Seems that it's somewhat better (6050 writes/s) but not at the previous | > level. | what about the reads? After Pedro's question I have tried to run ISPN-3221 fix with both old an new bundler (in configuration), and the result pretty surprises me as there should not be much of a difference (as all messages that are part of the test should be OOB => DONT_BUNDLE). 113842c8cf91cbb5a1bbd26e05fab7024fdec081 (last OK build) new bundler: 1.5M reads/s, 7800 writes/s old bundler: 1.3M reads/s, 4300 writes/s 73da108cdcf9db4f3edbcd6dbda6938d6e45d148 (removing DONT_BUNDLE) new bundler: 1.2M reads/s, 4800 writes/s old bundler: 150k reads/s, 250 writes/s ISPN-3221 fix (Pedro's branch) new bundler: 1.5M reads/s, 6050 writes/s old bundler: 1.3M reads/s, 3100 writes/s ISPN-3221 cherry-picked onto 73da108cdcf9db4f3edbcd6dbda6938d6e45d148 new bundler: 1.5M reads/s, 7800 writes/s old bundler: 1.3M reads/s, 4300 writes/s Radim | > I am trying to cherry-pick it directly on the commit that failed it and see | > where is the difference, then maybe I'll have to generate alternative | > branch with ISPN-3221 commit after ISPN-2848 | > 73da108cdcf9db4f3edbcd6dbda6938d6e45d148 and iterate to next regression | > origin. | +1. | > | > Pedro: I've noticed that in JGroupsTransport:577 you're changing the third | > parameter of CommandAwareRpcDispatcher.constructMessage from false (as | > found in 113842c8cf91cbb5a1bbd26e05fab7024fdec081) to true (as appears in | > later commits), is this desired? | | That has been changed on purpose to be true later on when fixing: ISPN-3043: | Make synchronous backup calls use the OOB thread pool. | | Cheers, | -- | Mircea Markus | Infinispan lead (www.infinispan.org) | | | | | | _______________________________________________ | infinispan-dev mailing list | infinispan-dev(a)lists.jboss.org | https://lists.jboss.org/mailman/listinfo/infinispan-dev |

So the new bundler is always *faster* than the old one. And ISPN-3221 seems to fix the degradation. Do I still need to look into this ? On 6/13/13 12:43 PM, Radim Vansa wrote: -- Bela Ban, JGroups lead (http://www.jgroups.org)

On 7/8/13 7:32 PM, Mircea Markus wrote: That's bad, we should look into it ! I get exactly the same or even better performance with the new bundler and DONT_BUNDLE removed. That's completely counter to what I've measured; and it should be exactly the other way round ! Bela Ban, JGroups lead (http://www.jgroups.org)

On Wed, Jun 12, 2013 at 5:39 PM, Radim Vansa <rvansa(a)redhat.com&gt; wrote: Higher CPU usage for the write commands will slow down everything else on the same machine. So as long as the test does reads and writes in parallel, there will be some interference. I suggest running a read-only test and a write-only test to get a more reliable picture of the performance for each of them.