Hi all, I'm interested in the performance regression, and since I saw a similar regression with JGroups 3.4.0 coupled with ISPN 5.2.x (until 6.0.x is ready), I decided to investigate: Here are some profiling snapshots that I shared with Pedro. This is of my system running UDP on a 12 node DIST cluster https://dl.dropboxusercontent.com/u/50401510/pruvio/dg1_320_interpreted.jps https://dl.dropboxusercontent.com/u/50401510/pruvio/dg1_340_interpreted2.jps The big thing we noticed was that the building of a Message was now in the main UDP receive() thread instead of the thread pools. To my poor brain (and since I haven't been involved in the latest ISPN/JGroups happenings), it appeared this would be the most likely culprit. So I reworked the UDP receiver to be a bit more like 3.2.x with the Message construction done in the appropriate thread pool (main, OOB, internal), while quasi-punting on the new bundler implementation. After 3 test runs, the average throughput (again, measured by my app) seems to have recaptured 50-80% of the performance degradation. The variance is high enough, though, to give me pause that I'm on the right track. Take this FWIW. Thanks. Erik -----Original Message----- From: infinispan-dev-bounces(a)lists.jboss.org [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Mircea Markus Sent: Wednesday, October 09, 2013 3:35 PM To: infinispan -Dev List Subject: [infinispan-dev] The windup of 6.0.0 Hi guys, - 6.0.0.CR2 was added for 16 Oct (Adrian) and 6.0.0.Final was moved to 23 Oct (Dan) - we have some 20% performance regressions we need to look at before going final - I've updated JIRA: - added tasks for creating documentation and quickstarts - some JIRAs were moved here - please follow the JIRA or let me know if there's anything missing: http://goo.gl/y4Ky7t Cheers, -- Mircea Markus Infinispan lead (www.infinispan.org) _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Hi all, I'm interested in the performance regression,

and since I saw a similar regression with JGroups 3.4.0 coupled with ISPN 5.2.x (until 6.0.x is ready), I decided to investigate: Here are some profiling snapshots that I shared with Pedro. This is of my system running UDP on a 12 node DIST cluster https://dl.dropboxusercontent.com/u/50401510/pruvio/dg1_320_interpreted.jps https://dl.dropboxusercontent.com/u/50401510/pruvio/dg1_340_interpreted2.jps The big thing we noticed was that the building of a Message was now in the main UDP receive() thread instead of the thread pools.

To my poor brain (and since I haven't been involved in the latest ISPN/JGroups happenings), it appeared this would be the most likely culprit.

So I reworked the UDP receiver to be a bit more like 3.2.x with the Message construction done in the appropriate thread pool (main, OOB, internal), while quasi-punting on the new bundler implementation. After 3 test runs, the average throughput (again, measured by my app) seems to have recaptured 50-80% of the performance degradation. The variance is high enough, though, to give me pause that I'm on the right track.

Take this FWIW. Thanks. Erik -----Original Message----- From: infinispan-dev-bounces(a)lists.jboss.org [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Mircea Markus Sent: Wednesday, October 09, 2013 3:35 PM To: infinispan -Dev List Subject: [infinispan-dev] The windup of 6.0.0 Hi guys, - 6.0.0.CR2 was added for 16 Oct (Adrian) and 6.0.0.Final was moved to 23 Oct (Dan) - we have some 20% performance regressions we need to look at before going final - I've updated JIRA: - added tasks for creating documentation and quickstarts - some JIRAs were moved here - please follow the JIRA or let me know if there's anything missing: http://goo.gl/y4Ky7t Cheers,

Hi all, I grabbed some sampling (2ms) profiling runs (5.2.x on JGroups 3.4.x) for review. https://dl.dropboxusercontent.com/u/50401510/profiling/jgroups_340_final2.jp s https://dl.dropboxusercontent.com/u/50401510/profiling/jgroups_340_test2.jps The "test" is my hacked-up branch. It's pretty consistent -- my branch is faster (~10%) with performance in my app, but it uses more CPU. But I'm still unsure this is the performance degradation culprit, especially since I haven't been working with 6.0. Erik -----Original Message----- From: infinispan-dev-bounces(a)lists.jboss.org [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Bela Ban Sent: Wednesday, October 16, 2013 9:50 AM To: infinispan-dev(a)lists.jboss.org Subject: Re: [infinispan-dev] The windup of 6.0.0 Hi Erik, On 10/11/13 3:30 AM, Erik Salter wrote: > Hi all, > > I'm interested in the performance regression, Which perf regression ? JGRP-1716 ? The diff between using DONT_BUNDLE and not using it ? > and since I saw a similar > regression with JGroups 3.4.0 coupled with ISPN 5.2.x (until 6.0.x is > ready), I decided to investigate: > > Here are some profiling snapshots that I shared with Pedro. This is > of my system running UDP on a 12 node DIST cluster > > https://dl.dropboxusercontent.com/u/50401510/pruvio/dg1_320_interprete > d.jps > https://dl.dropboxusercontent.com/u/50401510/pruvio/dg1_340_interprete > d2.jps > > The big thing we noticed was that the building of a Message was now in > the main UDP receive() thread instead of the thread pools. How do you connect this to the regression; and what makes you conclude that moving the parsing up front is the culprit ? There were many changes between 3.2 and 3.3/3.4, so why did you pick this one as culprit ? Looking at your second snapshot, TP.receive() (which does the parsing) takes almost no time. I've actualy confirmed in my own (extensive) tests that moving the parsing up made things a bit faster and the code a lot simpler. It was the latter argument which made me move the parsing from the pools to receive(). > To my poor brain > (and since I haven't been involved in the latest ISPN/JGroups > happenings), it appeared this would be the most likely culprit. Again, why ? JProfiler (2nd snapshot) shows 4'350ms total for the receive() method, this is a blip on the radar (0% overall)... > So I reworked the UDP > receiver to be a bit more like 3.2.x with the Message construction done in > the appropriate thread pool (main, OOB, internal), while quasi-punting on > the new bundler implementation. After 3 test runs, the average throughput > (again, measured by my app) seems to have recaptured 50-80% of the > performance degradation. The variance is high enough, though, to give me > pause that I'm on the right track. Perhaps it is something else. But I'd like to learn what the actual problem is before diving into it... :-) Time for another skype/webex session? > Take this FWIW. Thanks. > > Erik > > -----Original Message----- > From: infinispan-dev-bounces(a)lists.jboss.org > [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Mircea Markus > Sent: Wednesday, October 09, 2013 3:35 PM > To: infinispan -Dev List > Subject: [infinispan-dev] The windup of 6.0.0 > > Hi guys, > > - 6.0.0.CR2 was added for 16 Oct (Adrian) and 6.0.0.Final was moved to 23 > Oct (Dan) > - we have some 20% performance regressions we need to look at before going > final > - I've updated JIRA: > - added tasks for creating documentation and quickstarts > - some JIRAs were moved here > - please follow the JIRA or let me know if there's anything missing: > http://goo.gl/y4Ky7t > > Cheers, >

On 10/17/2013 05:26 AM, Bela Ban wrote: > The other thing to look at is the apparent cost of Thread.setName(): if > we cannot avoid thread many creations, we could experiment with *not* > naming threads, although this is IMO not a good idea. I agree that not naming the threads is a bad idea. The thread names are vital for debugging -- both in log messages and in thread dumps. Not naming the threads would lose a whole lot of information. -Dennis _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 10/17/2013 05:26 AM, Bela Ban wrote: > The other thing to look at is the apparent cost of Thread.setName(): if > we cannot avoid thread many creations, we could experiment with *not* > naming threads, although this is IMO not a good idea. I agree that not naming the threads is a bad idea. The thread names are vital for debugging -- both in log messages and in thread dumps. Not naming the threads would lose a whole lot of information.