[infinispan-dev] The windup of 6.0.0

[Bela Ban]

OK, so the first snapshot shows 18 secs for TP.receive() while the 
second lists 22 seconds. In both cases, the bulk of the time is taken up 
by thread creation and naming. Thread.setName() is 90% of the thread 
creation.

This shows that the parsing of the message is insignificant, whereas the 
passing of the message to the thread pool takes time (although not much 
as we're still only at about 4%).

The question is why do we create so many threads ? Is this possibly a 
thread pool config issue where we have a thread idle time which is too 
low ? Ideally, we should see only max_threads thread creation if we set 
idle time accordingly.

The other thing to look at is the apparent cost of Thread.setName(): if 
we cannot avoid thread many creations, we could experiment with *not* 
naming threads, although this is IMO not a good idea.

Note that the thread naming is done in DefaultThreadFactory.renameThread().

On 10/16/13 11:53 PM, Erik Salter wrote:
> Hi all,
>
> I grabbed some sampling (2ms) profiling runs (5.2.x on JGroups 3.4.x) for
> review.
>
> https://dl.dropboxusercontent.com/u/50401510/profiling/jgroups_340_final2.jp
> s
> https://dl.dropboxusercontent.com/u/50401510/profiling/jgroups_340_test2.jps
>
> The "test" is my hacked-up branch.
>
> It's pretty consistent -- my branch is faster (~10%) with performance in my
> app, but it uses more CPU.  But I'm still unsure this is the performance
> degradation culprit, especially since I haven't been working with 6.0.
>
> Erik
>
> -----Original Message-----
> From: infinispan-dev-bounces at lists.jboss.org
> [mailto:infinispan-dev-bounces at lists.jboss.org] On Behalf Of Bela Ban
> Sent: Wednesday, October 16, 2013 9:50 AM
> To: infinispan-dev at lists.jboss.org
> Subject: Re: [infinispan-dev] The windup of 6.0.0
>
> Hi Erik,
>
> On 10/11/13 3:30 AM, Erik Salter wrote:
>> Hi all,
>>
>> I'm interested in the performance regression,
>
> Which perf regression ? JGRP-1716 ? The diff between using DONT_BUNDLE and
> not using it ?
>
>> and since I saw a similar
>> regression with JGroups 3.4.0 coupled with ISPN 5.2.x (until 6.0.x is
>> ready), I decided to investigate:
>>
>> Here are some profiling snapshots that I shared with Pedro.  This is
>> of my system running UDP on a 12 node DIST cluster
>>
>> https://dl.dropboxusercontent.com/u/50401510/pruvio/dg1_320_interprete
>> d.jps
>> https://dl.dropboxusercontent.com/u/50401510/pruvio/dg1_340_interprete
>> d2.jps
>>
>> The big thing we noticed was that the building of a Message was now in
>> the main UDP receive() thread instead of the thread pools.
>
>
> How do you connect this to the regression; and what makes you conclude that
> moving the parsing up front is the culprit ? There were many changes between
> 3.2 and 3.3/3.4, so why did you pick this one as culprit ?
>
> Looking at your second snapshot, TP.receive() (which does the parsing) takes
> almost no time. I've actualy confirmed in my own (extensive) tests that
> moving the parsing up made things a bit faster and the code a lot simpler.
> It was the latter argument which made me move the parsing from the pools to
> receive().
>
>
>> To my poor brain
>> (and since I haven't been involved in the latest ISPN/JGroups
>> happenings),  it appeared this would be the most likely culprit.
>
>
> Again, why ? JProfiler (2nd snapshot) shows 4'350ms total for the
> receive() method, this is a blip on the radar (0% overall)...
>
>
>>   So I reworked the UDP
>> receiver to be a bit more like 3.2.x with the Message construction done in
>> the appropriate thread pool (main, OOB, internal), while quasi-punting on
>> the new bundler implementation.  After 3 test runs, the average throughput
>> (again, measured by my app) seems to have recaptured 50-80% of the
>> performance degradation.   The variance is high enough, though, to give me
>> pause that I'm on the right track.
>
>
> Perhaps it is something else. But I'd like to learn what the actual
> problem is before diving into it... :-) Time for another skype/webex
> session?
>
>
>> Take this FWIW.  Thanks.
>>
>> Erik
>>
>> -----Original Message-----
>> From: infinispan-dev-bounces at lists.jboss.org
>> [mailto:infinispan-dev-bounces at lists.jboss.org] On Behalf Of Mircea Markus
>> Sent: Wednesday, October 09, 2013 3:35 PM
>> To: infinispan -Dev List
>> Subject: [infinispan-dev] The windup of 6.0.0
>>
>> Hi guys,
>>
>> - 6.0.0.CR2 was added for 16 Oct (Adrian) and 6.0.0.Final was moved to 23
>> Oct (Dan)
>> - we have some 20% performance regressions we need to look at before going
>> final
>> - I've updated JIRA:
>>       - added tasks for creating documentation and quickstarts
>>       - some JIRAs were moved here
>>       - please follow the JIRA or let me know if there's anything missing:
>> http://goo.gl/y4Ky7t
>>
>> Cheers,
>>
>

-- 
Bela Ban, JGroups lead (http://www.jgroups.org)