[infinispan-dev] Re: JProfiler snapshots for Infinispan+JBMAR

[Galder Zamarreno]

Galder Zamarreno wrote:
> I've managed to get tests run making sure that the right JBMAR versions 
> and the clusters are formed correctly.
> 
> JBMAR looks faster is most cases of async communications already. It's 
> worth noting that with sync, the differences are minimal, network and 
> waiting for responses is probably a bigger latency than what u can gain 
> with marshalling.
> 
> I'll run another batch of async tests to see whether it's clearer which 
> JBMAR version I should go for.
> 
> Note that in async mode and 8 nodes, I started to see some view changes 
> under load and also some NAKACK errors.
> 
> I'll move now to implement pooling and improvement explained in 
> https://jira.jboss.org/jira/browse/ISPN-59.

Manik, once I've done these two, I'll have a look at your two marshaller 
related emails and will port whatever is necessary to the JBMAR.

> 
> Regards,
> 
> Galder Zamarreno wrote:
>>
>>
>> Galder Zamarreno wrote:
>>>
>>>
>>> David M. Lloyd wrote:
>>>> OK, then I think there must be something outside of JBMAR that is 
>>>> impacting this.  I've gone through and literally removed a number of 
>>>> the top hotspots from the profiler reports which should have made a 
>>>> big difference if the reports were correct.
>>>>
>>>> Is there any way I can get ahold of your whole test case?
>>>
>>> I'm a real ^%%$£%:
>>>
>>> 1.- In the cluster tests I had run, the nodes never joined and hence 
>>> never formed a cluster, so each node run independently!
>>> Todo: improve benchmark framework so that if a cluster test is being 
>>> run with multiple nodes, the cluster view does indeed have that 
>>> number of nodes!!
>>>
>>> 2.- More importantly, all along I've been ignoring 
>>> cache-products/xyz/config.sh and hence when updating JBMAR, I was 
>>> simply copying the folder, without updating config.sh. Result? All 
>>> the JBMAR tests have been running with the very 1st JBMAR version 
>>> tested.
>>
>> Todo: cache-products/xyz/config.sh should not have the directory 
>> hard-coded, it should retrieve the directory name of pwd or something 
>> similar.
>>
>>>
>>> Once again, I'm a real ^%%$£%^%*&%*£!!
>>>
>>> I'm working to fix these two issues asap and will come back with 
>>> information asap.
>>>
>>>>
>>>> - DML
>>>>
>>>> On 05/11/2009 11:06 AM, Galder Zamarreno wrote:
>>>>> Hey,
>>>>>
>>>>> Run a couple of round of tests with JBMAR trunk (r200) and please 
>>>>> find attached the results. No real improvement I'm afraid. I'll 
>>>>> look into the profile data from r200 tomorrow.
>>>>>
>>>>> I'm currently working on the pooling.
>>>>>
>>>>> Regards,
>>>>>
>>>>> David M. Lloyd wrote:
>>>>>> Galder, would you mind switching to trunk and trying that version 
>>>>>> (unofficial 1.2.0.CR1 - yes I *promise* I'll fix the versioning 
>>>>>> and get a source blob built asap!)?
>>>>>>
>>>>>> - DML
>>>>>>
>>>>>> On 05/08/2009 12:09 PM, Galder Zamarreno wrote:
>>>>>>> Profiler data with JBMAR r174
>>>>>>>
>>>>>>> Galder Zamarreno wrote:
>>>>>>>> Hi David,
>>>>>>>>
>>>>>>>> Please find attach graphs belonging to two runs that compare:
>>>>>>>>
>>>>>>>> infinispan-4.0.0(repl-sync) - home grown marshalling layer
>>>>>>>> infinispan-4.0.0(repl-sync-jbmar) - infinispan + jbmar 1.1.2
>>>>>>>> infinispan-4.0.0(repl-sync-jbmar)-rXYZ - infinispan + jbmar with 
>>>>>>>> revision
>>>>>>>>
>>>>>>>> Not sure what's the conclusion here tbh. The results of 1.1.2 
>>>>>>>> almost look opposite in each test.
>>>>>>>>
>>>>>>>> I've also attached some information from previously run 
>>>>>>>> profiling sessions with a couple of local machines we have in 
>>>>>>>> Neuchatel. I profiled the faster of the two machines.
>>>>>>>>
>>>>>>>> Actually, looking at this profiled data, these tests are for 
>>>>>>>> synchronous replicated caches but I see no traces of actually 
>>>>>>>> reading the stream, only writing to it, hmmmm.
>>>>>>>>
>>>>>>>> I'm adding Externalizers to class table and implementing 
>>>>>>>> marshaller/unmarshaller pooling as my next tasks.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> David M. Lloyd wrote:
>>>>>>>>> OK, I tried out a few things.  You might want to try 
>>>>>>>>> introducing these one at a time (i.e. update up to rev 173, 
>>>>>>>>> then 174, then 175 and see how each one does).  In particular, 
>>>>>>>>> I think 175 has just as much chance of slowing things down as 
>>>>>>>>> speeding them up - either you're getting tons of collisions in 
>>>>>>>>> the hash table or the profiler is skewing the results there 
>>>>>>>>> (maybe try filtering out 
>>>>>>>>> org.jboss.marshalling.util.IdentityIntMap and java.lang.System 
>>>>>>>>> to see if that gives a different picture).
>>>>>>>>>
>>>>>>>>> I feel pretty good about 173 and 174 though I think the 
>>>>>>>>> profiler will skew 173 unless you have that UTFUtils filter 
>>>>>>>>> installed.  If 175 slows things down (outside of the profiler), 
>>>>>>>>> let me know and I'll revert it.  None of my tests showed much 
>>>>>>>>> difference but I don't have any good benchmarks that really 
>>>>>>>>> exercise that code right now.
>>>>>>>>>
>>>>>>>>> There's a couple things left to try yet, like looking at 
>>>>>>>>> replacing ConcurrentReferenceHashMap (assuming that isn't the 
>>>>>>>>> profiler again).
>>>>>>>>>
>>>>>>>>> ------------------------------------------------------------------------ 
>>>>>>>>>
>>>>>>>>> r175 | david.lloyd at jboss.com | 2009-05-08 00:17:46 -0500 (Fri, 
>>>>>>>>> 08 May 2009) | 1 line
>>>>>>>>>
>>>>>>>>> Try a trick to decrease the liklihood of collisions
>>>>>>>>> ------------------------------------------------------------------------ 
>>>>>>>>>
>>>>>>>>> r174 | david.lloyd at jboss.com | 2009-05-08 00:04:39 -0500 (Fri, 
>>>>>>>>> 08 May 2009) | 1 line
>>>>>>>>>
>>>>>>>>> Replacement caching is not economical; the cost is one extra 
>>>>>>>>> hash table get for non-replaced objects, two hash table gets 
>>>>>>>>> (total) for replaced objects.  Removing the cache gets rid of 
>>>>>>>>> the cost for non-replaced objects, while replaced objects now 
>>>>>>>>> have to be replaced again before the single hash table hit.
>>>>>>>>> ------------------------------------------------------------------------ 
>>>>>>>>>
>>>>>>>>> r173 | david.lloyd at jboss.com | 2009-05-07 23:44:52 -0500 (Thu, 
>>>>>>>>> 07 May 2009) | 1 line
>>>>>>>>>
>>>>>>>>> JBMAR-52 - Avoid extra copy of char array (1.5 of 2)
>>>>>>>>> ------------------------------------------------------------------------ 
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> - DML
>>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------ 
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------ 
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------ 
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------ 
>>>>>
>>>>>
>>>
>>
> 
> 
> ------------------------------------------------------------------------
> 
> This body part will be downloaded on demand.

-- 
Galder Zamarreño
Sr. Software Maintenance Engineer
JBoss, a division of Red Hat