[infinispan-dev] Reproduce repl sync cache locking issue

[Mircea Markus]

Thanks for rising the two issues Bela, both are caused by the way storeAsBinary works. I think storeAsBinary is flawed as it currently is - I've sent an email on -dev to ask for feedback on that. 

RE: MFC I'm pretty sure there's a problem somewhere, as the system always deadlocks in that protocol on my machine. I'll try to narrow it down.

On 18 Jul 2013, at 08:53, Bela Ban <bban at redhat.com> wrote:

> I could not reproduce this, I ran the test *many times* with both 
> Infinispan 5.3.0 (JGroups 3.3.1 and 3.3.3) and 5.2.4 (JGroups 3.2.7), 
> both on my mac and linux box, and was *never able to reproduce the 
> blocking*.
> 
> To be honest, this is what I expected, as MFC has run in production for 
> a few years now and I have yet to receive a report on it locking up...
> 
> However, I did run into 2 Infinispan problems (ISPN 5.3.0 / JG 3.3.3), 
> probably related:
> 
> #1
> - Start nodes 1 and 2
> - Hit enter in node 1 to populate the cache, these modifications are 
> replicated to node 2
> - 100'000 elements with a total of ca. 470MB of data are added; in a 
> single node we use ca. 520MB of heap, which is fine considering there's 
> some overhead
> - However, node 1 has 1.4 *GB* of data, and using jvisualvm we can see 
> that we have *200'000* byte[] arrays instead of 100'000 !
> - Node 2 is fine, with ca. 520MB of heap used and 100'000 byte arrays
> 
> #2
> - Start node 1
> - Populate node 1 with data, 100'000 elements with a total of ca. 520MB 
> of heap
> - Start node 2
> - After the state transfer, node 2 has ca. 520MB of data, which is fine
> - However, node 1 has *1.4 GB of heap* !
> - We can see that node 1 holds *200'000* byte[] arrays instead of 100'000
> 
> 
> Mircea and I looked at this yesterday and a possible culprit could be 
> MarshalledValue, but Mircea's looking into it. We believe the root cause 
> for #1 and #2 is the same.
> 
> 
> On 7/17/13 12:46 PM, Mircea Markus wrote:
>> Thanks Ray!
>> 
>> I think the issue is: https://issues.jboss.org/browse/JGRP-1659
>> Bela the test is attached to the JIRA.
>> 
>> Ray, I think your harness can be pretty useful as a general purpose tool for reporting issues, I think it's worth cleaning it up a bit + doc and add it to the infinispan repo. Wdyt?
>> 
>> On 16 Jul 2013, at 20:11, Ray Tsang <rtsang at redhat.com> wrote:
>> 
>>> Hi All,
>>> 
>>> Please see attached test.
>>> 
>>> It generates random payloads of different sizes according to a distribution.
>>> At client, the actual payload size ranges from 10k to 2mb.  However, this test only simulates btwn 10 bytes to 50k bytes - and locking still occurs.
>>> 
>>> Do not run unit tests - those tests are for other things ;)  To run the actual test, do:
>>> 
>>> mvn -e exec:exec -Dnode=n  // where n is the node number 1 to 4 etc.
>>> 
>>> What I do is open 3 terminals/tabs/screens, whichever you prefer, each run:
>>> mvn -e exec:exec -Dnode=1
>>> mvn -e exec:exec -Dnode=2
>>> mvn -e exec:exec -Dnode=3
>>> ...
>>> 
>>> It'll prompt you to press a key when ready.  When you confirm cluster has formed, press any key to continue on all the nodes.
>>> 
>>> Thanks,
>>> 
>>> <replication-test.zip>
>> 
>> Cheers,
>> 
> 
> -- 
> Bela Ban, JGroups lead (http://www.jgroups.org)
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

Cheers,
-- 
Mircea Markus
Infinispan lead (www.infinispan.org)