[infinispan-dev] Infinispan Large Scale support

[Bela Ban]

On 4/4/11 5:45 PM, david marion wrote:
>
>
>    Good news! I was able to use the system property from ISPN-83 and remove the FLUSH from the jgroups config with 4.2.1.FINAL, and start-up times are much much better. We have a replicated cache on about 420+ nodes up in under 2 minutes.


Great ! Just to confirm: this is 420+ Infinispan instances, with 
replication enabled, correct ?

Did you use a specific JGroups config (e.g. udp-largecluster.xml) ?


> I am seeing an issue with the distributed cache though with as little as 5 nodes.
>
>    In the coordinator log I see
>
> org.infinispan.distribution.DistributionmanagerImpl: Detected a view change. Member list changed.......
> org.infinispan.distribution.DistributionmanagerImpl: This is a JOIN event! Wait for notification from new joiner<name>
>
>    In the log from the joining node I see:
>
> org.infinispan.distribution.JoinTask: Commencing rehash on node:<name>. Before start, distributionManager.joinComplete=false
> org.infinispan.distribution.JoinTask: Requesting old consistent hash from coordinator
>
>    I jstack'd the joiner, the DefaultCacheManager.getCache() method is waiting on org.infinispan.distribution.DistributionManagerImpl.waitForJoinToComplete() and the Rehasher thread
>    is waiting on:
>
> at org.infinispan.util.concurrent.ReclosableLatch.await(ReclosableLatch.java:75)
> at org.infinipsan.remoting.transport.jgroups.JGroupsDistSync.blockUntilNoJoinsInProgress(JGroupsDistSync.java:113)
>
> Any thoughts?


I recently took a look at the distribution code, and this part is very 
brittle with respect to parallel startup and merging. Plus, I believe 
the (blocking) RPC to fetch the old CH from the coordinator might 
deadlock in certain cases...

I've got a pull request for a push based rebalancing versus pull based 
rebalancing pending. It'll likely make it into 5.x, as a matter of fact 
I've got a chat about this this afternoon.




>> Date: Wed, 23 Mar 2011 15:58:19 +0100
>> From: bban at redhat.com
>> To: infinispan-dev at lists.jboss.org
>> Subject: Re: [infinispan-dev] Infinispan Large Scale support
>>
>>
>>
>> On 3/23/11 2:39 PM, david marion wrote:
>>>
>>> Bela,
>>>
>>> Is there a way to start up the JGroups stack on every node without using Infinispan?
>>
>>
>> You could use ViewDemo [1] or Draw. Or write your own small test
>> program; if you take a look at ViewDemo's src, you'll see that it's onyl
>> a page of code.
>>
>>
>>> Is there some functional test that I can run or something? I know I can't remove the FLUSH from Infinispan until 5.0.0 and I don't know if I can upgrade the underlying
>>> JGroups jar.
>>
>>
>> I suggest test with the latest JGroups (2.12.0) and +FLUSH and -FLUSH.
>> The +FLUSH config should be less painful now, with the introduction of
>> view bundling: we need to run flush fewer times than before.
>>
>>
>> [1] http://community.jboss.org/wiki/TestingJBoss
>>
>> --
>> Bela Ban
>> Lead JGroups / Clustering Team
>> JBoss
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>   		 	   		
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

-- 
Bela Ban
Lead JGroups / Clustering Team
JBoss