[infinispan-dev] Infinispan Large Scale support

[Galder Zamarreño]

On Apr 7, 2011, at 9:05 AM, Bela Ban wrote:

> OK, so the question is whether FLUSH removal causes problems with 
> replication or distribution, or both ?

FLUSH removal causes issues with state transfer in replication as shown in https://issues.jboss.org/browse/ISPN-937

And I 'suspect' that this is the same for dist rehashing cos both work on similar concepts. I haven't investigated the issue deeply to ¡00% guarantee this.

> 
> David got 420+ nodes with replication, and FLUSH removed. However, with 
> distribution and FLUSH removed, it failed. Are you suggesting that FLUSH 
> is (currently) required only for *distribution* mode ?

FLUSH is required until we've confirmed that it everything works fine without it.

> 
> David, maybe worth a try to add <FLUSH timeout="3000".../> to 
> udp-largecluster.xml and try to start the cluster with mode=distribution...
> Cheers,
> 
> 
> On 4/6/11 8:44 PM, Galder Zamarreño wrote:
>> David,
>> 
>> Removing FLUSH is something that causes issues as highlighted by https://issues.jboss.org/browse/ISPN-937
>> 
>> Cheers,
>> 
>> On Apr 4, 2011, at 5:45 PM, david marion wrote:
>> 
>>> 
>>>   Good news! I was able to use the system property from ISPN-83 and remove the FLUSH from the jgroups config with 4.2.1.FINAL, and start-up times are much much better. We have a replicated cache on about 420+ nodes up in under 2 minutes. I am seeing an issue with the distributed cache though with as little as 5 nodes.
>>> 
>>>   In the coordinator log I see
>>> 
>>> org.infinispan.distribution.DistributionmanagerImpl: Detected a view change. Member list changed.......
>>> org.infinispan.distribution.DistributionmanagerImpl: This is a JOIN event! Wait for notification from new joiner<name>
>>> 
>>>   In the log from the joining node I see:
>>> 
>>> org.infinispan.distribution.JoinTask: Commencing rehash on node:<name>. Before start, distributionManager.joinComplete=false
>>> org.infinispan.distribution.JoinTask: Requesting old consistent hash from coordinator
>>> 
>>>   I jstack'd the joiner, the DefaultCacheManager.getCache() method is waiting on org.infinispan.distribution.DistributionManagerImpl.waitForJoinToComplete() and the Rehasher thread
>>>   is waiting on:
>>> 
>>> at org.infinispan.util.concurrent.ReclosableLatch.await(ReclosableLatch.java:75)
>>> at org.infinipsan.remoting.transport.jgroups.JGroupsDistSync.blockUntilNoJoinsInProgress(JGroupsDistSync.java:113)
>>> 
>>> Any thoughts?
>>> 
>>>> Date: Wed, 23 Mar 2011 15:58:19 +0100
>>>> From: bban at redhat.com
>>>> To: infinispan-dev at lists.jboss.org
>>>> Subject: Re: [infinispan-dev] Infinispan Large Scale support
>>>> 
>>>> 
>>>> 
>>>> On 3/23/11 2:39 PM, david marion wrote:
>>>>> 
>>>>> Bela,
>>>>> 
>>>>> Is there a way to start up the JGroups stack on every node without using Infinispan?
>>>> 
>>>> 
>>>> You could use ViewDemo [1] or Draw. Or write your own small test
>>>> program; if you take a look at ViewDemo's src, you'll see that it's onyl
>>>> a page of code.
>>>> 
>>>> 
>>>>> Is there some functional test that I can run or something? I know I can't remove the FLUSH from Infinispan until 5.0.0 and I don't know if I can upgrade the underlying
>>>>> JGroups jar.
>>>> 
>>>> 
>>>> I suggest test with the latest JGroups (2.12.0) and +FLUSH and -FLUSH.
>>>> The +FLUSH config should be less painful now, with the introduction of
>>>> view bundling: we need to run flush fewer times than before.
>>>> 
>>>> 
>>>> [1] http://community.jboss.org/wiki/TestingJBoss
>>>> 
>>>> --
>>>> Bela Ban
>>>> Lead JGroups / Clustering Team
>>>> JBoss
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> infinispan-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> 
>> --
>> Galder Zamarreño
>> Sr. Software Engineer
>> Infinispan, JBoss Cache
>> 
>> 
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 
> -- 
> Bela Ban
> Lead JGroups / Clustering Team
> JBoss
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache