[infinispan-issues] [JBoss JIRA] (ISPN-2970) TopologyAwareConsistentHashFactory.rebalance can fail with an IndexOutOfBoundsException

[Dan Berindei (JIRA)]

    [ https://issues.jboss.org/browse/ISPN-2970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12763422#comment-12763422 ] 

Dan Berindei edited comment on ISPN-2970 at 3/27/13 7:54 AM:
-------------------------------------------------------------

The ISPN-2918 fix solved this problem as well.

FTR, this is the initial CH that TopologyAwareConsistentHashFactory couldn't rebalance:

{noformat}
DefaultConsistentHash{numSegments=40, numOwners=2, members=[TestAddress#3|m3|r1|s1, TestAddress#4|m4|r1|s1, TestAddress#1|m1|r1|s1]}
routing table: 0: 0 1, 1: 0 1, 2: 0 1, 3: 0 1, 4: 0 1, 5: 0 1, 6: 0 1, 7: 0 1, 8: 0 1, 9: 0 1, 10: 0, 11: 0, 12: 0, 13: 0, 14: 2 0, 15: 2 0, 16: 2 0, 17: 2 0, 18: 2 0, 19: 2 0, 20: 1 0, 21: 1 0, 22: 1 0, 23: 1 0, 24: 1 0, 25: 1 0, 26: 1 0, 27: 1 0, 28: 1 0, 29: 1 0, 30: 1, 31: 1, 32: 1, 33: 2 1, 34: 2 1, 35: 2 1, 36: 2 1, 37: 2, 38: 2, 39: 2
{noformat}
                
      was (Author: dan.berindei):
    The ISPN-2918 fix solved this problem as well.

FTR, this is the initial CH that TopologyAwareConsistentHashFactory couldn't rebalance:

{noformat}
DefaultConsistentHash{numSegments=40, numOwners=2, members=[TestAddress#3|m3|r1|s1, TestAddress#4|m4|r1|s1, TestAddress#1|m1|r1|s1]}, routing table: 0: 0 1, 1: 0 1, 2: 0 1, 3: 0 1, 4: 0 1, 5: 0 1, 6: 0 1, 7: 0 1, 8: 0 1, 9: 0 1, 10: 0, 11: 0, 12: 0, 13: 0, 14: 2 0, 15: 2 0, 16: 2 0, 17: 2 0, 18: 2 0, 19: 2 0, 20: 1 0, 21: 1 0, 22: 1 0, 23: 1 0, 24: 1 0, 25: 1 0, 26: 1 0, 27: 1 0, 28: 1 0, 29: 1 0, 30: 1, 31: 1, 32: 1, 33: 2 1, 34: 2 1, 35: 2 1, 36: 2 1, 37: 2, 38: 2, 39: 2
{noformat}
                  
> TopologyAwareConsistentHashFactory.rebalance can fail with an IndexOutOfBoundsException
> ---------------------------------------------------------------------------------------
>
>                 Key: ISPN-2970
>                 URL: https://issues.jboss.org/browse/ISPN-2970
>             Project: Infinispan
>          Issue Type: Bug
>          Components: Distributed Cache
>    Affects Versions: 5.2.0.Final
>            Reporter: Dan Berindei
>            Assignee: Dan Berindei
>             Fix For: 5.2.6.Final, 5.3.0.Alpha1
>
>
> In rare circumstances, TopologyAwareConsistentHashFactory can fail to allocate numOwners owners to each segment, resulting in an IndexOutOfBoundsException:
> {code}
> 09:41:23,009 ERROR [org.infinispan.topology.ClusterTopologyManagerImpl] (undefined) Failed to start rebalance: Index: 1, Size: 1: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
> 	at java.util.ArrayList.RangeCheck(ArrayList.java:547) [rt.jar:1.6.0_43]
> 	at java.util.ArrayList.get(ArrayList.java:322) [rt.jar:1.6.0_43]
> 	at org.infinispan.distribution.ch.TopologyAwareConsistentHashFactory.doReplaceBackupOwnersSameMachine(TopologyAwareConsistentHashFactory.java:149) [infinispan-core-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1]
> 	at org.infinispan.distribution.ch.TopologyAwareConsistentHashFactory.replaceBackupOwnerForMachineLevel(TopologyAwareConsistentHashFactory.java:131) [infinispan-core-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1]
> 	at org.infinispan.distribution.ch.TopologyAwareConsistentHashFactory.addBackupOwners(TopologyAwareConsistentHashFactory.java:67) [infinispan-core-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1]
> 	at org.infinispan.distribution.ch.DefaultConsistentHashFactory.rebalanceBuilder(DefaultConsistentHashFactory.java:125) [infinispan-core-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1]
> 	at org.infinispan.distribution.ch.DefaultConsistentHashFactory.rebalance(DefaultConsistentHashFactory.java:106) [infinispan-core-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1]
> 	at org.infinispan.distribution.ch.DefaultConsistentHashFactory.rebalance(DefaultConsistentHashFactory.java:45) [infinispan-core-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1]
> 	at org.infinispan.topology.ClusterTopologyManagerImpl.startRebalance(ClusterTopologyManagerImpl.java:370) [infinispan-core-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1]
> 	at org.infinispan.topology.ClusterTopologyManagerImpl.access$000(ClusterTopologyManagerImpl.java:66) [infinispan-core-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1]
> 	at org.infinispan.topology.ClusterTopologyManagerImpl$1.call(ClusterTopologyManagerImpl.java:128) [infinispan-core-5.2.4.Final-redhat-1.jar:5.2.4.Final-redhat-1]
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_43]
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_43]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_43]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_43]
> 	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_43]
> 	at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.0.0.GA-redhat-2.jar:2.0.0.GA-redhat-2]
> {code}
> This can happen only after a node leaves. When it happens, the rebalance is stopped and the some keys will remain with less than numOwners owners. If another node leaves, some keys may be lost. If another node joins, the next rebalance will succeed and the keys will be redistributed properly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira