[infinispan-issues] [JBoss JIRA] (ISPN-2043) Node fails to join already formed replication mode cluster

Node fails to join already formed replication mode cluster ---------------------------------------------------------- Key: ISPN-2043 URL: https://issues.jboss.org/browse/ISPN-2043 Project: Infinispan Issue Type: Bug Components: State transfer Affects Versions: 5.1.4.FINAL Reporter: Ondrej Nevelik Assignee: Ondrej Nevelik Priority: Critical During elasticity test with this timeline: start node1, REPLICATED clustering mode wait for start load data start node2 wait for cluster to form start node3 wait for cluster to form ... node3 fails to join the cluster with the following stack trace: node03: 06:58:06,731 ERROR [org.infinispan.cacheviews.CacheViewsManagerImpl] (CacheViewInstaller-3,node03/default) ISPN000172: Failed to prepare view CacheView{viewId=1, members=[node03/default]} for cache ___hotRodTopologyCache, rolling back to view CacheView{viewId=-1, members=[]}: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Cannot prepare new view CacheView{viewId=1, members=[node03/default]} on cache ___hotRodTopologyCache, we are currently preparing view CacheView{viewId=3, members=[node01/default, node02/default, node03/default]} node03: at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232) [rt.jar:1.6.0_30] node03: at java.util.concurrent.FutureTask.get(FutureTask.java:91) [rt.jar:1.6.0_30] node03: at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterPrepareView(CacheViewsManagerImpl.java:321) [infinispan-core-5.1.5.ER8-redhat-1.jar:5.1.5.ER8-redhat-1] node03: at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterInstallView(CacheViewsManagerImpl.java:249) [infinispan-core-5.1.5.ER8-redhat-1.jar:5.1.5.ER8-redhat-1] node03: at org.infinispan.cacheviews.CacheViewsManagerImpl$ViewInstallationTask.call(CacheViewsManagerImpl.java:875) [infinispan-core-5.1.5.ER8-redhat-1.jar:5.1.5.ER8-redhat-1] node03: at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_30] node03: at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_30] node03: at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30] node03: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30] node03: at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30] node03: Caused by: java.lang.IllegalStateException: Cannot prepare new view CacheView{viewId=1, members=[node03/default]} on cache ___hotRodTopologyCache, we are currently preparing view CacheView{viewId=3, members=[node01/default, node02/default, node03/default]} node03: at org.infinispan.cacheviews.CacheViewInfo.prepareView(CacheViewInfo.java:102) [infinispan-core-5.1.5.ER8-redhat-1.jar:5.1.5.ER8-redhat-1] node03: at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:481) [infinispan-core-5.1.5.ER8-redhat-1.jar:5.1.5.ER8-redhat-1] node03: at org.infinispan.cacheviews.CacheViewsManagerImpl$3.call(CacheViewsManagerImpl.java:312) [infinispan-core-5.1.5.ER8-redhat-1.jar:5.1.5.ER8-redhat-1] node03: at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_30] node03: at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_30] node03: at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30] node03: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30] node03: at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30] node03: at org.jboss.threads.JBossThread.run(JBossThread.java:122) node03: node03: 06:58:07,728 ERROR [org.infinispan.cacheviews.CacheViewsManagerImpl] (CacheViewInstaller-3,node03/default) ISPN000172: Failed to prepare view CacheView{viewId=3, members=[node03/default]} for cache ___hotRodTopologyCache, rolling back to view CacheView{viewId=2, members=[]}: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Trying to block write commands but they are already blocked for view 3 node03: at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232) [rt.jar:1.6.0_30] node03: at java.util.concurrent.FutureTask.get(FutureTask.java:91) [rt.jar:1.6.0_30] node03: at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterPrepareView(CacheViewsManagerImpl.java:321) [infinispan-core-5.1.5.ER8-redhat-1.jar:5.1.5.ER8-redhat-1] node03: at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterInstallView(CacheViewsManagerImpl.java:249) [infinispan-core-5.1.5.ER8-redhat-1.jar:5.1.5.ER8-redhat-1] node03: at org.infinispan.cacheviews.CacheViewsManagerImpl$ViewInstallationTask.call(CacheViewsManagerImpl.java:875) [infinispan-core-5.1.5.ER8-redhat-1.jar:5.1.5.ER8-redhat-1] node03: at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_30] node03: at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_30] node03: at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30] node03: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30] node03: at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30] node03: Caused by: java.lang.IllegalStateException: Trying to block write commands but they are already blocked for view 3 node03: at org.infinispan.statetransfer.StateTransferLockImpl.blockNewTransactions(StateTransferLockImpl.java:233) [infinispan-core-5.1.5.ER8-redhat-1.jar:5.1.5.ER8-redhat-1] node03: at org.infinispan.statetransfer.ReplicatedStateTransferTask.doPerformStateTransfer(ReplicatedStateTransferTask.java:84) [infinispan-core-5.1.5.ER8-redhat-1.jar:5.1.5.ER8-redhat-1] node03: at org.infinispan.statetransfer.BaseStateTransferTask.performStateTransfer(BaseStateTransferTask.java:93) [infinispan-core-5.1.5.ER8-redhat-1.jar:5.1.5.ER8-redhat-1] node03: at org.infinispan.statetransfer.BaseStateTransferManagerImpl.prepareView(BaseStateTransferManagerImpl.java:331) [infinispan-core-5.1.5.ER8-redhat-1.jar:5.1.5.ER8-redhat-1] node03: at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:485) [infinispan-core-5.1.5.ER8-redhat-1.jar:5.1.5.ER8-redhat-1] node03: at org.infinispan.cacheviews.CacheViewsManagerImpl$3.call(CacheViewsManagerImpl.java:312) [infinispan-core-5.1.5.ER8-redhat-1.jar:5.1.5.ER8-redhat-1] node03: at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_30] node03: at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_30] node03: at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30] node03: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30] node03: at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30] node03: at org.jboss.threads.JBossThread.run(JBossThread.java:122) node03: node03: 07:19:37,679 ERROR [org.infinispan.cacheviews.CacheViewInfo] (undefined) ISPN000164: Rolling back to cache view 2, but last committed view is 1 node03: 07:19:37,687 ERROR [org.infinispan.cacheviews.CacheViewInfo] (undefined) ISPN000164: Rolling back to cache view 5, but last committed view is 1 node03: 07:19:37,689 ERROR [org.infinispan.cacheviews.CacheViewInfo] (undefined) ISPN000164: Rolling back to cache view 6, but last committed view is 1 node03: 07:19:47,684 ERROR [org.infinispan.cacheviews.CacheViewInfo] (undefined) ISPN000164: Rolling back to cache view 3, but last committed view is 5