[infinispan-issues] [JBoss JIRA] (ISPN-7996) ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus2[SCATTERED_SYNC, tx=false] random failures

[Dan Berindei (JIRA)]

    [ https://issues.jboss.org/browse/ISPN-7996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439969#comment-13439969 ] 

Dan Berindei commented on ISPN-7996:
------------------------------------

Random failures in {{RehashWithSharedStoreTest.testRehashes\[SCATTERED_SYNC, tx=false, numOwners=1, l1=false\]}} probably have the same cause:

{noformat}
11:47:55,248 ERROR (transport-thread-Test-NodeB-p150-t6:[Topology-dist]) [LocalTopologyManagerImpl] ISPN000230: Failed to start rebalance for cache dist
java.lang.IllegalStateException: Segment 71 is in state OWNED
	at org.infinispan.scattered.impl.ScatteredVersionManagerImpl.registerSegment(ScatteredVersionManagerImpl.java:248) ~[classes/:?]
	at org.infinispan.scattered.impl.ScatteredStateConsumerImpl.beforeTopologyInstalled(ScatteredStateConsumerImpl.java:125) ~[classes/:?]
	at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:336) ~[classes/:?]
	at org.infinispan.scattered.impl.ScatteredStateConsumerImpl.onTopologyUpdate(ScatteredStateConsumerImpl.java:98) ~[classes/:?]
	at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:215) ~[classes/:?]
	at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:53) ~[classes/:?]
	at org.infinispan.statetransfer.StateTransferManagerImpl$1.rebalance(StateTransferManagerImpl.java:138) ~[classes/:?]
	at org.infinispan.topology.LocalTopologyManagerImpl.doHandleRebalance(LocalTopologyManagerImpl.java:530) ~[classes/:?]
	at org.infinispan.topology.LocalTopologyManagerImpl.lambda$handleRebalance$3(LocalTopologyManagerImpl.java:490) ~[classes/:?]
	at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:144) [classes/:?]
	at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:33) [classes/:?]
	at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:174) [classes/:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
11:48:54,864 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.distribution.rehash.RehashWithSharedStoreTest.testRehashes[SCATTERED_SYNC, tx=false, numOwners=1, l1=false]
java.lang.RuntimeException: Timed out waiting for rebalancing to complete on node Test-NodeB-32389, current topology is CacheTopology{id=8, rebalanceId=5, currentCH=PartitionerConsistentHash:ScatteredConsistentHash{ns=256, rebalanced=false, owners = (2)[Test-NodeB-32389: 85, Test-NodeC-55673: 85]}, pendingCH=null, unionCH=null, phase=NO_REBALANCE, actualMembers=[Test-NodeB-32389, Test-NodeC-55673], persistentUUIDs=[c86368c2-10f2-49a0-88e8-f975a68ba0cc, da487490-24d6-4b28-a4f8-6fb7efbe0ce0]}. rebalanceInProgress=false, currentChIsBalanced=false
	at org.infinispan.test.TestingUtil.waitForNoRebalance(TestingUtil.java:383) ~[test-classes/:?]
	at org.infinispan.test.TestingUtil.waitForNoRebalance(TestingUtil.java:419) ~[test-classes/:?]
	at org.infinispan.distribution.rehash.RehashWithSharedStoreTest.testRehashes(RehashWithSharedStoreTest.java:82) ~[test-classes/:?]
{noformat}

> ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus2[SCATTERED_SYNC, tx=false] random failures
> ---------------------------------------------------------------------------------------------------
>
>                 Key: ISPN-7996
>                 URL: https://issues.jboss.org/browse/ISPN-7996
>             Project: Infinispan
>          Issue Type: Bug
>          Components: Core, Test Suite - Core
>            Reporter: Dan Berindei
>            Assignee: Radim Vansa
>            Priority: Critical
>              Labels: testsuite_stability
>         Attachments: ClusterTopologyManagerTest_master_20170628.log.gz
>
>
> The test first logs a few exceptions:
> {noformat}
> 23:16:53,126 ERROR (transport-thread-Test-NodeE-p55594-t1:[Topology-___defaultcache]) [LocalTopologyManagerImpl] ISPN000230: Failed to start rebalance for cache ___defaultcache
> java.lang.IllegalStateException: Segment 7 is in state OWNED
> 	at org.infinispan.scattered.impl.ScatteredVersionManagerImpl.registerSegment(ScatteredVersionManagerImpl.java:248) ~[classes/:?]
> 	at org.infinispan.scattered.impl.ScatteredStateConsumerImpl.beforeTopologyInstalled(ScatteredStateConsumerImpl.java:125) ~[classes/:?]
> 	at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:336) ~[classes/:?]
> 	at org.infinispan.scattered.impl.ScatteredStateConsumerImpl.onTopologyUpdate(ScatteredStateConsumerImpl.java:98) ~[classes/:?]
> 	at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:215) ~[classes/:?]
> 	at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:53) ~[classes/:?]
> 	at org.infinispan.statetransfer.StateTransferManagerImpl$1.rebalance(StateTransferManagerImpl.java:138) ~[classes/:?]
> 	at org.infinispan.topology.LocalTopologyManagerImpl.doHandleRebalance(LocalTopologyManagerImpl.java:526) ~[classes/:?]
> 	at org.infinispan.topology.LocalTopologyManagerImpl.lambda$handleRebalance$3(LocalTopologyManagerImpl.java:486) ~[classes/:?]
> 	at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:144) [classes/:?]
> {noformat}
> And then it times out:
> {noformat}
> 23:17:52,949 ERROR (testng-Test:[]) [TestingUtil] Timed out waiting for rebalancing to complete on node Test-NodeE-8881, expected member list is [Test-NodeE-8881], current member list is [Test-NodeE-8881, Test-NodeF-9273]!
> 23:17:52,949 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.statetransfer.ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus2[SCATTERED_SYNC, tx=false]
> java.lang.RuntimeException: Timed out waiting for rebalancing to complete on node Test-NodeE-8881, expected member list is [Test-NodeE-8881], current member list is [Test-NodeE-8881, Test-NodeF-9273]!
> 	at org.infinispan.test.TestingUtil.waitForNoRebalance(TestingUtil.java:383) ~[test-classes/:?]
> 	at org.infinispan.statetransfer.ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus2(ClusterTopologyManagerTest.java:433) ~[test-classes/:?]
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.2.3#72005)