CoordinatorStopTest.testCoordinatorLeaving random failures ---------------------------------------------------------- Key: ISPN-9940 URL: https://issues.jboss.org/browse/ISPN-9940 Project: Infinispan Issue Type: Bug Reporter: Dan Berindei Priority: Major Labels: testsuite_stability Attachments: short_ISPN-9588_reproducer_20190206-1100_CoordinatorStopTest-infinispan-core.log There appear to be multiple failure modes: {noformat} 11:32:17,920 ERROR (jgroups-4,NodeB-49595:[]) [DelayedViewJGroupsTransport] Timed out waiting for view to be unblocked: [NodeB-49595|3] (2) [NodeB-49595, NodeC-2725] 11:32:17,925 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.scattered.statetransfer.CoordinatorStopTest.testCoordinatorLeaves[bias=NEVER] org.infinispan.test.TestException: java.util.concurrent.TimeoutException at org.infinispan.util.ControlledRpcManager.uncheckedGet(ControlledRpcManager.java:257) ~[test-classes/:?] at org.infinispan.util.ControlledRpcManager$SentRequest.expectAllResponses(ControlledRpcManager.java:580) ~[test-classes/:?] at org.infinispan.util.ControlledRpcManager$SentRequest.receiveAll(ControlledRpcManager.java:596) ~[test-classes/:?] at org.infinispan.scattered.statetransfer.CoordinatorStopTest.testCoordinatorLeaves(CoordinatorStopTest.java:162) ~[test-classes/:?] {noformat} {noformat} org.infinispan.util.concurrent.TimeoutException: Timed out waiting for a response for StateRequestCommand{cache=___defaultcache, origin=CoordinatorStopTest-NodeC-51924, type=CONFIRM_REVOKED_SEGMENTS, topologyId=9, segments={0}} at org.infinispan.util.ControlledRpcManager$InternalRequest.peekResponse(ControlledRpcManager.java:334) at org.infinispan.util.ControlledRpcManager$SentRequest.receiveAll(ControlledRpcManager.java:610) at org.infinispan.scattered.statetransfer.CoordinatorStopTest.testCoordinatorLeaves(CoordinatorStopTest.java:161) at org.infinispan.commons.test.TestNGLongTestsHook.run(TestNGLongTestsHook.java:24) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Suppressed: org.infinispan.util.logging.TraceException at org.infinispan.remoting.rpc.RpcManagerImpl.blocking(RpcManagerImpl.java:277) at org.infinispan.util.AbstractDelegatingRpcManager.blocking(AbstractDelegatingRpcManager.java:112) at org.infinispan.scattered.impl.ScatteredStateConsumerImpl.handleSegments(ScatteredStateConsumerImpl.java:159) at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:364) at org.infinispan.scattered.impl.ScatteredStateConsumerImpl.onTopologyUpdate(ScatteredStateConsumerImpl.java:101) at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:198) at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:57) at org.infinispan.statetransfer.StateTransferManagerImpl$1.rebalance(StateTransferManagerImpl.java:116) at org.infinispan.topology.LocalTopologyManagerImpl.doHandleRebalance(LocalTopologyManagerImpl.java:519) at org.infinispan.topology.LocalTopologyManagerImpl.lambda$handleRebalance$3(LocalTopologyManagerImpl.java:477) {noformat} Initially I thought it was related to ISPN-9588 and JGRP-2293, but the failure seems to be related to the way the test blocks the view installation and sometimes does not unblock it.