org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.testAsync & testSync fail randomly on RHEL && JDK6 ------------------------------------------------------------------------------------------------------------------------------- Key: ISPN-3595 URL: https://issues.jboss.org/browse/ISPN-3595 Project: Infinispan Issue Type: Bug Components: Core API Affects Versions: 6.0.0.Beta1 Environment: RHEL5 && x64 && JDK6 Reporter: Anna Manukyan Assignee: Mircea Markus Labels: testsuite_stability The tests: org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.testAsync org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.testSync fail randomly on RHEL 5,6 - 32 and 64 bit machines with JDK6. The error message is: {code} 2013-10-01 11:05:02,877 WARN [CacheNotifierImpl] (testng-StaleLocksWithLockOnlyTxDuringStateTransferTest) ISPN000133: Attempted to register listener of class class org.infinispan.statetransfer.StateProviderImpl$$EnhancerByMockitoWithCGLIB$$1b069019, but no valid, public methods annotated with method-level event annotations found! Ignoring listener. 2013-10-01 11:05:05,306 WARN [arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff0a1059e0:ce1c:524adeb6:2fd9 in state RUN 2013-10-01 11:05:05,308 WARN [jta] (Transaction Reaper Worker 0) ARJUNA016029: SynchronizationImple.afterCompletion - failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=[StaleLocksWithCommitDuringStateTransferTest-NodeK-42948, StaleLocksWithCommitDuringStateTransferTest-NodeL-43885], isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=[MagicKey#k2{f9dc9ad7@StaleLocksWithCommitDuringStateTransferTest-NodeL-43885/39}], topologyId=2, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@32a6} org.infinispan.transaction.synchronization.SynchronizationAdapter@32c5 with exception org.infinispan.commons.CacheException: Could not commit. at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:67) at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:96) at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:402) at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:334) at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.cancel(TwoPhaseCoordinator.java:120) at com.arjuna.ats.arjuna.AtomicAction.cancel(AtomicAction.java:215) at com.arjuna.ats.arjuna.coordinator.TransactionReaper.doCancellations(TransactionReaper.java:370) at com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:78) Caused by: javax.transaction.xa.XAException at org.infinispan.transaction.TransactionCoordinator.rollback(TransactionCoordinator.java:186) at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:65) ... 7 more 2013-10-01 11:05:05,308 WARN [arjuna] (Transaction Reaper Worker 0) ARJUNA012127: TwoPhaseCoordinator.afterCompletion - returned failure for SynchronizationImple< 0:ffff0a1059e0:ce1c:524adeb6:2fda, SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=[StaleLocksWithCommitDuringStateTransferTest-NodeK-42948, StaleLocksWithCommitDuringStateTransferTest-NodeL-43885], isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=[MagicKey#k2{f9dc9ad7@StaleLocksWithCommitDuringStateTransferTest-NodeL-43885/39}], topologyId=2, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@32a6} org.infinispan.transaction.synchronization.SynchronizationAdapter@32c5 > 2013-10-01 11:05:05,309 WARN [arjuna] (Transaction Reaper Worker 0) ARJUNA012121: TransactionReaper::doCancellations worker Thread[Transaction Reaper Worker 0,5,main] successfully canceled TX 0:ffff0a1059e0:ce1c:524adeb6:2fd9 2013-10-01 11:05:12,943 ERROR [CheckPoint] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA) Timed out waiting for event allow_get_transactions_1_from_StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB-26402 * 1 (available = 0, total = 0 2013-10-01 11:05:12,943 ERROR [InboundInvocationHandlerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA) Exception executing command java.util.concurrent.TimeoutException: Timed out waiting for event allow_get_transactions_1_from_StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB-26402 at org.infinispan.test.fwk.CheckPoint.awaitStrict(CheckPoint.java:43) at org.infinispan.test.fwk.CheckPoint.awaitStrict(CheckPoint.java:33) at org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest$1.answer(StaleLocksWithLockOnlyTxDuringStateTransferTest.java:101) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:91) at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29) at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:38) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:51) at org.infinispan.statetransfer.StateProviderImpl$$EnhancerByMockitoWithCGLIB$$1b069019.getTransactionsForSegments(<generated>) at org.infinispan.statetransfer.StateRequestCommand.perform(StateRequestCommand.java:65) at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:97) at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:46) at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:169) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) 2013-10-01 11:05:12,950 WARN [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000209: Failed to retrieve transactions for segments [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 16, 19, 18, 21, 20, 23, 22, 25, 24, 27, 26, 29, 28, 31, 30, 34, 35, 32, 33, 38, 39, 36, 37, 42, 43, 40, 41, 46, 47, 44, 45, 51, 50, 49, 48, 55, 54, 53, 52, 59, 58, 57, 56] of cache testCache from node StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865 (node will not be retried) org.infinispan.remoting.RemoteException: ISPN000217: Received exception from StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865, see cause for remote stack trace at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:38) at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:63) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:534) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:288) at org.infinispan.statetransfer.StateConsumerImpl.getTransactions(StateConsumerImpl.java:686) at org.infinispan.statetransfer.StateConsumerImpl.requestTransactions(StateConsumerImpl.java:653) at org.infinispan.statetransfer.StateConsumerImpl.addTransfers(StateConsumerImpl.java:603) at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:367) at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:178) at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:38) at org.infinispan.statetransfer.StateTransferManagerImpl$1.rebalance(StateTransferManagerImpl.java:105) at org.infinispan.topology.LocalTopologyManagerImpl.handleRebalance(LocalTopologyManagerImpl.java:228) at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:155) at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:124) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$4.run(CommandAwareRpcDispatcher.java:270) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) Caused by: java.util.concurrent.TimeoutException: Timed out waiting for event allow_get_transactions_1_from_StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB-26402 at org.infinispan.test.fwk.CheckPoint.awaitStrict(CheckPoint.java:43) at org.infinispan.test.fwk.CheckPoint.awaitStrict(CheckPoint.java:33) at org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest$1.answer(StaleLocksWithLockOnlyTxDuringStateTransferTest.java:101) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34) at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:91) at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29) at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:38) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:51) at org.infinispan.statetransfer.StateProviderImpl$$EnhancerByMockitoWithCGLIB$$1b069019.getTransactionsForSegments(<generated>) at org.infinispan.statetransfer.StateRequestCommand.perform(StateRequestCommand.java:65) at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:97) at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:46) at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:169) ... 3 more 2013-10-01 11:05:12,951 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 0 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,951 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 1 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,951 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 2 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,951 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 3 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 4 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 5 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 6 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 7 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 8 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 9 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 10 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 11 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,953 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 12 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,953 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 13 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,953 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 14 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,954 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 15 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,954 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 17 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,954 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 16 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,955 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 19 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,955 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 18 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,955 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 21 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,955 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 20 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,956 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 23 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,956 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 22 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,956 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 25 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,956 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 24 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,957 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 27 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,957 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 26 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,957 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 29 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,957 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 28 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,958 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 31 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,958 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 30 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,958 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 34 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,959 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 35 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,959 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 32 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,959 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 33 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,959 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 38 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,960 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 39 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,960 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 36 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,960 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 37 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,960 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 42 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,961 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 43 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,961 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 40 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,961 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 41 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,962 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 46 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,962 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 47 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:12,999 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 44 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,000 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 45 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,000 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 51 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,001 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 50 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,001 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 49 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,001 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 48 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,002 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 55 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,002 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 54 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,002 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 53 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,002 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 52 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,003 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 59 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,003 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 58 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,003 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 57 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,004 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 56 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,004 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 0 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,004 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 1 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,005 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 2 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,005 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 3 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,005 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 4 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,005 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 5 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,006 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 6 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,006 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 7 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,006 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 8 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,007 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 9 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,007 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 10 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,007 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 11 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,008 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 12 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,008 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 13 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,008 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 14 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,008 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 15 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,009 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 17 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,009 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 16 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,009 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 19 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,010 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 18 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,010 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 21 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,010 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 20 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,010 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 23 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,011 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 22 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,011 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 25 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,011 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 24 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,012 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 27 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,012 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 26 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,012 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 29 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,012 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 28 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,013 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 31 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,013 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 30 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,013 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 34 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,014 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 35 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,014 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 32 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,014 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 33 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,014 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 38 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,015 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 39 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,015 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 36 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,015 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 37 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,016 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 42 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,016 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 43 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,016 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 40 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,016 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 41 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,017 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 46 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,017 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 47 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,017 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 44 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,018 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 45 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,018 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 51 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,018 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 50 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,019 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 49 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,019 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 48 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,019 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 55 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,019 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 54 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,020 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 53 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,020 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 52 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,020 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 59 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,021 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 58 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,021 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 57 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] 2013-10-01 11:05:13,021 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 56 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865] [testng-StaleLocksWithLockOnlyTxDuringStateTransferTest] Test testAsync(org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest) failed. 2013-10-01 11:05:13,035 ERROR [UnitTestTestNGListener] (testng-StaleLocksWithLockOnlyTxDuringStateTransferTest) Test testAsync(org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest) failed. java.lang.IllegalStateException: Thread already timed out waiting for event allow_get_transactions_1_from_StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB-26402 at org.infinispan.test.fwk.CheckPoint.trigger(CheckPoint.java:131) at org.infinispan.test.fwk.CheckPoint.trigger(CheckPoint.java:116) at org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.doTest(StaleLocksWithLockOnlyTxDuringStateTransferTest.java:144) at org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.testAsync(StaleLocksWithLockOnlyTxDuringStateTransferTest.java:75) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80) at org.testng.internal.Invoker.invokeMethod(Invoker.java:715) at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:907) at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1237) at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127) at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111) at org.testng.TestRunner.privateRun(TestRunner.java:767) at org.testng.TestRunner.run(TestRunner.java:617) at org.testng.SuiteRunner.runTest(SuiteRunner.java:334) at org.testng.SuiteRunner.access$000(SuiteRunner.java:37) at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368) at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) Test suite progress: tests succeeded: 1536, failed: 1, skipped: 0. {code}