[JBoss JIRA] (ISPN-3626) Add tests for JGroups AUTH and ENCRYPT protocols to server
by Martin Gencur (JIRA)
[ https://issues.jboss.org/browse/ISPN-3626?page=com.atlassian.jira.plugin.... ]
Martin Gencur updated ISPN-3626:
--------------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan-server/pull/176
> Add tests for JGroups AUTH and ENCRYPT protocols to server
> ----------------------------------------------------------
>
> Key: ISPN-3626
> URL: https://issues.jboss.org/browse/ISPN-3626
> Project: Infinispan
> Issue Type: Task
> Components: Server
> Affects Versions: 6.0.0.CR1
> Reporter: Martin Gencur
> Assignee: Martin Gencur
> Fix For: 6.0.0.Final
>
>
> Usage:
> https://community.jboss.org/wiki/JGroupsAUTH
> https://community.jboss.org/wiki/JGroupsENCRYPT
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 3 months
- 1
- 0
[JBoss JIRA] (ISPN-3595) org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.testAsync & testSync fail randomly on RHEL && JDK6
by Vitalii Chepeliuk (JIRA)
[ https://issues.jboss.org/browse/ISPN-3595?page=com.atlassian.jira.plugin.... ]
Vitalii Chepeliuk edited comment on ISPN-3595 at 10/15/13 4:03 AM:
-------------------------------------------------------------------
Add trace logs for StaleLocksWithLockOnlyTxDuringStateTransferTest.testSync running with IBM JDK7
was (Author: vchepeli):
Add trace logs for StaleLocksWithLockOnlyTxDuringStateTransferTest.testSync running with JDK7
> org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.testAsync & testSync fail randomly on RHEL && JDK6
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-3595
> URL: https://issues.jboss.org/browse/ISPN-3595
> Project: Infinispan
> Issue Type: Bug
> Components: Core API
> Affects Versions: 6.0.0.Beta1
> Environment: RHEL5 && {JDK6, JDK7}
> Reporter: Anna Manukyan
> Assignee: Mircea Markus
> Labels: testsuite_stability
> Attachments: StaleLocksWithLockOnlyTxDuringStateTransferTest.testSync.log.zip, StaleLocksWithLockOnlyTxDuringStateTransferTest.testSync.log.zip
>
>
> The tests:
> org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.testAsync
> org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.testSync
> fail randomly on RHEL 5,6 - 32 and 64 bit machines with JDK6.
> The error message is:
> {code}
> 2013-10-01 11:05:02,877 WARN [CacheNotifierImpl] (testng-StaleLocksWithLockOnlyTxDuringStateTransferTest) ISPN000133: Attempted to register listener of class class org.infinispan.statetransfer.StateProviderImpl$$EnhancerByMockitoWithCGLIB$$1b069019, but no valid, public methods annotated with method-level event annotations found! Ignoring listener.
> 2013-10-01 11:05:05,306 WARN [arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff0a1059e0:ce1c:524adeb6:2fd9 in state RUN
> 2013-10-01 11:05:05,308 WARN [jta] (Transaction Reaper Worker 0) ARJUNA016029: SynchronizationImple.afterCompletion - failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=[StaleLocksWithCommitDuringStateTransferTest-NodeK-42948, StaleLocksWithCommitDuringStateTransferTest-NodeL-43885], isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=[MagicKey#k2{f9dc9ad7@StaleLocksWithCommitDuringStateTransferTest-NodeL-43885/39}], topologyId=2, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@32a6} org.infinispan.transaction.synchronization.SynchronizationAdapter@32c5 with exception
> org.infinispan.commons.CacheException: Could not commit.
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:67)
> at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:96)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:402)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:334)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.cancel(TwoPhaseCoordinator.java:120)
> at com.arjuna.ats.arjuna.AtomicAction.cancel(AtomicAction.java:215)
> at com.arjuna.ats.arjuna.coordinator.TransactionReaper.doCancellations(TransactionReaper.java:370)
> at com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:78)
> Caused by: javax.transaction.xa.XAException
> at org.infinispan.transaction.TransactionCoordinator.rollback(TransactionCoordinator.java:186)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:65)
> ... 7 more
> 2013-10-01 11:05:05,308 WARN [arjuna] (Transaction Reaper Worker 0) ARJUNA012127: TwoPhaseCoordinator.afterCompletion - returned failure for SynchronizationImple< 0:ffff0a1059e0:ce1c:524adeb6:2fda, SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=[StaleLocksWithCommitDuringStateTransferTest-NodeK-42948, StaleLocksWithCommitDuringStateTransferTest-NodeL-43885], isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=[MagicKey#k2{f9dc9ad7@StaleLocksWithCommitDuringStateTransferTest-NodeL-43885/39}], topologyId=2, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@32a6} org.infinispan.transaction.synchronization.SynchronizationAdapter@32c5 >
> 2013-10-01 11:05:05,309 WARN [arjuna] (Transaction Reaper Worker 0) ARJUNA012121: TransactionReaper::doCancellations worker Thread[Transaction Reaper Worker 0,5,main] successfully canceled TX 0:ffff0a1059e0:ce1c:524adeb6:2fd9
> 2013-10-01 11:05:12,943 ERROR [CheckPoint] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA) Timed out waiting for event allow_get_transactions_1_from_StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB-26402 * 1 (available = 0, total = 0
> 2013-10-01 11:05:12,943 ERROR [InboundInvocationHandlerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA) Exception executing command
> java.util.concurrent.TimeoutException: Timed out waiting for event allow_get_transactions_1_from_StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB-26402
> at org.infinispan.test.fwk.CheckPoint.awaitStrict(CheckPoint.java:43)
> at org.infinispan.test.fwk.CheckPoint.awaitStrict(CheckPoint.java:33)
> at org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest$1.answer(StaleLocksWithLockOnlyTxDuringStateTransferTest.java:101)
> at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34)
> at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:91)
> at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
> at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:38)
> at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:51)
> at org.infinispan.statetransfer.StateProviderImpl$$EnhancerByMockitoWithCGLIB$$1b069019.getTransactionsForSegments(<generated>)
> at org.infinispan.statetransfer.StateRequestCommand.perform(StateRequestCommand.java:65)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:97)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:46)
> at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:169)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> 2013-10-01 11:05:12,950 WARN [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000209: Failed to retrieve transactions for segments [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 16, 19, 18, 21, 20, 23, 22, 25, 24, 27, 26, 29, 28, 31, 30, 34, 35, 32, 33, 38, 39, 36, 37, 42, 43, 40, 41, 46, 47, 44, 45, 51, 50, 49, 48, 55, 54, 53, 52, 59, 58, 57, 56] of cache testCache from node StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865 (node will not be retried)
> org.infinispan.remoting.RemoteException: ISPN000217: Received exception from StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:38)
> at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:63)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:534)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:288)
> at org.infinispan.statetransfer.StateConsumerImpl.getTransactions(StateConsumerImpl.java:686)
> at org.infinispan.statetransfer.StateConsumerImpl.requestTransactions(StateConsumerImpl.java:653)
> at org.infinispan.statetransfer.StateConsumerImpl.addTransfers(StateConsumerImpl.java:603)
> at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:367)
> at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:178)
> at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:38)
> at org.infinispan.statetransfer.StateTransferManagerImpl$1.rebalance(StateTransferManagerImpl.java:105)
> at org.infinispan.topology.LocalTopologyManagerImpl.handleRebalance(LocalTopologyManagerImpl.java:228)
> at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:155)
> at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:124)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$4.run(CommandAwareRpcDispatcher.java:270)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> Caused by: java.util.concurrent.TimeoutException: Timed out waiting for event allow_get_transactions_1_from_StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB-26402
> at org.infinispan.test.fwk.CheckPoint.awaitStrict(CheckPoint.java:43)
> at org.infinispan.test.fwk.CheckPoint.awaitStrict(CheckPoint.java:33)
> at org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest$1.answer(StaleLocksWithLockOnlyTxDuringStateTransferTest.java:101)
> at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34)
> at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:91)
> at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
> at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:38)
> at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:51)
> at org.infinispan.statetransfer.StateProviderImpl$$EnhancerByMockitoWithCGLIB$$1b069019.getTransactionsForSegments(<generated>)
> at org.infinispan.statetransfer.StateRequestCommand.perform(StateRequestCommand.java:65)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:97)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:46)
> at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:169)
> ... 3 more
> 2013-10-01 11:05:12,951 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 0 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,951 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 1 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,951 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 2 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,951 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 3 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 4 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 5 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 6 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 7 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 8 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 9 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 10 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 11 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,953 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 12 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,953 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 13 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,953 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 14 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,954 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 15 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,954 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 17 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,954 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 16 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,955 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 19 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,955 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 18 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,955 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 21 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,955 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 20 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,956 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 23 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,956 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 22 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,956 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 25 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,956 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 24 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,957 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 27 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,957 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 26 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,957 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 29 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,957 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 28 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,958 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 31 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,958 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 30 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,958 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 34 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,959 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 35 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,959 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 32 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,959 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 33 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,959 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 38 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,960 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 39 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,960 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 36 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,960 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 37 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,960 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 42 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,961 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 43 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,961 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 40 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,961 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 41 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,962 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 46 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,962 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 47 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,999 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 44 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,000 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 45 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,000 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 51 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,001 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 50 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,001 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 49 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,001 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 48 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,002 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 55 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,002 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 54 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,002 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 53 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,002 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 52 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,003 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 59 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,003 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 58 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,003 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 57 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,004 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 56 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,004 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 0 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,004 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 1 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,005 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 2 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,005 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 3 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,005 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 4 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,005 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 5 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,006 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 6 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,006 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 7 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,006 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 8 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,007 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 9 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,007 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 10 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,007 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 11 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,008 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 12 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,008 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 13 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,008 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 14 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,008 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 15 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,009 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 17 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,009 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 16 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,009 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 19 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,010 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 18 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,010 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 21 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,010 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 20 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,010 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 23 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,011 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 22 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,011 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 25 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,011 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 24 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,012 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 27 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,012 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 26 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,012 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 29 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,012 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 28 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,013 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 31 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,013 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 30 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,013 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 34 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,014 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 35 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,014 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 32 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,014 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 33 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,014 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 38 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,015 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 39 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,015 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 36 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,015 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 37 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,016 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 42 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,016 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 43 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,016 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 40 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,016 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 41 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,017 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 46 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,017 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 47 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,017 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 44 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,018 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 45 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,018 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 51 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,018 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 50 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,019 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 49 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,019 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 48 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,019 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 55 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,019 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 54 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,020 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 53 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,020 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 52 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,020 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 59 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,021 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 58 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,021 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 57 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,021 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 56 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> [testng-StaleLocksWithLockOnlyTxDuringStateTransferTest] Test testAsync(org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest) failed.
> 2013-10-01 11:05:13,035 ERROR [UnitTestTestNGListener] (testng-StaleLocksWithLockOnlyTxDuringStateTransferTest) Test testAsync(org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest) failed.
> java.lang.IllegalStateException: Thread already timed out waiting for event allow_get_transactions_1_from_StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB-26402
> at org.infinispan.test.fwk.CheckPoint.trigger(CheckPoint.java:131)
> at org.infinispan.test.fwk.CheckPoint.trigger(CheckPoint.java:116)
> at org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.doTest(StaleLocksWithLockOnlyTxDuringStateTransferTest.java:144)
> at org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.testAsync(StaleLocksWithLockOnlyTxDuringStateTransferTest.java:75)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:715)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:907)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1237)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> Test suite progress: tests succeeded: 1536, failed: 1, skipped: 0.
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 3 months
- 1
- 0
[JBoss JIRA] (ISPN-3595) org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.testAsync & testSync fail randomly on RHEL && JDK6
by Vitalii Chepeliuk (JIRA)
[ https://issues.jboss.org/browse/ISPN-3595?page=com.atlassian.jira.plugin.... ]
Vitalii Chepeliuk edited comment on ISPN-3595 at 10/15/13 4:03 AM:
-------------------------------------------------------------------
Add traces for StaleLocksWithLockOnlyTxDuringStateTransferTest.testSync with running on ORACLE JDK6
was (Author: vchepeli):
Add traces for StaleLocksWithLockOnlyTxDuringStateTransferTest.testSync with running on JDK6
> org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.testAsync & testSync fail randomly on RHEL && JDK6
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-3595
> URL: https://issues.jboss.org/browse/ISPN-3595
> Project: Infinispan
> Issue Type: Bug
> Components: Core API
> Affects Versions: 6.0.0.Beta1
> Environment: RHEL5 && {JDK6, JDK7}
> Reporter: Anna Manukyan
> Assignee: Mircea Markus
> Labels: testsuite_stability
> Attachments: StaleLocksWithLockOnlyTxDuringStateTransferTest.testSync.log.zip, StaleLocksWithLockOnlyTxDuringStateTransferTest.testSync.log.zip
>
>
> The tests:
> org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.testAsync
> org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.testSync
> fail randomly on RHEL 5,6 - 32 and 64 bit machines with JDK6.
> The error message is:
> {code}
> 2013-10-01 11:05:02,877 WARN [CacheNotifierImpl] (testng-StaleLocksWithLockOnlyTxDuringStateTransferTest) ISPN000133: Attempted to register listener of class class org.infinispan.statetransfer.StateProviderImpl$$EnhancerByMockitoWithCGLIB$$1b069019, but no valid, public methods annotated with method-level event annotations found! Ignoring listener.
> 2013-10-01 11:05:05,306 WARN [arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff0a1059e0:ce1c:524adeb6:2fd9 in state RUN
> 2013-10-01 11:05:05,308 WARN [jta] (Transaction Reaper Worker 0) ARJUNA016029: SynchronizationImple.afterCompletion - failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=[StaleLocksWithCommitDuringStateTransferTest-NodeK-42948, StaleLocksWithCommitDuringStateTransferTest-NodeL-43885], isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=[MagicKey#k2{f9dc9ad7@StaleLocksWithCommitDuringStateTransferTest-NodeL-43885/39}], topologyId=2, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@32a6} org.infinispan.transaction.synchronization.SynchronizationAdapter@32c5 with exception
> org.infinispan.commons.CacheException: Could not commit.
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:67)
> at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:96)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:402)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:334)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.cancel(TwoPhaseCoordinator.java:120)
> at com.arjuna.ats.arjuna.AtomicAction.cancel(AtomicAction.java:215)
> at com.arjuna.ats.arjuna.coordinator.TransactionReaper.doCancellations(TransactionReaper.java:370)
> at com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:78)
> Caused by: javax.transaction.xa.XAException
> at org.infinispan.transaction.TransactionCoordinator.rollback(TransactionCoordinator.java:186)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:65)
> ... 7 more
> 2013-10-01 11:05:05,308 WARN [arjuna] (Transaction Reaper Worker 0) ARJUNA012127: TwoPhaseCoordinator.afterCompletion - returned failure for SynchronizationImple< 0:ffff0a1059e0:ce1c:524adeb6:2fda, SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=[StaleLocksWithCommitDuringStateTransferTest-NodeK-42948, StaleLocksWithCommitDuringStateTransferTest-NodeL-43885], isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=[MagicKey#k2{f9dc9ad7@StaleLocksWithCommitDuringStateTransferTest-NodeL-43885/39}], topologyId=2, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@32a6} org.infinispan.transaction.synchronization.SynchronizationAdapter@32c5 >
> 2013-10-01 11:05:05,309 WARN [arjuna] (Transaction Reaper Worker 0) ARJUNA012121: TransactionReaper::doCancellations worker Thread[Transaction Reaper Worker 0,5,main] successfully canceled TX 0:ffff0a1059e0:ce1c:524adeb6:2fd9
> 2013-10-01 11:05:12,943 ERROR [CheckPoint] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA) Timed out waiting for event allow_get_transactions_1_from_StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB-26402 * 1 (available = 0, total = 0
> 2013-10-01 11:05:12,943 ERROR [InboundInvocationHandlerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA) Exception executing command
> java.util.concurrent.TimeoutException: Timed out waiting for event allow_get_transactions_1_from_StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB-26402
> at org.infinispan.test.fwk.CheckPoint.awaitStrict(CheckPoint.java:43)
> at org.infinispan.test.fwk.CheckPoint.awaitStrict(CheckPoint.java:33)
> at org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest$1.answer(StaleLocksWithLockOnlyTxDuringStateTransferTest.java:101)
> at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34)
> at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:91)
> at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
> at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:38)
> at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:51)
> at org.infinispan.statetransfer.StateProviderImpl$$EnhancerByMockitoWithCGLIB$$1b069019.getTransactionsForSegments(<generated>)
> at org.infinispan.statetransfer.StateRequestCommand.perform(StateRequestCommand.java:65)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:97)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:46)
> at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:169)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> 2013-10-01 11:05:12,950 WARN [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000209: Failed to retrieve transactions for segments [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 16, 19, 18, 21, 20, 23, 22, 25, 24, 27, 26, 29, 28, 31, 30, 34, 35, 32, 33, 38, 39, 36, 37, 42, 43, 40, 41, 46, 47, 44, 45, 51, 50, 49, 48, 55, 54, 53, 52, 59, 58, 57, 56] of cache testCache from node StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865 (node will not be retried)
> org.infinispan.remoting.RemoteException: ISPN000217: Received exception from StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:38)
> at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:63)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:534)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:288)
> at org.infinispan.statetransfer.StateConsumerImpl.getTransactions(StateConsumerImpl.java:686)
> at org.infinispan.statetransfer.StateConsumerImpl.requestTransactions(StateConsumerImpl.java:653)
> at org.infinispan.statetransfer.StateConsumerImpl.addTransfers(StateConsumerImpl.java:603)
> at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:367)
> at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:178)
> at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:38)
> at org.infinispan.statetransfer.StateTransferManagerImpl$1.rebalance(StateTransferManagerImpl.java:105)
> at org.infinispan.topology.LocalTopologyManagerImpl.handleRebalance(LocalTopologyManagerImpl.java:228)
> at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:155)
> at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:124)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$4.run(CommandAwareRpcDispatcher.java:270)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> Caused by: java.util.concurrent.TimeoutException: Timed out waiting for event allow_get_transactions_1_from_StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB-26402
> at org.infinispan.test.fwk.CheckPoint.awaitStrict(CheckPoint.java:43)
> at org.infinispan.test.fwk.CheckPoint.awaitStrict(CheckPoint.java:33)
> at org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest$1.answer(StaleLocksWithLockOnlyTxDuringStateTransferTest.java:101)
> at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34)
> at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:91)
> at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
> at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:38)
> at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:51)
> at org.infinispan.statetransfer.StateProviderImpl$$EnhancerByMockitoWithCGLIB$$1b069019.getTransactionsForSegments(<generated>)
> at org.infinispan.statetransfer.StateRequestCommand.perform(StateRequestCommand.java:65)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:97)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:46)
> at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:169)
> ... 3 more
> 2013-10-01 11:05:12,951 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 0 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,951 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 1 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,951 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 2 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,951 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 3 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 4 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 5 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 6 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 7 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 8 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 9 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 10 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 11 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,953 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 12 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,953 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 13 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,953 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 14 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,954 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 15 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,954 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 17 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,954 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 16 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,955 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 19 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,955 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 18 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,955 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 21 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,955 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 20 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,956 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 23 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,956 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 22 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,956 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 25 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,956 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 24 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,957 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 27 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,957 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 26 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,957 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 29 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,957 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 28 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,958 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 31 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,958 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 30 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,958 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 34 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,959 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 35 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,959 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 32 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,959 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 33 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,959 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 38 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,960 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 39 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,960 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 36 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,960 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 37 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,960 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 42 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,961 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 43 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,961 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 40 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,961 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 41 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,962 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 46 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,962 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 47 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,999 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 44 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,000 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 45 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,000 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 51 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,001 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 50 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,001 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 49 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,001 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 48 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,002 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 55 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,002 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 54 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,002 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 53 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,002 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 52 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,003 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 59 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,003 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 58 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,003 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 57 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,004 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 56 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,004 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 0 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,004 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 1 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,005 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 2 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,005 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 3 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,005 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 4 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,005 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 5 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,006 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 6 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,006 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 7 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,006 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 8 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,007 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 9 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,007 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 10 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,007 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 11 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,008 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 12 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,008 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 13 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,008 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 14 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,008 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 15 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,009 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 17 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,009 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 16 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,009 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 19 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,010 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 18 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,010 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 21 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,010 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 20 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,010 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 23 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,011 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 22 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,011 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 25 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,011 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 24 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,012 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 27 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,012 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 26 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,012 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 29 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,012 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 28 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,013 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 31 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,013 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 30 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,013 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 34 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,014 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 35 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,014 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 32 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,014 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 33 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,014 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 38 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,015 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 39 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,015 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 36 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,015 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 37 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,016 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 42 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,016 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 43 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,016 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 40 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,016 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 41 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,017 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 46 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,017 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 47 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,017 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 44 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,018 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 45 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,018 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 51 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,018 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 50 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,019 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 49 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,019 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 48 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,019 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 55 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,019 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 54 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,020 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 53 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,020 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 52 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,020 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 59 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,021 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 58 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,021 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 57 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,021 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 56 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> [testng-StaleLocksWithLockOnlyTxDuringStateTransferTest] Test testAsync(org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest) failed.
> 2013-10-01 11:05:13,035 ERROR [UnitTestTestNGListener] (testng-StaleLocksWithLockOnlyTxDuringStateTransferTest) Test testAsync(org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest) failed.
> java.lang.IllegalStateException: Thread already timed out waiting for event allow_get_transactions_1_from_StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB-26402
> at org.infinispan.test.fwk.CheckPoint.trigger(CheckPoint.java:131)
> at org.infinispan.test.fwk.CheckPoint.trigger(CheckPoint.java:116)
> at org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.doTest(StaleLocksWithLockOnlyTxDuringStateTransferTest.java:144)
> at org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.testAsync(StaleLocksWithLockOnlyTxDuringStateTransferTest.java:75)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:715)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:907)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1237)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> Test suite progress: tests succeeded: 1536, failed: 1, skipped: 0.
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 3 months
- 1
- 0
[JBoss JIRA] (ISPN-3595) org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.testAsync & testSync fail randomly on RHEL && JDK6
by Vitalii Chepeliuk (JIRA)
[ https://issues.jboss.org/browse/ISPN-3595?page=com.atlassian.jira.plugin.... ]
Vitalii Chepeliuk updated ISPN-3595:
------------------------------------
Attachment: StaleLocksWithLockOnlyTxDuringStateTransferTest.testSync.log.zip
Add trace logs for StaleLocksWithLockOnlyTxDuringStateTransferTest.testSync running with JDK7
> org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.testAsync & testSync fail randomly on RHEL && JDK6
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-3595
> URL: https://issues.jboss.org/browse/ISPN-3595
> Project: Infinispan
> Issue Type: Bug
> Components: Core API
> Affects Versions: 6.0.0.Beta1
> Environment: RHEL5 && {JDK6, JDK7}
> Reporter: Anna Manukyan
> Assignee: Mircea Markus
> Labels: testsuite_stability
> Attachments: StaleLocksWithLockOnlyTxDuringStateTransferTest.testSync.log.zip, StaleLocksWithLockOnlyTxDuringStateTransferTest.testSync.log.zip
>
>
> The tests:
> org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.testAsync
> org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.testSync
> fail randomly on RHEL 5,6 - 32 and 64 bit machines with JDK6.
> The error message is:
> {code}
> 2013-10-01 11:05:02,877 WARN [CacheNotifierImpl] (testng-StaleLocksWithLockOnlyTxDuringStateTransferTest) ISPN000133: Attempted to register listener of class class org.infinispan.statetransfer.StateProviderImpl$$EnhancerByMockitoWithCGLIB$$1b069019, but no valid, public methods annotated with method-level event annotations found! Ignoring listener.
> 2013-10-01 11:05:05,306 WARN [arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff0a1059e0:ce1c:524adeb6:2fd9 in state RUN
> 2013-10-01 11:05:05,308 WARN [jta] (Transaction Reaper Worker 0) ARJUNA016029: SynchronizationImple.afterCompletion - failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=[StaleLocksWithCommitDuringStateTransferTest-NodeK-42948, StaleLocksWithCommitDuringStateTransferTest-NodeL-43885], isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=[MagicKey#k2{f9dc9ad7@StaleLocksWithCommitDuringStateTransferTest-NodeL-43885/39}], topologyId=2, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@32a6} org.infinispan.transaction.synchronization.SynchronizationAdapter@32c5 with exception
> org.infinispan.commons.CacheException: Could not commit.
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:67)
> at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:96)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:402)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:334)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.cancel(TwoPhaseCoordinator.java:120)
> at com.arjuna.ats.arjuna.AtomicAction.cancel(AtomicAction.java:215)
> at com.arjuna.ats.arjuna.coordinator.TransactionReaper.doCancellations(TransactionReaper.java:370)
> at com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:78)
> Caused by: javax.transaction.xa.XAException
> at org.infinispan.transaction.TransactionCoordinator.rollback(TransactionCoordinator.java:186)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:65)
> ... 7 more
> 2013-10-01 11:05:05,308 WARN [arjuna] (Transaction Reaper Worker 0) ARJUNA012127: TwoPhaseCoordinator.afterCompletion - returned failure for SynchronizationImple< 0:ffff0a1059e0:ce1c:524adeb6:2fda, SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=[StaleLocksWithCommitDuringStateTransferTest-NodeK-42948, StaleLocksWithCommitDuringStateTransferTest-NodeL-43885], isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=[MagicKey#k2{f9dc9ad7@StaleLocksWithCommitDuringStateTransferTest-NodeL-43885/39}], topologyId=2, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@32a6} org.infinispan.transaction.synchronization.SynchronizationAdapter@32c5 >
> 2013-10-01 11:05:05,309 WARN [arjuna] (Transaction Reaper Worker 0) ARJUNA012121: TransactionReaper::doCancellations worker Thread[Transaction Reaper Worker 0,5,main] successfully canceled TX 0:ffff0a1059e0:ce1c:524adeb6:2fd9
> 2013-10-01 11:05:12,943 ERROR [CheckPoint] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA) Timed out waiting for event allow_get_transactions_1_from_StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB-26402 * 1 (available = 0, total = 0
> 2013-10-01 11:05:12,943 ERROR [InboundInvocationHandlerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA) Exception executing command
> java.util.concurrent.TimeoutException: Timed out waiting for event allow_get_transactions_1_from_StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB-26402
> at org.infinispan.test.fwk.CheckPoint.awaitStrict(CheckPoint.java:43)
> at org.infinispan.test.fwk.CheckPoint.awaitStrict(CheckPoint.java:33)
> at org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest$1.answer(StaleLocksWithLockOnlyTxDuringStateTransferTest.java:101)
> at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34)
> at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:91)
> at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
> at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:38)
> at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:51)
> at org.infinispan.statetransfer.StateProviderImpl$$EnhancerByMockitoWithCGLIB$$1b069019.getTransactionsForSegments(<generated>)
> at org.infinispan.statetransfer.StateRequestCommand.perform(StateRequestCommand.java:65)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:97)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:46)
> at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:169)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> 2013-10-01 11:05:12,950 WARN [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000209: Failed to retrieve transactions for segments [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 16, 19, 18, 21, 20, 23, 22, 25, 24, 27, 26, 29, 28, 31, 30, 34, 35, 32, 33, 38, 39, 36, 37, 42, 43, 40, 41, 46, 47, 44, 45, 51, 50, 49, 48, 55, 54, 53, 52, 59, 58, 57, 56] of cache testCache from node StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865 (node will not be retried)
> org.infinispan.remoting.RemoteException: ISPN000217: Received exception from StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:38)
> at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:63)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:534)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:288)
> at org.infinispan.statetransfer.StateConsumerImpl.getTransactions(StateConsumerImpl.java:686)
> at org.infinispan.statetransfer.StateConsumerImpl.requestTransactions(StateConsumerImpl.java:653)
> at org.infinispan.statetransfer.StateConsumerImpl.addTransfers(StateConsumerImpl.java:603)
> at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:367)
> at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:178)
> at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:38)
> at org.infinispan.statetransfer.StateTransferManagerImpl$1.rebalance(StateTransferManagerImpl.java:105)
> at org.infinispan.topology.LocalTopologyManagerImpl.handleRebalance(LocalTopologyManagerImpl.java:228)
> at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:155)
> at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:124)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$4.run(CommandAwareRpcDispatcher.java:270)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> Caused by: java.util.concurrent.TimeoutException: Timed out waiting for event allow_get_transactions_1_from_StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB-26402
> at org.infinispan.test.fwk.CheckPoint.awaitStrict(CheckPoint.java:43)
> at org.infinispan.test.fwk.CheckPoint.awaitStrict(CheckPoint.java:33)
> at org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest$1.answer(StaleLocksWithLockOnlyTxDuringStateTransferTest.java:101)
> at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34)
> at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:91)
> at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
> at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:38)
> at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:51)
> at org.infinispan.statetransfer.StateProviderImpl$$EnhancerByMockitoWithCGLIB$$1b069019.getTransactionsForSegments(<generated>)
> at org.infinispan.statetransfer.StateRequestCommand.perform(StateRequestCommand.java:65)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:97)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:46)
> at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:169)
> ... 3 more
> 2013-10-01 11:05:12,951 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 0 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,951 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 1 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,951 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 2 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,951 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 3 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 4 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 5 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 6 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 7 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 8 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 9 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 10 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 11 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,953 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 12 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,953 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 13 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,953 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 14 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,954 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 15 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,954 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 17 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,954 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 16 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,955 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 19 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,955 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 18 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,955 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 21 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,955 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 20 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,956 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 23 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,956 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 22 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,956 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 25 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,956 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 24 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,957 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 27 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,957 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 26 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,957 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 29 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,957 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 28 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,958 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 31 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,958 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 30 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,958 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 34 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,959 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 35 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,959 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 32 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,959 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 33 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,959 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 38 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,960 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 39 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,960 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 36 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,960 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 37 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,960 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 42 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,961 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 43 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,961 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 40 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,961 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 41 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,962 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 46 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,962 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 47 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,999 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 44 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,000 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 45 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,000 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 51 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,001 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 50 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,001 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 49 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,001 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 48 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,002 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 55 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,002 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 54 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,002 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 53 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,002 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 52 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,003 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 59 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,003 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 58 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,003 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 57 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,004 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 56 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,004 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 0 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,004 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 1 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,005 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 2 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,005 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 3 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,005 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 4 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,005 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 5 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,006 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 6 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,006 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 7 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,006 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 8 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,007 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 9 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,007 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 10 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,007 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 11 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,008 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 12 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,008 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 13 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,008 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 14 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,008 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 15 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,009 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 17 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,009 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 16 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,009 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 19 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,010 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 18 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,010 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 21 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,010 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 20 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,010 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 23 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,011 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 22 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,011 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 25 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,011 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 24 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,012 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 27 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,012 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 26 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,012 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 29 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,012 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 28 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,013 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 31 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,013 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 30 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,013 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 34 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,014 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 35 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,014 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 32 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,014 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 33 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,014 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 38 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,015 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 39 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,015 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 36 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,015 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 37 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,016 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 42 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,016 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 43 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,016 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 40 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,016 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 41 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,017 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 46 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,017 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 47 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,017 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 44 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,018 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 45 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,018 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 51 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,018 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 50 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,019 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 49 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,019 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 48 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,019 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 55 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,019 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 54 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,020 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 53 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,020 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 52 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,020 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 59 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,021 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 58 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,021 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 57 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,021 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 56 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> [testng-StaleLocksWithLockOnlyTxDuringStateTransferTest] Test testAsync(org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest) failed.
> 2013-10-01 11:05:13,035 ERROR [UnitTestTestNGListener] (testng-StaleLocksWithLockOnlyTxDuringStateTransferTest) Test testAsync(org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest) failed.
> java.lang.IllegalStateException: Thread already timed out waiting for event allow_get_transactions_1_from_StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB-26402
> at org.infinispan.test.fwk.CheckPoint.trigger(CheckPoint.java:131)
> at org.infinispan.test.fwk.CheckPoint.trigger(CheckPoint.java:116)
> at org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.doTest(StaleLocksWithLockOnlyTxDuringStateTransferTest.java:144)
> at org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.testAsync(StaleLocksWithLockOnlyTxDuringStateTransferTest.java:75)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:715)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:907)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1237)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> Test suite progress: tests succeeded: 1536, failed: 1, skipped: 0.
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 3 months
- 1
- 0
[JBoss JIRA] (ISPN-3595) org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.testAsync & testSync fail randomly on RHEL && JDK6
by Vitalii Chepeliuk (JIRA)
[ https://issues.jboss.org/browse/ISPN-3595?page=com.atlassian.jira.plugin.... ]
Vitalii Chepeliuk updated ISPN-3595:
------------------------------------
Attachment: StaleLocksWithLockOnlyTxDuringStateTransferTest.testSync.log.zip
Add traces for StaleLocksWithLockOnlyTxDuringStateTransferTest.testSync with running on JDK6
> org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.testAsync & testSync fail randomly on RHEL && JDK6
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-3595
> URL: https://issues.jboss.org/browse/ISPN-3595
> Project: Infinispan
> Issue Type: Bug
> Components: Core API
> Affects Versions: 6.0.0.Beta1
> Environment: RHEL5 && {JDK6, JDK7}
> Reporter: Anna Manukyan
> Assignee: Mircea Markus
> Labels: testsuite_stability
> Attachments: StaleLocksWithLockOnlyTxDuringStateTransferTest.testSync.log.zip
>
>
> The tests:
> org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.testAsync
> org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.testSync
> fail randomly on RHEL 5,6 - 32 and 64 bit machines with JDK6.
> The error message is:
> {code}
> 2013-10-01 11:05:02,877 WARN [CacheNotifierImpl] (testng-StaleLocksWithLockOnlyTxDuringStateTransferTest) ISPN000133: Attempted to register listener of class class org.infinispan.statetransfer.StateProviderImpl$$EnhancerByMockitoWithCGLIB$$1b069019, but no valid, public methods annotated with method-level event annotations found! Ignoring listener.
> 2013-10-01 11:05:05,306 WARN [arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff0a1059e0:ce1c:524adeb6:2fd9 in state RUN
> 2013-10-01 11:05:05,308 WARN [jta] (Transaction Reaper Worker 0) ARJUNA016029: SynchronizationImple.afterCompletion - failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=[StaleLocksWithCommitDuringStateTransferTest-NodeK-42948, StaleLocksWithCommitDuringStateTransferTest-NodeL-43885], isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=[MagicKey#k2{f9dc9ad7@StaleLocksWithCommitDuringStateTransferTest-NodeL-43885/39}], topologyId=2, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@32a6} org.infinispan.transaction.synchronization.SynchronizationAdapter@32c5 with exception
> org.infinispan.commons.CacheException: Could not commit.
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:67)
> at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:96)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:402)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:334)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.cancel(TwoPhaseCoordinator.java:120)
> at com.arjuna.ats.arjuna.AtomicAction.cancel(AtomicAction.java:215)
> at com.arjuna.ats.arjuna.coordinator.TransactionReaper.doCancellations(TransactionReaper.java:370)
> at com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:78)
> Caused by: javax.transaction.xa.XAException
> at org.infinispan.transaction.TransactionCoordinator.rollback(TransactionCoordinator.java:186)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:65)
> ... 7 more
> 2013-10-01 11:05:05,308 WARN [arjuna] (Transaction Reaper Worker 0) ARJUNA012127: TwoPhaseCoordinator.afterCompletion - returned failure for SynchronizationImple< 0:ffff0a1059e0:ce1c:524adeb6:2fda, SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=[StaleLocksWithCommitDuringStateTransferTest-NodeK-42948, StaleLocksWithCommitDuringStateTransferTest-NodeL-43885], isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=[MagicKey#k2{f9dc9ad7@StaleLocksWithCommitDuringStateTransferTest-NodeL-43885/39}], topologyId=2, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@32a6} org.infinispan.transaction.synchronization.SynchronizationAdapter@32c5 >
> 2013-10-01 11:05:05,309 WARN [arjuna] (Transaction Reaper Worker 0) ARJUNA012121: TransactionReaper::doCancellations worker Thread[Transaction Reaper Worker 0,5,main] successfully canceled TX 0:ffff0a1059e0:ce1c:524adeb6:2fd9
> 2013-10-01 11:05:12,943 ERROR [CheckPoint] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA) Timed out waiting for event allow_get_transactions_1_from_StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB-26402 * 1 (available = 0, total = 0
> 2013-10-01 11:05:12,943 ERROR [InboundInvocationHandlerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA) Exception executing command
> java.util.concurrent.TimeoutException: Timed out waiting for event allow_get_transactions_1_from_StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB-26402
> at org.infinispan.test.fwk.CheckPoint.awaitStrict(CheckPoint.java:43)
> at org.infinispan.test.fwk.CheckPoint.awaitStrict(CheckPoint.java:33)
> at org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest$1.answer(StaleLocksWithLockOnlyTxDuringStateTransferTest.java:101)
> at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34)
> at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:91)
> at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
> at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:38)
> at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:51)
> at org.infinispan.statetransfer.StateProviderImpl$$EnhancerByMockitoWithCGLIB$$1b069019.getTransactionsForSegments(<generated>)
> at org.infinispan.statetransfer.StateRequestCommand.perform(StateRequestCommand.java:65)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:97)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:46)
> at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:169)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> 2013-10-01 11:05:12,950 WARN [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000209: Failed to retrieve transactions for segments [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 16, 19, 18, 21, 20, 23, 22, 25, 24, 27, 26, 29, 28, 31, 30, 34, 35, 32, 33, 38, 39, 36, 37, 42, 43, 40, 41, 46, 47, 44, 45, 51, 50, 49, 48, 55, 54, 53, 52, 59, 58, 57, 56] of cache testCache from node StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865 (node will not be retried)
> org.infinispan.remoting.RemoteException: ISPN000217: Received exception from StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:38)
> at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:63)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:534)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:288)
> at org.infinispan.statetransfer.StateConsumerImpl.getTransactions(StateConsumerImpl.java:686)
> at org.infinispan.statetransfer.StateConsumerImpl.requestTransactions(StateConsumerImpl.java:653)
> at org.infinispan.statetransfer.StateConsumerImpl.addTransfers(StateConsumerImpl.java:603)
> at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:367)
> at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:178)
> at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:38)
> at org.infinispan.statetransfer.StateTransferManagerImpl$1.rebalance(StateTransferManagerImpl.java:105)
> at org.infinispan.topology.LocalTopologyManagerImpl.handleRebalance(LocalTopologyManagerImpl.java:228)
> at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:155)
> at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:124)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$4.run(CommandAwareRpcDispatcher.java:270)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> Caused by: java.util.concurrent.TimeoutException: Timed out waiting for event allow_get_transactions_1_from_StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB-26402
> at org.infinispan.test.fwk.CheckPoint.awaitStrict(CheckPoint.java:43)
> at org.infinispan.test.fwk.CheckPoint.awaitStrict(CheckPoint.java:33)
> at org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest$1.answer(StaleLocksWithLockOnlyTxDuringStateTransferTest.java:101)
> at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34)
> at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:91)
> at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
> at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:38)
> at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:51)
> at org.infinispan.statetransfer.StateProviderImpl$$EnhancerByMockitoWithCGLIB$$1b069019.getTransactionsForSegments(<generated>)
> at org.infinispan.statetransfer.StateRequestCommand.perform(StateRequestCommand.java:65)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:97)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:46)
> at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:169)
> ... 3 more
> 2013-10-01 11:05:12,951 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 0 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,951 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 1 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,951 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 2 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,951 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 3 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 4 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 5 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 6 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 7 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 8 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 9 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 10 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 11 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,953 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 12 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,953 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 13 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,953 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 14 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,954 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 15 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,954 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 17 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,954 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 16 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,955 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 19 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,955 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 18 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,955 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 21 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,955 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 20 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,956 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 23 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,956 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 22 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,956 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 25 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,956 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 24 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,957 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 27 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,957 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 26 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,957 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 29 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,957 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 28 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,958 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 31 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,958 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 30 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,958 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 34 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,959 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 35 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,959 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 32 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,959 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 33 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,959 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 38 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,960 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 39 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,960 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 36 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,960 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 37 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,960 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 42 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,961 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 43 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,961 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 40 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,961 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 41 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,962 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 46 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,962 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 47 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,999 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 44 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,000 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 45 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,000 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 51 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,001 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 50 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,001 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 49 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,001 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 48 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,002 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 55 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,002 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 54 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,002 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 53 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,002 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 52 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,003 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 59 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,003 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 58 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,003 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 57 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,004 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 56 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,004 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 0 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,004 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 1 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,005 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 2 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,005 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 3 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,005 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 4 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,005 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 5 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,006 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 6 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,006 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 7 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,006 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 8 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,007 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 9 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,007 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 10 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,007 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 11 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,008 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 12 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,008 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 13 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,008 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 14 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,008 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 15 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,009 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 17 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,009 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 16 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,009 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 19 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,010 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 18 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,010 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 21 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,010 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 20 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,010 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 23 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,011 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 22 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,011 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 25 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,011 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 24 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,012 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 27 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,012 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 26 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,012 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 29 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,012 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 28 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,013 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 31 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,013 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 30 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,013 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 34 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,014 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 35 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,014 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 32 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,014 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 33 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,014 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 38 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,015 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 39 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,015 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 36 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,015 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 37 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,016 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 42 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,016 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 43 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,016 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 40 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,016 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 41 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,017 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 46 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,017 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 47 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,017 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 44 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,018 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 45 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,018 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 51 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,018 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 50 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,019 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 49 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,019 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 48 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,019 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 55 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,019 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 54 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,020 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 53 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,020 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 52 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,020 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 59 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,021 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 58 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,021 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 57 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,021 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 56 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> [testng-StaleLocksWithLockOnlyTxDuringStateTransferTest] Test testAsync(org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest) failed.
> 2013-10-01 11:05:13,035 ERROR [UnitTestTestNGListener] (testng-StaleLocksWithLockOnlyTxDuringStateTransferTest) Test testAsync(org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest) failed.
> java.lang.IllegalStateException: Thread already timed out waiting for event allow_get_transactions_1_from_StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB-26402
> at org.infinispan.test.fwk.CheckPoint.trigger(CheckPoint.java:131)
> at org.infinispan.test.fwk.CheckPoint.trigger(CheckPoint.java:116)
> at org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.doTest(StaleLocksWithLockOnlyTxDuringStateTransferTest.java:144)
> at org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.testAsync(StaleLocksWithLockOnlyTxDuringStateTransferTest.java:75)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:715)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:907)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1237)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> Test suite progress: tests succeeded: 1536, failed: 1, skipped: 0.
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 3 months
- 1
- 0
[JBoss JIRA] (ISPN-3595) org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.testAsync & testSync fail randomly on RHEL && JDK6
by Vitalii Chepeliuk (JIRA)
[ https://issues.jboss.org/browse/ISPN-3595?page=com.atlassian.jira.plugin.... ]
Vitalii Chepeliuk updated ISPN-3595:
------------------------------------
Environment: RHEL5 && {JDK6, JDK7} (was: RHEL5 && x64 && JDK6)
> org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.testAsync & testSync fail randomly on RHEL && JDK6
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-3595
> URL: https://issues.jboss.org/browse/ISPN-3595
> Project: Infinispan
> Issue Type: Bug
> Components: Core API
> Affects Versions: 6.0.0.Beta1
> Environment: RHEL5 && {JDK6, JDK7}
> Reporter: Anna Manukyan
> Assignee: Mircea Markus
> Labels: testsuite_stability
>
> The tests:
> org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.testAsync
> org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.testSync
> fail randomly on RHEL 5,6 - 32 and 64 bit machines with JDK6.
> The error message is:
> {code}
> 2013-10-01 11:05:02,877 WARN [CacheNotifierImpl] (testng-StaleLocksWithLockOnlyTxDuringStateTransferTest) ISPN000133: Attempted to register listener of class class org.infinispan.statetransfer.StateProviderImpl$$EnhancerByMockitoWithCGLIB$$1b069019, but no valid, public methods annotated with method-level event annotations found! Ignoring listener.
> 2013-10-01 11:05:05,306 WARN [arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff0a1059e0:ce1c:524adeb6:2fd9 in state RUN
> 2013-10-01 11:05:05,308 WARN [jta] (Transaction Reaper Worker 0) ARJUNA016029: SynchronizationImple.afterCompletion - failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=[StaleLocksWithCommitDuringStateTransferTest-NodeK-42948, StaleLocksWithCommitDuringStateTransferTest-NodeL-43885], isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=[MagicKey#k2{f9dc9ad7@StaleLocksWithCommitDuringStateTransferTest-NodeL-43885/39}], topologyId=2, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@32a6} org.infinispan.transaction.synchronization.SynchronizationAdapter@32c5 with exception
> org.infinispan.commons.CacheException: Could not commit.
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:67)
> at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:96)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:402)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:334)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.cancel(TwoPhaseCoordinator.java:120)
> at com.arjuna.ats.arjuna.AtomicAction.cancel(AtomicAction.java:215)
> at com.arjuna.ats.arjuna.coordinator.TransactionReaper.doCancellations(TransactionReaper.java:370)
> at com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:78)
> Caused by: javax.transaction.xa.XAException
> at org.infinispan.transaction.TransactionCoordinator.rollback(TransactionCoordinator.java:186)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:65)
> ... 7 more
> 2013-10-01 11:05:05,308 WARN [arjuna] (Transaction Reaper Worker 0) ARJUNA012127: TwoPhaseCoordinator.afterCompletion - returned failure for SynchronizationImple< 0:ffff0a1059e0:ce1c:524adeb6:2fda, SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=[StaleLocksWithCommitDuringStateTransferTest-NodeK-42948, StaleLocksWithCommitDuringStateTransferTest-NodeL-43885], isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=[MagicKey#k2{f9dc9ad7@StaleLocksWithCommitDuringStateTransferTest-NodeL-43885/39}], topologyId=2, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@32a6} org.infinispan.transaction.synchronization.SynchronizationAdapter@32c5 >
> 2013-10-01 11:05:05,309 WARN [arjuna] (Transaction Reaper Worker 0) ARJUNA012121: TransactionReaper::doCancellations worker Thread[Transaction Reaper Worker 0,5,main] successfully canceled TX 0:ffff0a1059e0:ce1c:524adeb6:2fd9
> 2013-10-01 11:05:12,943 ERROR [CheckPoint] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA) Timed out waiting for event allow_get_transactions_1_from_StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB-26402 * 1 (available = 0, total = 0
> 2013-10-01 11:05:12,943 ERROR [InboundInvocationHandlerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA) Exception executing command
> java.util.concurrent.TimeoutException: Timed out waiting for event allow_get_transactions_1_from_StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB-26402
> at org.infinispan.test.fwk.CheckPoint.awaitStrict(CheckPoint.java:43)
> at org.infinispan.test.fwk.CheckPoint.awaitStrict(CheckPoint.java:33)
> at org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest$1.answer(StaleLocksWithLockOnlyTxDuringStateTransferTest.java:101)
> at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34)
> at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:91)
> at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
> at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:38)
> at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:51)
> at org.infinispan.statetransfer.StateProviderImpl$$EnhancerByMockitoWithCGLIB$$1b069019.getTransactionsForSegments(<generated>)
> at org.infinispan.statetransfer.StateRequestCommand.perform(StateRequestCommand.java:65)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:97)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:46)
> at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:169)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> 2013-10-01 11:05:12,950 WARN [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000209: Failed to retrieve transactions for segments [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 16, 19, 18, 21, 20, 23, 22, 25, 24, 27, 26, 29, 28, 31, 30, 34, 35, 32, 33, 38, 39, 36, 37, 42, 43, 40, 41, 46, 47, 44, 45, 51, 50, 49, 48, 55, 54, 53, 52, 59, 58, 57, 56] of cache testCache from node StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865 (node will not be retried)
> org.infinispan.remoting.RemoteException: ISPN000217: Received exception from StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:38)
> at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:63)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:534)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:288)
> at org.infinispan.statetransfer.StateConsumerImpl.getTransactions(StateConsumerImpl.java:686)
> at org.infinispan.statetransfer.StateConsumerImpl.requestTransactions(StateConsumerImpl.java:653)
> at org.infinispan.statetransfer.StateConsumerImpl.addTransfers(StateConsumerImpl.java:603)
> at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:367)
> at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:178)
> at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:38)
> at org.infinispan.statetransfer.StateTransferManagerImpl$1.rebalance(StateTransferManagerImpl.java:105)
> at org.infinispan.topology.LocalTopologyManagerImpl.handleRebalance(LocalTopologyManagerImpl.java:228)
> at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:155)
> at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:124)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$4.run(CommandAwareRpcDispatcher.java:270)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> Caused by: java.util.concurrent.TimeoutException: Timed out waiting for event allow_get_transactions_1_from_StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB-26402
> at org.infinispan.test.fwk.CheckPoint.awaitStrict(CheckPoint.java:43)
> at org.infinispan.test.fwk.CheckPoint.awaitStrict(CheckPoint.java:33)
> at org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest$1.answer(StaleLocksWithLockOnlyTxDuringStateTransferTest.java:101)
> at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34)
> at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:91)
> at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
> at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:38)
> at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:51)
> at org.infinispan.statetransfer.StateProviderImpl$$EnhancerByMockitoWithCGLIB$$1b069019.getTransactionsForSegments(<generated>)
> at org.infinispan.statetransfer.StateRequestCommand.perform(StateRequestCommand.java:65)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:97)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:46)
> at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:169)
> ... 3 more
> 2013-10-01 11:05:12,951 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 0 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,951 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 1 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,951 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 2 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,951 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 3 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 4 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 5 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 6 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 7 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 8 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 9 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 10 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,952 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 11 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,953 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 12 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,953 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 13 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,953 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 14 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,954 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 15 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,954 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 17 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,954 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 16 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,955 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 19 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,955 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 18 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,955 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 21 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,955 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 20 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,956 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 23 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,956 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 22 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,956 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 25 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,956 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 24 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,957 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 27 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,957 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 26 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,957 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 29 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,957 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 28 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,958 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 31 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,958 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 30 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,958 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 34 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,959 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 35 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,959 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 32 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,959 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 33 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,959 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 38 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,960 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 39 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,960 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 36 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,960 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 37 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,960 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 42 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,961 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 43 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,961 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 40 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,961 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 41 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,962 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 46 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,962 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 47 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:12,999 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 44 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,000 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 45 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,000 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 51 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,001 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 50 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,001 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 49 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,001 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 48 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,002 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 55 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,002 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 54 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,002 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 53 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,002 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 52 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,003 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 59 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,003 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 58 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,003 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 57 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,004 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 56 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,004 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 0 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,004 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 1 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,005 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 2 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,005 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 3 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,005 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 4 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,005 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 5 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,006 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 6 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,006 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 7 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,006 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 8 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,007 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 9 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,007 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 10 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,007 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 11 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,008 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 12 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,008 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 13 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,008 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 14 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,008 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 15 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,009 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 17 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,009 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 16 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,009 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 19 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,010 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 18 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,010 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 21 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,010 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 20 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,010 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 23 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,011 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 22 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,011 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 25 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,011 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 24 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,012 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 27 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,012 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 26 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,012 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 29 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,012 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 28 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,013 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 31 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,013 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 30 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,013 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 34 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,014 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 35 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,014 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 32 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,014 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 33 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,014 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 38 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,015 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 39 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,015 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 36 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,015 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 37 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,016 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 42 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,016 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 43 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,016 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 40 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,016 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 41 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,017 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 46 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,017 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 47 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,017 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 44 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,018 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 45 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,018 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 51 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,018 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 50 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,019 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 49 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,019 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 48 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,019 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 55 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,019 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 54 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,020 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 53 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,020 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 52 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,020 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 59 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,021 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 58 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,021 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 57 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> 2013-10-01 11:05:13,021 ERROR [StateConsumerImpl] (remote-thread-0,StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB) ISPN000208: No live owners found for segment 56 of cache testCache. Current owners are: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]. Faulty owners: [StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeA-8865]
> [testng-StaleLocksWithLockOnlyTxDuringStateTransferTest] Test testAsync(org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest) failed.
> 2013-10-01 11:05:13,035 ERROR [UnitTestTestNGListener] (testng-StaleLocksWithLockOnlyTxDuringStateTransferTest) Test testAsync(org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest) failed.
> java.lang.IllegalStateException: Thread already timed out waiting for event allow_get_transactions_1_from_StaleLocksWithLockOnlyTxDuringStateTransferTest-NodeB-26402
> at org.infinispan.test.fwk.CheckPoint.trigger(CheckPoint.java:131)
> at org.infinispan.test.fwk.CheckPoint.trigger(CheckPoint.java:116)
> at org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.doTest(StaleLocksWithLockOnlyTxDuringStateTransferTest.java:144)
> at org.infinispan.statetransfer.StaleLocksWithLockOnlyTxDuringStateTransferTest.testAsync(StaleLocksWithLockOnlyTxDuringStateTransferTest.java:75)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:715)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:907)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1237)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> Test suite progress: tests succeeded: 1536, failed: 1, skipped: 0.
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 3 months
- 1
- 0
[JBoss JIRA] (ISPN-3448) Tests from org.infinispan.distribution.rehash package fail randomly
by Vitalii Chepeliuk (JIRA)
[ https://issues.jboss.org/browse/ISPN-3448?page=com.atlassian.jira.plugin.... ]
Vitalii Chepeliuk updated ISPN-3448:
------------------------------------
Attachment: NonTxStateTransferOverwritingValue2Test.log.zip
Add trace logs for
org.infinispan.distribution.rehash.NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringPutOverwrite
org.infinispan.distribution.rehash.NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringRemoveWithPreviousValue
org.infinispan.distribution.rehash.NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringReplace
> Tests from org.infinispan.distribution.rehash package fail randomly
> -------------------------------------------------------------------
>
> Key: ISPN-3448
> URL: https://issues.jboss.org/browse/ISPN-3448
> Project: Infinispan
> Issue Type: Bug
> Components: Core API
> Affects Versions: 6.0.0.Alpha3
> Reporter: Jiří Holuša
> Assignee: Mircea Markus
> Labels: testsuite_stability
> Attachments: NonTxPutIfAbsentDuringLeaveStressTest.testNodeLeavingDuringPutIfAbsent.log.zip, NonTxStateTransferOverwritingValue2Test.log.zip, NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringPutOverwrite.log.zip, NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringRemove.log.zip, NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuring{Remove, Replace}WithPreviousValue.log.zip
>
>
> These tests are randomly failing on various platforms (RHEL, Solaris) with various JDKs.
> org.infinispan.distribution.rehash.NonTxPutIfAbsentDuringLeaveStressTest.testNodeLeavingDuringPutIfAbsent
> org.infinispan.distribution.rehash.NonTxJoinerBecomingBackupOwnerTest.testBackupOwnerJoiningDuringPutIfAbsent
> org.infinispan.distribution.rehash.NonTxPutIfAbsentDuringJoinStressTest.testNodeJoiningDuringPutIfAbsent
> org.infinispan.distribution.rehash.NonTxJoinerBecomingBackupOwnerTest.testBackupOwnerJoiningDuringPut
> org.infinispan.distribution.rehash.NonTxPutIfAbsentDuringLeaveStressTest.testNodeLeavingDuringPutIfAbsent
> org.infinispan.distribution.rehash.NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringRemove
> org.infinispan.distribution.rehash.NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringPutOverwrite
> org.infinispan.distribution.rehash.NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringRemoveWithPreviousValue
> org.infinispan.distribution.rehash.NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringReplaceWithPreviousValue
> org.infinispan.distribution.rehash.NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringReplace
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 3 months
- 1
- 0
[JBoss JIRA] (ISPN-3448) Tests from org.infinispan.distribution.rehash package fail randomly
by Vitalii Chepeliuk (JIRA)
[ https://issues.jboss.org/browse/ISPN-3448?page=com.atlassian.jira.plugin.... ]
Vitalii Chepeliuk updated ISPN-3448:
------------------------------------
Attachment: (was: NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringReplace.log.zip)
> Tests from org.infinispan.distribution.rehash package fail randomly
> -------------------------------------------------------------------
>
> Key: ISPN-3448
> URL: https://issues.jboss.org/browse/ISPN-3448
> Project: Infinispan
> Issue Type: Bug
> Components: Core API
> Affects Versions: 6.0.0.Alpha3
> Reporter: Jiří Holuša
> Assignee: Mircea Markus
> Labels: testsuite_stability
> Attachments: NonTxPutIfAbsentDuringLeaveStressTest.testNodeLeavingDuringPutIfAbsent.log.zip, NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringPutOverwrite.log.zip, NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringRemove.log.zip, NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuring{Remove, Replace}WithPreviousValue.log.zip
>
>
> These tests are randomly failing on various platforms (RHEL, Solaris) with various JDKs.
> org.infinispan.distribution.rehash.NonTxPutIfAbsentDuringLeaveStressTest.testNodeLeavingDuringPutIfAbsent
> org.infinispan.distribution.rehash.NonTxJoinerBecomingBackupOwnerTest.testBackupOwnerJoiningDuringPutIfAbsent
> org.infinispan.distribution.rehash.NonTxPutIfAbsentDuringJoinStressTest.testNodeJoiningDuringPutIfAbsent
> org.infinispan.distribution.rehash.NonTxJoinerBecomingBackupOwnerTest.testBackupOwnerJoiningDuringPut
> org.infinispan.distribution.rehash.NonTxPutIfAbsentDuringLeaveStressTest.testNodeLeavingDuringPutIfAbsent
> org.infinispan.distribution.rehash.NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringRemove
> org.infinispan.distribution.rehash.NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringPutOverwrite
> org.infinispan.distribution.rehash.NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringRemoveWithPreviousValue
> org.infinispan.distribution.rehash.NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringReplaceWithPreviousValue
> org.infinispan.distribution.rehash.NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringReplace
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 3 months
- 1
- 0
[JBoss JIRA] (ISPN-3448) Tests from org.infinispan.distribution.rehash package fail randomly
by Vitalii Chepeliuk (JIRA)
[ https://issues.jboss.org/browse/ISPN-3448?page=com.atlassian.jira.plugin.... ]
Vitalii Chepeliuk updated ISPN-3448:
------------------------------------
Attachment: NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringReplace.log.zip
Add trace log for 3 tests
org.infinispan.distribution.rehash.NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringPutOverwrite
org.infinispan.distribution.rehash.NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringRemoveWithPreviousValue
org.infinispan.distribution.rehash.NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringReplace
> Tests from org.infinispan.distribution.rehash package fail randomly
> -------------------------------------------------------------------
>
> Key: ISPN-3448
> URL: https://issues.jboss.org/browse/ISPN-3448
> Project: Infinispan
> Issue Type: Bug
> Components: Core API
> Affects Versions: 6.0.0.Alpha3
> Reporter: Jiří Holuša
> Assignee: Mircea Markus
> Labels: testsuite_stability
> Attachments: NonTxPutIfAbsentDuringLeaveStressTest.testNodeLeavingDuringPutIfAbsent.log.zip, NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringPutOverwrite.log.zip, NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringRemove.log.zip, NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuring{Remove, Replace}WithPreviousValue.log.zip
>
>
> These tests are randomly failing on various platforms (RHEL, Solaris) with various JDKs.
> org.infinispan.distribution.rehash.NonTxPutIfAbsentDuringLeaveStressTest.testNodeLeavingDuringPutIfAbsent
> org.infinispan.distribution.rehash.NonTxJoinerBecomingBackupOwnerTest.testBackupOwnerJoiningDuringPutIfAbsent
> org.infinispan.distribution.rehash.NonTxPutIfAbsentDuringJoinStressTest.testNodeJoiningDuringPutIfAbsent
> org.infinispan.distribution.rehash.NonTxJoinerBecomingBackupOwnerTest.testBackupOwnerJoiningDuringPut
> org.infinispan.distribution.rehash.NonTxPutIfAbsentDuringLeaveStressTest.testNodeLeavingDuringPutIfAbsent
> org.infinispan.distribution.rehash.NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringRemove
> org.infinispan.distribution.rehash.NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringPutOverwrite
> org.infinispan.distribution.rehash.NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringRemoveWithPreviousValue
> org.infinispan.distribution.rehash.NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringReplaceWithPreviousValue
> org.infinispan.distribution.rehash.NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringReplace
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 3 months
- 1
- 0
[JBoss JIRA] (ISPN-3448) Tests from org.infinispan.distribution.rehash package fail randomly
by Vitalii Chepeliuk (JIRA)
[ https://issues.jboss.org/browse/ISPN-3448?page=com.atlassian.jira.plugin.... ]
Vitalii Chepeliuk updated ISPN-3448:
------------------------------------
Comment: was deleted
(was: Add trace log for 3 tests
org.infinispan.distribution.rehash.NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringPutOverwrite
org.infinispan.distribution.rehash.NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringRemoveWithPreviousValue
org.infinispan.distribution.rehash.NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringReplace)
> Tests from org.infinispan.distribution.rehash package fail randomly
> -------------------------------------------------------------------
>
> Key: ISPN-3448
> URL: https://issues.jboss.org/browse/ISPN-3448
> Project: Infinispan
> Issue Type: Bug
> Components: Core API
> Affects Versions: 6.0.0.Alpha3
> Reporter: Jiří Holuša
> Assignee: Mircea Markus
> Labels: testsuite_stability
> Attachments: NonTxPutIfAbsentDuringLeaveStressTest.testNodeLeavingDuringPutIfAbsent.log.zip, NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringPutOverwrite.log.zip, NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringRemove.log.zip, NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuring{Remove, Replace}WithPreviousValue.log.zip
>
>
> These tests are randomly failing on various platforms (RHEL, Solaris) with various JDKs.
> org.infinispan.distribution.rehash.NonTxPutIfAbsentDuringLeaveStressTest.testNodeLeavingDuringPutIfAbsent
> org.infinispan.distribution.rehash.NonTxJoinerBecomingBackupOwnerTest.testBackupOwnerJoiningDuringPutIfAbsent
> org.infinispan.distribution.rehash.NonTxPutIfAbsentDuringJoinStressTest.testNodeJoiningDuringPutIfAbsent
> org.infinispan.distribution.rehash.NonTxJoinerBecomingBackupOwnerTest.testBackupOwnerJoiningDuringPut
> org.infinispan.distribution.rehash.NonTxPutIfAbsentDuringLeaveStressTest.testNodeLeavingDuringPutIfAbsent
> org.infinispan.distribution.rehash.NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringRemove
> org.infinispan.distribution.rehash.NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringPutOverwrite
> org.infinispan.distribution.rehash.NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringRemoveWithPreviousValue
> org.infinispan.distribution.rehash.NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringReplaceWithPreviousValue
> org.infinispan.distribution.rehash.NonTxStateTransferOverwritingValue2Test.testBackupOwnerJoiningDuringReplace
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 3 months
- 1
- 0