Shay Matasaro created ISPN-2582: ----------------------------------- Summary: When a node rejoins a cluster , the existing node freezes for a minute Key: ISPN-2582 URL: https://issues.jboss.org/browse/ISPN-2582 Project: Infinispan Issue Type: Bug Environment: EAP 6.0.0.GA Reporter: Shay Matasaro Assignee: Mircea Markus 1) 2 node cluster 2) distributable web app 3) run jemeter with 10 threads 4) start node 1 5) start node 2 6) shutdown node 1 7) restart node 1 result: node 2 freezes for one minute then throws exceptions exceptions then continues 15:25:19,478 ERROR [org.infinispan.cacheviews.CacheViewsManagerImpl] (CacheViewInstaller-5,master:server-two/web) ISPN000172: Failed to prepare view CacheView{viewId=4, members=[master:server-two/web, master:server-one/web]} for cache default-host/demo7, rolling back to view CacheView{viewId=3, members=[master:server-two/web]}: java.util.concurrent.TimeoutException 15:25:19,688 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (ajp-/0.0.0.0:8159-3) ISPN000136: Execution error: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4 15:25:19,687 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (ajp-/0.0.0.0:8159-6) ISPN000136: Execution error: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4 15:25:19,687 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (ajp-/0.0.0.0:8159-1) ISPN000136: Execution error: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4 15:25:19,688 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (ajp-/0.0.0.0:8159-9) ISPN000136: Execution error: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4 15:25:19,685 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (ajp-/0.0.0.0:8159-2) ISPN000136: Execution error: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4 15:25:19,688 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (ajp-/0.0.0.0:8159-5) ISPN000136: Execution error: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4 15:25:19,686 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (ajp-/0.0.0.0:8159-4) ISPN000136: Execution error: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4 15:25:19,688 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (ajp-/0.0.0.0:8159-10) ISPN000136: Execution error: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4 15:25:19,688 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (ajp-/0.0.0.0:8159-12) ISPN000136: Execution error: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4 15:25:19,696 ERROR [org.infinispan.transaction.TransactionCoordinator] (ajp-/0.0.0.0:8159-3) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4 15:25:19,689 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (ajp-/0.0.0.0:8159-11) ISPN000136: Execution error: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4 15:25:19,705 ERROR [org.infinispan.transaction.TransactionCoordinator] (ajp-/0.0.0.0:8159-4) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4 15:25:19,706 ERROR [org.infinispan.transaction.TransactionCoordinator] (ajp-/0.0.0.0:8159-10) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4 15:25:19,699 ERROR [org.infinispan.transaction.TransactionCoordinator] (ajp-/0.0.0.0:8159-1) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4 15:25:19,696 ERROR [org.infinispan.transaction.TransactionCoordinator] (ajp-/0.0.0.0:8159-6) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4 15:25:19,714 ERROR [org.infinispan.transaction.tm.DummyTransaction] (ajp-/0.0.0.0:8159-10) ISPN000111: afterCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, transaction=DummyTransaction{xid=DummyXid{id=34869}, status=3}, lockedKeys=null, backupKeyLocks=null, viewId=3} org.infinispan.transaction.synchronization.SyncLocalTransaction@8834} org.infinispan.transaction.synchronization.SynchronizationAdapter@8853: org.infinispan.CacheException: Could not commit. Caused by: javax.transaction.xa.XAException 15:25:19,708 ERROR [org.infinispan.transaction.tm.DummyTransaction] (ajp-/0.0.0.0:8159-3) ISPN000111: afterCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, transaction=DummyTransaction{xid=DummyXid{id=34867}, status=3}, lockedKeys=null, backupKeyLocks=null, viewId=3} org.infinispan.transaction.synchronization.SyncLocalTransaction@8832} org.infinispan.transaction.synchronization.SynchronizationAdapter@8851: org.infinispan.CacheException: Could not commit. Caused by: javax.transaction.xa.XAException 15:25:19,717 ERROR [org.infinispan.transaction.tm.DummyTransaction] (ajp-/0.0.0.0:8159-6) ISPN000111: afterCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, transaction=DummyTransaction{xid=DummyXid{id=34864}, status=3}, lockedKeys=null, backupKeyLocks=null, viewId=3} org.infinispan.transaction.synchronization.SyncLocalTransaction@882f} org.infinispan.transaction.synchronization.SynchronizationAdapter@884e: org.infinispan.CacheException: Could not commit. Caused by: javax.transaction.xa.XAException 15:25:19,699 ERROR [org.infinispan.transaction.TransactionCoordinator] (ajp-/0.0.0.0:8159-9) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4 15:25:19,704 ERROR [org.infinispan.transaction.TransactionCoordinator] (ajp-/0.0.0.0:8159-5) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4 15:25:19,707 ERROR [org.infinispan.transaction.TransactionCoordinator] (ajp-/0.0.0.0:8159-12) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4 15:25:19,704 ERROR [org.infinispan.transaction.TransactionCoordinator] (ajp-/0.0.0.0:8159-2) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4 15:25:19,722 ERROR [org.infinispan.transaction.tm.DummyTransaction] (ajp-/0.0.0.0:8159-5) ISPN000111: afterCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, transaction=DummyTransaction{xid=DummyXid{id=34866}, status=3}, lockedKeys=null, backupKeyLocks=null, viewId=3} org.infinispan.transaction.synchronization.SyncLocalTransaction@8831} org.infinispan.transaction.synchronization.SynchronizationAdapter@8850: org.infinispan.CacheException: Could not commit. Caused by: javax.transaction.xa.XAException 15:25:19,721 ERROR [org.infinispan.transaction.tm.DummyTransaction] (ajp-/0.0.0.0:8159-9) ISPN000111: afterCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, transaction=DummyTransaction{xid=DummyXid{id=34870}, status=3}, lockedKeys=null, backupKeyLocks=null, viewId=3} org.infinispan.transaction.synchronization.SyncLocalTransaction@8835} org.infinispan.transaction.synchronization.SynchronizationAdapter@8854: org.infinispan.CacheException: Could not commit. Caused by: javax.transaction.xa.XAException 15:25:19,712 ERROR [org.infinispan.transaction.TransactionCoordinator] (ajp-/0.0.0.0:8159-11) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4 15:25:19,723 ERROR [org.infinispan.transaction.tm.DummyTransaction] (ajp-/0.0.0.0:8159-12) ISPN000111: afterCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, transaction=DummyTransaction{xid=DummyXid{id=34868}, status=3}, lockedKeys=null, backupKeyLocks=null, viewId=3} org.infinispan.transaction.synchronization.SyncLocalTransaction@8833} org.infinispan.transaction.synchronization.SynchronizationAdapter@8852: org.infinispan.CacheException: Could not commit. Caused by: javax.transaction.xa.XAException 15:25:19,712 ERROR [org.infinispan.transaction.tm.DummyTransaction] (ajp-/0.0.0.0:8159-4) ISPN000111: afterCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, transaction=DummyTransaction{xid=DummyXid{id=34863}, status=3}, lockedKeys=null, backupKeyLocks=null, viewId=3} org.infinispan.transaction.synchronization.SyncLocalTransaction@882e} org.infinispan.transaction.synchronization.SynchronizationAdapter@884d: org.infinispan.CacheException: Could not commit. Caused by: javax.transaction.xa.XAException 15:25:19,724 ERROR [org.infinispan.transaction.tm.DummyTransaction] (ajp-/0.0.0.0:8159-2) ISPN000111: afterCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, transaction=DummyTransaction{xid=DummyXid{id=34862}, status=3}, lockedKeys=null, backupKeyLocks=null, viewId=3} org.infinispan.transaction.synchronization.SyncLocalTransaction@882d} org.infinispan.transaction.synchronization.SynchronizationAdapter@884c: org.infinispan.CacheException: Could not commit. Caused by: javax.transaction.xa.XAException 15:25:19,715 ERROR [org.infinispan.transaction.tm.DummyTransaction] (ajp-/0.0.0.0:8159-1) ISPN000111: afterCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, transaction=DummyTransaction{xid=DummyXid{id=34865}, status=3}, lockedKeys=null, backupKeyLocks=null, viewId=3} org.infinispan.transaction.synchronization.SyncLocalTransaction@8830} org.infinispan.transaction.synchronization.SynchronizationAdapter@884f: org.infinispan.CacheException: Could not commit. Caused by: javax.transaction.xa.XAException 15:25:19,726 ERROR [org.infinispan.transaction.tm.DummyTransaction] (ajp-/0.0.0.0:8159-11) ISPN000111: afterCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, transaction=DummyTransaction{xid=DummyXid{id=34871}, status=3}, lockedKeys=null, backupKeyLocks=null, viewId=3} org.infinispan.transaction.synchronization.SyncLocalTransaction@8836} org.infinispan.transaction.synchronization.SynchronizationAdapter@8855: org.infinispan.CacheException: Could not commit. Caused by: javax.transaction.xa.XAException -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira