[JBoss JIRA] (ISPN-2582) When a node rejoins a cluster , the existing node freezes for a minute
by Shay Matasaro (JIRA)
Shay Matasaro created ISPN-2582:
-----------------------------------
Summary: When a node rejoins a cluster , the existing node freezes for a minute
Key: ISPN-2582
URL: https://issues.jboss.org/browse/ISPN-2582
Project: Infinispan
Issue Type: Bug
Environment: EAP 6.0.0.GA
Reporter: Shay Matasaro
Assignee: Mircea Markus
1) 2 node cluster
2) distributable web app
3) run jemeter with 10 threads
4) start node 1
5) start node 2
6) shutdown node 1
7) restart node 1
result:
node 2 freezes for one minute then throws exceptions exceptions then continues
15:25:19,478 ERROR [org.infinispan.cacheviews.CacheViewsManagerImpl] (CacheViewInstaller-5,master:server-two/web) ISPN000172: Failed to prepare view CacheView{viewId=4, members=[master:server-two/web, master:server-one/web]} for cache default-host/demo7, rolling back to view CacheView{viewId=3, members=[master:server-two/web]}: java.util.concurrent.TimeoutException
15:25:19,688 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (ajp-/0.0.0.0:8159-3) ISPN000136: Execution error: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4
15:25:19,687 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (ajp-/0.0.0.0:8159-6) ISPN000136: Execution error: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4
15:25:19,687 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (ajp-/0.0.0.0:8159-1) ISPN000136: Execution error: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4
15:25:19,688 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (ajp-/0.0.0.0:8159-9) ISPN000136: Execution error: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4
15:25:19,685 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (ajp-/0.0.0.0:8159-2) ISPN000136: Execution error: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4
15:25:19,688 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (ajp-/0.0.0.0:8159-5) ISPN000136: Execution error: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4
15:25:19,686 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (ajp-/0.0.0.0:8159-4) ISPN000136: Execution error: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4
15:25:19,688 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (ajp-/0.0.0.0:8159-10) ISPN000136: Execution error: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4
15:25:19,688 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (ajp-/0.0.0.0:8159-12) ISPN000136: Execution error: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4
15:25:19,696 ERROR [org.infinispan.transaction.TransactionCoordinator] (ajp-/0.0.0.0:8159-3) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4
15:25:19,689 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (ajp-/0.0.0.0:8159-11) ISPN000136: Execution error: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4
15:25:19,705 ERROR [org.infinispan.transaction.TransactionCoordinator] (ajp-/0.0.0.0:8159-4) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4
15:25:19,706 ERROR [org.infinispan.transaction.TransactionCoordinator] (ajp-/0.0.0.0:8159-10) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4
15:25:19,699 ERROR [org.infinispan.transaction.TransactionCoordinator] (ajp-/0.0.0.0:8159-1) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4
15:25:19,696 ERROR [org.infinispan.transaction.TransactionCoordinator] (ajp-/0.0.0.0:8159-6) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4
15:25:19,714 ERROR [org.infinispan.transaction.tm.DummyTransaction] (ajp-/0.0.0.0:8159-10) ISPN000111: afterCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, transaction=DummyTransaction{xid=DummyXid{id=34869}, status=3}, lockedKeys=null, backupKeyLocks=null, viewId=3} org.infinispan.transaction.synchronization.SyncLocalTransaction@8834} org.infinispan.transaction.synchronization.SynchronizationAdapter@8853: org.infinispan.CacheException: Could not commit.
Caused by: javax.transaction.xa.XAException
15:25:19,708 ERROR [org.infinispan.transaction.tm.DummyTransaction] (ajp-/0.0.0.0:8159-3) ISPN000111: afterCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, transaction=DummyTransaction{xid=DummyXid{id=34867}, status=3}, lockedKeys=null, backupKeyLocks=null, viewId=3} org.infinispan.transaction.synchronization.SyncLocalTransaction@8832} org.infinispan.transaction.synchronization.SynchronizationAdapter@8851: org.infinispan.CacheException: Could not commit.
Caused by: javax.transaction.xa.XAException
15:25:19,717 ERROR [org.infinispan.transaction.tm.DummyTransaction] (ajp-/0.0.0.0:8159-6) ISPN000111: afterCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, transaction=DummyTransaction{xid=DummyXid{id=34864}, status=3}, lockedKeys=null, backupKeyLocks=null, viewId=3} org.infinispan.transaction.synchronization.SyncLocalTransaction@882f} org.infinispan.transaction.synchronization.SynchronizationAdapter@884e: org.infinispan.CacheException: Could not commit.
Caused by: javax.transaction.xa.XAException
15:25:19,699 ERROR [org.infinispan.transaction.TransactionCoordinator] (ajp-/0.0.0.0:8159-9) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4
15:25:19,704 ERROR [org.infinispan.transaction.TransactionCoordinator] (ajp-/0.0.0.0:8159-5) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4
15:25:19,707 ERROR [org.infinispan.transaction.TransactionCoordinator] (ajp-/0.0.0.0:8159-12) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4
15:25:19,704 ERROR [org.infinispan.transaction.TransactionCoordinator] (ajp-/0.0.0.0:8159-2) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4
15:25:19,722 ERROR [org.infinispan.transaction.tm.DummyTransaction] (ajp-/0.0.0.0:8159-5) ISPN000111: afterCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, transaction=DummyTransaction{xid=DummyXid{id=34866}, status=3}, lockedKeys=null, backupKeyLocks=null, viewId=3} org.infinispan.transaction.synchronization.SyncLocalTransaction@8831} org.infinispan.transaction.synchronization.SynchronizationAdapter@8850: org.infinispan.CacheException: Could not commit.
Caused by: javax.transaction.xa.XAException
15:25:19,721 ERROR [org.infinispan.transaction.tm.DummyTransaction] (ajp-/0.0.0.0:8159-9) ISPN000111: afterCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, transaction=DummyTransaction{xid=DummyXid{id=34870}, status=3}, lockedKeys=null, backupKeyLocks=null, viewId=3} org.infinispan.transaction.synchronization.SyncLocalTransaction@8835} org.infinispan.transaction.synchronization.SynchronizationAdapter@8854: org.infinispan.CacheException: Could not commit.
Caused by: javax.transaction.xa.XAException
15:25:19,712 ERROR [org.infinispan.transaction.TransactionCoordinator] (ajp-/0.0.0.0:8159-11) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 4
15:25:19,723 ERROR [org.infinispan.transaction.tm.DummyTransaction] (ajp-/0.0.0.0:8159-12) ISPN000111: afterCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, transaction=DummyTransaction{xid=DummyXid{id=34868}, status=3}, lockedKeys=null, backupKeyLocks=null, viewId=3} org.infinispan.transaction.synchronization.SyncLocalTransaction@8833} org.infinispan.transaction.synchronization.SynchronizationAdapter@8852: org.infinispan.CacheException: Could not commit.
Caused by: javax.transaction.xa.XAException
15:25:19,712 ERROR [org.infinispan.transaction.tm.DummyTransaction] (ajp-/0.0.0.0:8159-4) ISPN000111: afterCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, transaction=DummyTransaction{xid=DummyXid{id=34863}, status=3}, lockedKeys=null, backupKeyLocks=null, viewId=3} org.infinispan.transaction.synchronization.SyncLocalTransaction@882e} org.infinispan.transaction.synchronization.SynchronizationAdapter@884d: org.infinispan.CacheException: Could not commit.
Caused by: javax.transaction.xa.XAException
15:25:19,724 ERROR [org.infinispan.transaction.tm.DummyTransaction] (ajp-/0.0.0.0:8159-2) ISPN000111: afterCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, transaction=DummyTransaction{xid=DummyXid{id=34862}, status=3}, lockedKeys=null, backupKeyLocks=null, viewId=3} org.infinispan.transaction.synchronization.SyncLocalTransaction@882d} org.infinispan.transaction.synchronization.SynchronizationAdapter@884c: org.infinispan.CacheException: Could not commit.
Caused by: javax.transaction.xa.XAException
15:25:19,715 ERROR [org.infinispan.transaction.tm.DummyTransaction] (ajp-/0.0.0.0:8159-1) ISPN000111: afterCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, transaction=DummyTransaction{xid=DummyXid{id=34865}, status=3}, lockedKeys=null, backupKeyLocks=null, viewId=3} org.infinispan.transaction.synchronization.SyncLocalTransaction@8830} org.infinispan.transaction.synchronization.SynchronizationAdapter@884f: org.infinispan.CacheException: Could not commit.
Caused by: javax.transaction.xa.XAException
15:25:19,726 ERROR [org.infinispan.transaction.tm.DummyTransaction] (ajp-/0.0.0.0:8159-11) ISPN000111: afterCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, transaction=DummyTransaction{xid=DummyXid{id=34871}, status=3}, lockedKeys=null, backupKeyLocks=null, viewId=3} org.infinispan.transaction.synchronization.SyncLocalTransaction@8836} org.infinispan.transaction.synchronization.SynchronizationAdapter@8855: org.infinispan.CacheException: Could not commit.
Caused by: javax.transaction.xa.XAException
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 5 months
[JBoss JIRA] (ISPN-2572) "CacheException: Initial state transfer timed out for cache" reliably on AS7 testsuite
by Radoslav Husar (JIRA)
Radoslav Husar created ISPN-2572:
------------------------------------
Summary: "CacheException: Initial state transfer timed out for cache" reliably on AS7 testsuite
Key: ISPN-2572
URL: https://issues.jboss.org/browse/ISPN-2572
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Beta4
Reporter: Radoslav Husar
Assignee: Dan Berindei
Priority: Blocker
Fix For: 5.2.0.CR1
While running AS7 testsuite with speedups implemented in my branch (https://github.com/jbossas/jboss-as/pull/3381) we are contantly seeing (log below) on Windows 2008.
Run:
http://teamcity.cafe-babe.org/viewLog.html?buildId=1689&tab=buildResultsD...
{code}
16:34:46,092 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 13) MSC00001: Failed to start service jboss.infinispan.ejb.remote-connector-client-mappings: org.jboss.msc.service.StartException in service jboss.infinispan.ejb.remote-connector-client-mappings: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:87)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_32]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_32]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_32]
at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.0.0.GA.jar:2.0.0.GA]
Caused by: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:205)
at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:883)
at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:654)
at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:643)
at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:546)
at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:199)
at org.infinispan.CacheImpl.start(CacheImpl.java:520)
at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:690)
at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:653)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:549)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:563)
at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:107)
at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:98)
at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78)
at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:82)
... 4 more
Caused by: org.infinispan.CacheException: Initial state transfer timed out for cache remote-connector-client-mappings on node-1/ejb
at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:209)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_32]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_32]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_32]
at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_32]
at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203)
... 18 more
{code}
Affected version -- current master (say 7dc531002539b078e429418d8ef204e401beafd1).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 5 months
[JBoss JIRA] (ISPN-2453) Locks not cleaned up - timeouts on write operations after contention is over
by Sanne Grinovero (JIRA)
Sanne Grinovero created ISPN-2453:
-------------------------------------
Summary: Locks not cleaned up - timeouts on write operations after contention is over
Key: ISPN-2453
URL: https://issues.jboss.org/browse/ISPN-2453
Project: Infinispan
Issue Type: Bug
Components: Locking and Concurrency
Reporter: Sanne Grinovero
Assignee: Manik Surtani
Priority: Blocker
Fix For: 5.2.0.Beta3
By artificially creating contention on a write operation which needs to be serialized among different threads, we get timeout exceptions of "could not acquire lock after 10 seconds" even if the contention happens very briefly and then the pending writer is given all the time without other threads disturbing it.
Created a stresstest to show it: ReplaceOperationStressTest
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 5 months
[JBoss JIRA] (ISPN-2564) Scope views to caches rather than entire cluster
by Vladimir Blagojevic (JIRA)
Vladimir Blagojevic created ISPN-2564:
-----------------------------------------
Summary: Scope views to caches rather than entire cluster
Key: ISPN-2564
URL: https://issues.jboss.org/browse/ISPN-2564
Project: Infinispan
Issue Type: Bug
Components: Core API
Affects Versions: 5.2.0.Beta4
Reporter: Vladimir Blagojevic
Assignee: Dan Berindei
Fix For: 5.2.0.CR1, 5.2.0.Final
Currently our codebase relies on the cache views to be equal to entire cluster, however, this might not be the case due to asymmetric cluster; certain cache instances running only on particular Infinispan nodes. We have to make sure that cache views, if needed are scoped properly to particular cache rather than entire cluster
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 5 months
[JBoss JIRA] (ISPN-2250) Handle all the remaining comments for the NBST pull reques
by Mircea Markus (JIRA)
Mircea Markus created ISPN-2250:
-----------------------------------
Summary: Handle all the remaining comments for the NBST pull reques
Key: ISPN-2250
URL: https://issues.jboss.org/browse/ISPN-2250
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Alpha3
Reporter: Mircea Markus
Assignee: Adrian Nistor
Priority: Critical
Fix For: 5.2.0.Alpha4
These are still visible even after the pull request is closed.
Dan, please don't delete the t_nbst branch before handling these as we might loose them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 5 months
[JBoss JIRA] (ISPN-2501) State transfer optimizations
by Adrian Nistor (JIRA)
Adrian Nistor created ISPN-2501:
-----------------------------------
Summary: State transfer optimizations
Key: ISPN-2501
URL: https://issues.jboss.org/browse/ISPN-2501
Project: Infinispan
Issue Type: Feature Request
Components: State transfer
Affects Versions: 5.2.0.Beta3
Reporter: Adrian Nistor
Assignee: Adrian Nistor
Fix For: 5.2.0.CR1
There are two obvious optimizations possible in the code that handles installation of a new topology.
1. Currently a new topology is not confirmed until the node successfully sends START_STATE_TRANSFER requests to all nodes it wants to fetch segments from. This does not wait for the actual data to arrive but it still blocks quite a lot. To fix this we need to confirm the topology right after transactions were received.
2. Fetching transactions from other members is done serially. To improve it we could split it into multiple concurrent tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 5 months