[JBoss JIRA] (AS7-2467) TimeoutException on node joining
by Radoslav Husar (Created) (JIRA)
TimeoutException on node joining
---------------------------------
Key: AS7-2467
URL: https://issues.jboss.org/browse/AS7-2467
Project: Application Server 7
Issue Type: Bug
Components: Clustering
Affects Versions: No Release
Environment: build is current head
Reporter: Radoslav Husar
Assignee: Paul Ferraro
Priority: Critical
Fix For: 7.1.0.Alpha1
The scenario was like this (manually)
# start 1 node
# start 2nd node
# when the node is joining (kind of hard to target the right time) post a request
This is all taking place in high perf lab without load so the timeout is not not anticipated.
{code}
10:35:05,501 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-4,null) ISPN000094: Received new cluster view: [perf18/web|1] [perf18/web, perf19/web]
10:35:26,055 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (ajp-perf18-10.16.90.54-8009-24) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [15 seconds] on key [//default-host/null/Ww+5NaIAnFIoXWEuhszfGqNb] for requestor [GlobalTransaction:<perf18/web>:87:local]! Lock held by [GlobalTransaction:<perf19/web>:1:local]
at org.infinispan.util.concurrent.locks.LockManagerImpl.lock(LockManagerImpl.java:207) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.util.concurrent.locks.LockManagerImpl.acquireLockNoCheck(LockManagerImpl.java:190) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockKey(AbstractLockingInterceptor.java:111) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.locking.OptimisticLockingInterceptor$LockAquisitionVisitor.visitPutKeyValueCommand(OptimisticLockingInterceptor.java:162) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:63) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:49) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:133) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:108) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:104) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:133) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:108) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.StateTransferLockInterceptor.visitPrepareCommand(StateTransferLockInterceptor.java:62) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:133) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:108) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:104) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:64) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:108) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:133) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:108) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:77) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:108) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:133) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.jboss.as.clustering.infinispan.ClassLoaderAwareCache$ClassLoaderAwareCommandInterceptor.handleDefault(ClassLoaderAwareCache.java:111) [jboss-as-clustering-infinispan-7.1.0.Alpha2-SNAPSHOT.jar:7.1.0.Alpha2-SNAPSHOT]
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:108) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:318) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:124) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:73) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:117)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:403)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:104)
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:159)
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1159)
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:119)
at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
at org.jboss.as.clustering.web.impl.TransactionBatchingManager.endBatch(TransactionBatchingManager.java:75) [jboss-as-clustering-web-spi-7.1.0.Alpha2-SNAPSHOT.jar:7.1.0.Alpha2-SNAPSHOT]
at org.jboss.as.web.session.DistributableSessionManager.processSessionRepl(DistributableSessionManager.java:1547)
at org.jboss.as.web.session.DistributableSessionManager.storeSession(DistributableSessionManager.java:877)
at org.jboss.as.web.session.InstantSnapshotManager.snapshot(InstantSnapshotManager.java:45)
at org.jboss.as.web.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:131)
at org.jboss.as.web.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:89)
at org.jboss.as.web.session.JvmRouteValve.invoke(JvmRouteValve.java:88)
at org.jboss.as.web.session.LockingValve.invoke(LockingValve.java:56)
at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:155)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:362)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505)
at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930)
at java.lang.Thread.run(Thread.java:619) [:1.6.0_20]
10:35:26,065 ERROR [org.infinispan.transaction.TransactionCoordinator] (ajp-perf18-10.16.90.54-8009-24) ISPN000097: Error while processing 1PC PrepareCommand: org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [15 seconds] on key [//default-host/null/Ww+5NaIAnFIoXWEuhszfGqNb] for requestor [GlobalTransaction:<perf18/web>:87:local]! Lock held by [GlobalTransaction:<perf19/web>:1:local]
at org.infinispan.util.concurrent.locks.LockManagerImpl.lock(LockManagerImpl.java:207)
at org.infinispan.util.concurrent.locks.LockManagerImpl.acquireLockNoCheck(LockManagerImpl.java:190)
at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockKey(AbstractLockingInterceptor.java:111)
at org.infinispan.interceptors.locking.OptimisticLockingInterceptor$LockAquisitionVisitor.visitPutKeyValueCommand(OptimisticLockingInterceptor.java:162)
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:63)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)
at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:49)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:133)
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:108)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)
at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:104)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:133)
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:108)
at org.infinispan.interceptors.StateTransferLockInterceptor.visitPrepareCommand(StateTransferLockInterceptor.java:62)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:133)
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:108)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:104)
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:64)
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:108)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:133)
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:108)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)
at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:77)
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:108)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:133)
at org.jboss.as.clustering.infinispan.ClassLoaderAwareCache$ClassLoaderAwareCommandInterceptor.handleDefault(ClassLoaderAwareCache.java:111)
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:108)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:318)
at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:124)
at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:73)
at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:117)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:403)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:104)
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:159)
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1159)
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:119)
at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
at org.jboss.as.clustering.web.impl.TransactionBatchingManager.endBatch(TransactionBatchingManager.java:75)
at org.jboss.as.web.session.DistributableSessionManager.processSessionRepl(DistributableSessionManager.java:1547) [jboss-as-web-7.1.0.Alpha2-SNAPSHOT.jar:7.1.0.Alpha2-SNAPSHOT]
at org.jboss.as.web.session.DistributableSessionManager.storeSession(DistributableSessionManager.java:877) [jboss-as-web-7.1.0.Alpha2-SNAPSHOT.jar:7.1.0.Alpha2-SNAPSHOT]
at org.jboss.as.web.session.InstantSnapshotManager.snapshot(InstantSnapshotManager.java:45) [jboss-as-web-7.1.0.Alpha2-SNAPSHOT.jar:7.1.0.Alpha2-SNAPSHOT]
at org.jboss.as.web.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:131) [jboss-as-web-7.1.0.Alpha2-SNAPSHOT.jar:7.1.0.Alpha2-SNAPSHOT]
at org.jboss.as.web.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:89) [jboss-as-web-7.1.0.Alpha2-SNAPSHOT.jar:7.1.0.Alpha2-SNAPSHOT]
at org.jboss.as.web.session.JvmRouteValve.invoke(JvmRouteValve.java:88) [jboss-as-web-7.1.0.Alpha2-SNAPSHOT.jar:7.1.0.Alpha2-SNAPSHOT]
at org.jboss.as.web.session.LockingValve.invoke(LockingValve.java:56) [jboss-as-web-7.1.0.Alpha2-SNAPSHOT.jar:7.1.0.Alpha2-SNAPSHOT]
at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:155) [jboss-as-web-7.1.0.Alpha2-SNAPSHOT.jar:7.1.0.Alpha2-SNAPSHOT]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.3.Final.jar:7.1.0.Alpha2-SNAPSHOT]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.3.Final.jar:7.1.0.Alpha2-SNAPSHOT]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.3.Final.jar:7.1.0.Alpha2-SNAPSHOT]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:362) [jbossweb-7.0.3.Final.jar:7.1.0.Alpha2-SNAPSHOT]
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.3.Final.jar:7.1.0.Alpha2-SNAPSHOT]
at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [jbossweb-7.0.3.Final.jar:7.1.0.Alpha2-SNAPSHOT]
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.3.Final.jar:7.1.0.Alpha2-SNAPSHOT]
at java.lang.Thread.run(Thread.java:619) [:1.6.0_20]
10:35:26,072 WARN [com.arjuna.ats.jta] (ajp-perf18-10.16.90.54-8009-24) ARJUNA16029: SynchronizationImple.afterCompletion - failed for org.infinispan.transaction.synchronization.SynchronizationAdapter@432398ed with exception: org.infinispan.CacheException: Could not commit.
at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:75)
at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:117)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:403)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:104)
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:159)
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1159)
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:119)
at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
at org.jboss.as.clustering.web.impl.TransactionBatchingManager.endBatch(TransactionBatchingManager.java:75)
at org.jboss.as.web.session.DistributableSessionManager.processSessionRepl(DistributableSessionManager.java:1547) [jboss-as-web-7.1.0.Alpha2-SNAPSHOT.jar:7.1.0.Alpha2-SNAPSHOT]
at org.jboss.as.web.session.DistributableSessionManager.storeSession(DistributableSessionManager.java:877) [jboss-as-web-7.1.0.Alpha2-SNAPSHOT.jar:7.1.0.Alpha2-SNAPSHOT]
at org.jboss.as.web.session.InstantSnapshotManager.snapshot(InstantSnapshotManager.java:45) [jboss-as-web-7.1.0.Alpha2-SNAPSHOT.jar:7.1.0.Alpha2-SNAPSHOT]
at org.jboss.as.web.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:131) [jboss-as-web-7.1.0.Alpha2-SNAPSHOT.jar:7.1.0.Alpha2-SNAPSHOT]
at org.jboss.as.web.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:89) [jboss-as-web-7.1.0.Alpha2-SNAPSHOT.jar:7.1.0.Alpha2-SNAPSHOT]
at org.jboss.as.web.session.JvmRouteValve.invoke(JvmRouteValve.java:88) [jboss-as-web-7.1.0.Alpha2-SNAPSHOT.jar:7.1.0.Alpha2-SNAPSHOT]
at org.jboss.as.web.session.LockingValve.invoke(LockingValve.java:56) [jboss-as-web-7.1.0.Alpha2-SNAPSHOT.jar:7.1.0.Alpha2-SNAPSHOT]
at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:155) [jboss-as-web-7.1.0.Alpha2-SNAPSHOT.jar:7.1.0.Alpha2-SNAPSHOT]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.3.Final.jar:7.1.0.Alpha2-SNAPSHOT]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.3.Final.jar:7.1.0.Alpha2-SNAPSHOT]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.3.Final.jar:7.1.0.Alpha2-SNAPSHOT]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:362) [jbossweb-7.0.3.Final.jar:7.1.0.Alpha2-SNAPSHOT]
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.3.Final.jar:7.1.0.Alpha2-SNAPSHOT]
at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [jbossweb-7.0.3.Final.jar:7.1.0.Alpha2-SNAPSHOT]
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.3.Final.jar:7.1.0.Alpha2-SNAPSHOT]
at java.lang.Thread.run(Thread.java:619) [:1.6.0_20]
Caused by: javax.transaction.xa.XAException
at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:129)
at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:73)
... 24 more
10:35:26,075 WARN [com.arjuna.ats.arjuna] (ajp-perf18-10.16.90.54-8009-24) ARJUNA12127: TwoPhaseCoordinator.afterCompletion - returned failure for SynchronizationImple< 0:ffff0a105a36:784a5ec6:4eb154d1:b6, org.infinispan.transaction.synchronization.SynchronizationAdapter@432398ed >
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 8 months
[JBoss JIRA] Created: (JGRP-1053) UNICAST: set retransmission timeout based on actual retransmission times
by Bela Ban (JIRA)
UNICAST: set retransmission timeout based on actual retransmission times
------------------------------------------------------------------------
Key: JGRP-1053
URL: https://jira.jboss.org/jira/browse/JGRP-1053
Project: JGroups
Issue Type: Feature Request
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.9
UNICAST needs to compute a rolling average of retransmission times, per sender (AckSenderWindow).
The retransmission timeout per sender can then be set based on the actual average retransmission times. The advantage is that we throttle retransmission when we have a lot of message loss, and speed it up again when there are no message drops.
The function to set the timeout should always compute the new timeot value based on (1) the old value times a decay factor and (2) a new value.
The average should go up relatively quickly if the actual retransmission values go up, but come down slowly when the actual values go down.
A potential function is shown below:
static final double SLOW_DECAY_FACTOR=0.9, FAST_DECAY_FACTOR=0.7;
static final double FAST_UP= 1 / FAST_DECAY_FACTOR, SLOW_UP= 1 / SLOW_DECAY_FACTOR;
static final double SAFETY_BUFFER=0.3;
static double avg=200;
public static void main(String[] args) {
final long[] times={200,200,400,400,500,500,500,500,500,100,100,100,100,100,100,100,100,100,100,100,100,100};
// final long[] times={200,200,200,200,200,200,200,200,200,200,200};
for(Long val: times) {
double result=avg(val);
System.out.println(val + ": " + result);
}
}
private static double avg(long val) {
double decay, up;
if(val > avg) {
decay=FAST_DECAY_FACTOR;
up=FAST_UP;
}
else {
decay=SLOW_DECAY_FACTOR;
up=SLOW_UP;
}
double old_val=avg * decay;
double result=(old_val + val * up) / 2;
avg=result;
return result * (1 + SAFETY_BUFFER);
}
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 8 months
[JBoss JIRA] Created: (JGRP-1352) STATE: queue requests during state transfer
by Bela Ban (JIRA)
STATE: queue requests during state transfer
-------------------------------------------
Key: JGRP-1352
URL: https://issues.jboss.org/browse/JGRP-1352
Project: JGroups
Issue Type: Feature Request
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 3.1
STATE should queue requests to be delivered to the application while a state transfer is in progress.
Draft design:
- On STATE-REQ:
- If busy --> add request to queue, return
- Else --> processRequest
- Close MessageQueue and deliver all messages in it to the application
- Kick off a STABLE round, wait until it completes
- If MessageQueue is not empty --> processRequests()
- processRequest():
- Close BARRIER, suspend STABLE
- Get digest
- Open MessageQueue: all new messages will be queued now, rather than delivered to the application
- (Only MSG, not VIEW !)
- Open BARRIER (leave STABLE suspended !)
- Transfer state to state requester
- Resume STABLE
Advantages:
- The state provider is not blocked, e.g. to handle JOIN requests
- Only the state *requester* is blocked, this is OK though
- Between the state requests, the state provider gets a little breathing room to process
queued-up messages, and to purge messages (through stability)
Misc:
- Should the MessageQueue use the disk to handle large message sets ?
- If modifications are idempotent, we *don't* need to queue messages (make this configurable !)
(for example Infinispan ?)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 8 months
[JBoss JIRA] Created: (JGRP-1354) Compress views
by Bela Ban (JIRA)
Compress views
--------------
Key: JGRP-1354
URL: https://issues.jboss.org/browse/JGRP-1354
Project: JGroups
Issue Type: Enhancement
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 3.1
In clusters with many members, a View can be large. See if we can compress views. Possible strategies:
- Canonicalization of addresses (replace UUIDs with shorts) see different JIRA)
- Compress digests (see different JIRA)
- Keep a list of ViewIds and Views and only send the ViewIds (not for JoinRsp !)
- Create a DiffView: send the previous ViewId V1, the new ViewId V2, and the diff between V1 and V2, for example
- V34=V33 + N1 + N2
- V34=V33 - N30 - N13 + N3 (2 nodes left, 1 node joined)
- The receiver can then construct a full view from V34, V33 and the diffs
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 8 months
[JBoss JIRA] (JGRP-1391) Compress MergeViews
by Bela Ban (Created) (JIRA)
Compress MergeViews
-------------------
Key: JGRP-1391
URL: https://issues.jboss.org/browse/JGRP-1391
Project: JGroups
Issue Type: Enhancement
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 3.1
The 'subgroups' field of MergeView is a list of Views, referring to the subclusters that formed the new view.
Each View in that list contains (1) the creator of the view, (2) an ID and (3) a list of member addresses. Both (1) and (3) are present in the new MergeView, so, to save memory and message sizes, 'subgroups' could be compressed: (1) and (3) could be indices into View's 'members' field.
Example: if we have MergeView A|22={A,B,C,D,E,X,Y,Z}, subgroups=A|21={A,B,C}, D|19={D,E}, X|20={X,Y,Z},
then 'subgroups' can be compressed as follows:
subgroups=0|21={0,1,2}, 3|19={3,4}, 5|20={5,6,7}
This is possible because the indices never change, as the contents of Views and MergeViews are static, e.g. we can never add or remove a member: to do this, a new View/MergeView would be created.
Substituting addresses with indices (shorts, possibly using variable-bit encoding) reduces the memory footprint of a view, and the size of a serialized view on the wire.
Getting the subgroups field, e.g. using method getSubgroups() would synthesize the List<View> just-in-time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 8 months
[JBoss JIRA] Created: (AS7-944) EJB lifecycle interceptors that are not included in the deployment are not applied
by Marius Bogoevici (JIRA)
EJB lifecycle interceptors that are not included in the deployment are not applied
----------------------------------------------------------------------------------
Key: AS7-944
URL: https://issues.jboss.org/browse/AS7-944
Project: Application Server 7
Issue Type: Bug
Components: EJB
Affects Versions: 7.0.0.Beta3
Reporter: Marius Bogoevici
Suppose that we have a class
{code}
@Startup @Singleton @Interceptors(A.class, B.class)
public class MyEJB {
}
{code}
Where A.class is included with the deployment and B.class is included in a module that is visible to the deployment.
B will be identified correctly as an interceptor for MyEJB, but LifecycleAnnotationParsingProcessor will not process it (as it apparently handles only components found within the deployment). Therefore, B will be ignored when creating the interceptor set for MyEJB, since ComponentInstallProcessor will treat it as a class that has no lifecycle methods.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 8 months
[JBoss JIRA] Created: (JGRP-1340) GossipRouter: view shows mix of logical address and UUID for nodes
by Bela Ban (JIRA)
GossipRouter: view shows mix of logical address and UUID for nodes
------------------------------------------------------------------
Key: JGRP-1340
URL: https://issues.jboss.org/browse/JGRP-1340
Project: JGroups
Issue Type: Bug
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.12.2, 3.0
[Vivek Sar]
On Sat, Jun 18, 2011 at 4:56 AM, Bela Ban <belaban(a)yahoo.com> wrote:
> > #1 When does this happen (after a restart of a node, or on starting a new
> > node) ?
[vivek]: this usually happens after a node has gone out of a group
(due to some intermittent network outage or communication issues) and
comes back to merge into the group. We have also seen this if we
restart the node without much time between stop and start (this was
the example above). So, it's kind of random - it happens both for
merge views and a new views.
> > #2 Do the UUIDs disappear after some time, and the logical names are shown ?
[vivek]: No, in my experience the UUID remain if it's the same view,
but if the view changes (if any node goes in and out of a group) then
the logical name come back, otherwise the UUID remains.
> > #3 Does this cause any issues, other than a not-so-nice view with mixed
> > logical and UUID addresses ?
[vivek]: It's both - in some cases we have seen this causes node
isolation - where we are not able to get the merges (see related
https://issues.jboss.org/browse/JGRP-1326) - even after a node come
back it's not able to merge back. In some cases, the communication is
ok and it's just not-so-nice view.
> > #4 What's you config ?
We are using Tunnel with PING with two GRs (this is with JGroups 2.12.1),
(see attached config)
Thanks,
-vivek
> >
> > On 6/17/11 10:27 PM, vivek sar wrote:
>> >>
>> >> Hi,
>> >>
>> >> We are currently using JGroups 2.12.1 with Tunnel protocol (using
>> >> PING for discovery). Once in a while we see node represented using
>> >> UUID rather than their logical address. I think this issue is not
>> >> specific to this version of JGroups as we have seen similar problem
>> >> with 2.10 as well. The problem for us is not the view logging, but
>> >> different view information on different nodes - some get the logical
>> >> name and others just the UUID for the same view id. In past this kind
>> >> of view (showing UUIDs instead of logical address) also indicated
>> >> nodes and GRs out-of-sync on the node list, which in turn was causing
>> >> merge issues.
>> >>
>> >> Questions,
>> >>
>> >> 1) When does this happen - getting both logical address and UUID in
>> >> the same view?
>> >> 2) Why different nodes with same view id get different node
>> >> information (some get UUID, while others get the right logical
>> >> address)?
>> >> 3) How to fix it?
>> >>
>> >> For ex.,
>> >>
>> >> On Coordinator (collector_192.168.50.23):
>> >>
>> >> 2011-06-17 17:25:56,233 INFO
>> >> [Incoming-4,192.168.50.22_group,collector_192.168.50.23:4576]
>> >> RpcServiceManager - viewAccepted()-> New View:
>> >> [collector_192.168.50.23:4576|51] [collector_192.168.50.23:4576,
>> >> probe_192.168.50.32:4576, probe_192.168.50.24:4576,
>> >> manager_192.168.50.22:4576, probe_192.168.50.56:4576]
>> >>
>> >> Node 1 (manager_192.168.50.22):
>> >>
>> >> 2011-06-17 17:25:56,429 INFO
>> >> [Incoming-5,192.168.50.22_group,manager_192.168.50.22:4576]
>> >> RpcServiceManager - viewAccepted()-> New View:
>> >> [collector_192.168.50.23:4576|51] [collector_192.168.50.23:4576,
>> >> 110fcd0e-6f80-7cf0-d76b-0813b913daf1,
>> >> dd068d2f-ef21-1b66-1430-e8bfc862e6f9, manager_192.168.50.22:4576,
>> >> probe_192.168.50.56:4576]
>> >>
>> >> Node 2 (probe_192.168.50.56):
>> >>
>> >> 2011-06-17 12:02:01,695 INFO
>> >> [Incoming-10,192.168.50.22_group,probe_192.168.50.56:4576]
>> >> RpcServiceManager - viewAccepted()-> New View:
>> >> [collector_192.168.50.23:4576|51] [collector_192.168.50.23:4576,
>> >> probe_192.168.50.32:4576, probe_192.168.50.24:4576,
>> >> manager_192.168.50.22:4576, probe_192.168.50.56:4576]
>> >>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 8 months