[JBoss JIRA] (AS7-3827) NotSerializableException, StateTransferInProgressException, TimeoutException while other node in the cluster crashed
by Radoslav Husar (JIRA)
Radoslav Husar created AS7-3827:
-----------------------------------
Summary: NotSerializableException, StateTransferInProgressException, TimeoutException while other node in the cluster crashed
Key: AS7-3827
URL: https://issues.jboss.org/browse/AS7-3827
Project: Application Server 7
Issue Type: Bug
Components: Clustering
Affects Versions: 7.1.0.Final
Reporter: Radoslav Husar
Assignee: Paul Ferraro
Fix For: 7.1.1.Final
One more very vague bug report, but something definitely looks wrong.
This set of 3 exceptions (first 2 repeatedly) could be seen while servers in the cluster crash and are brought up again (while there are requests from EJB3 remote client) when using replicated SYNC cache.
The cluster has 1 minute in between membership changes (which is not very graceful but shows how cluster can deal with state transfer :-))
NotSerializableException seems very suspicious though.
{noformat}
[JBossINF] 10:24:21,545 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (OOB-17,null) ISPN000136: Execution error: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 6
[JBossINF] at org.infinispan.interceptors.StateTransferLockInterceptor.signalStateTransferInProgress(StateTransferLockInterceptor.java:199) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at org.infinispan.interceptors.StateTransferLockInterceptor.visitPrepareCommand(StateTransferLockInterceptor.java:80) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:130) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:89) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:70) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at org.infinispan.commands.tx.PrepareCommand.perform(PrepareCommand.java:127) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:127) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:136) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithRetry(InboundInvocationHandlerImpl.java:162) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:114) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:161) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:141) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:447) [jgroups-3.0.4.Final.jar:3.0.4.Final]
[JBossINF] at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:354) [jgroups-3.0.4.Final.jar:3.0.4.Final]
[JBossINF] at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:230) [jgroups-3.0.4.Final.jar:3.0.4.Final]
[JBossINF] at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543) [jgroups-3.0.4.Final.jar:3.0.4.Final]
[JBossINF] at org.jboss.as.clustering.jgroups.ClassLoaderAwareUpHandler.up(ClassLoaderAwareUpHandler.java:56) [jboss-as-clustering-jgroups-7.1.1.Final-SNAPSHOT.jar:7.1.1.Final-SNAPSHOT]
[JBossINF] at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [jgroups-3.0.4.Final.jar:3.0.4.Final]
[JBossINF] at org.jboss.as.clustering.jgroups.MuxChannel$ClassLoaderAwareMuxUpHandler.up(MuxChannel.java:64) [jboss-as-clustering-jgroups-7.1.1.Final-SNAPSHOT.jar:7.1.1.Final-SNAPSHOT]
[JBossINF] at org.jgroups.JChannel.up(JChannel.java:716) [jgroups-3.0.4.Final.jar:3.0.4.Final]
[JBossINF] at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026) [jgroups-3.0.4.Final.jar:3.0.4.Final]
[JBossINF] at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) [jgroups-3.0.4.Final.jar:3.0.4.Final]
[JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [jgroups-3.0.4.Final.jar:3.0.4.Final]
[JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [jgroups-3.0.4.Final.jar:3.0.4.Final]
[JBossINF] at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881) [jgroups-3.0.4.Final.jar:3.0.4.Final]
[JBossINF] at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244) [jgroups-3.0.4.Final.jar:3.0.4.Final]
[JBossINF] at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383) [jgroups-3.0.4.Final.jar:3.0.4.Final]
[JBossINF] at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:697) [jgroups-3.0.4.Final.jar:3.0.4.Final]
[JBossINF] at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559) [jgroups-3.0.4.Final.jar:3.0.4.Final]
[JBossINF] at org.jgroups.protocols.BARRIER.up(BARRIER.java:126) [jgroups-3.0.4.Final.jar:3.0.4.Final]
[JBossINF] at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:140) [jgroups-3.0.4.Final.jar:3.0.4.Final]
[JBossINF] at org.jgroups.protocols.FD.up(FD.java:273) [jgroups-3.0.4.Final.jar:3.0.4.Final]
[JBossINF] at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282) [jgroups-3.0.4.Final.jar:3.0.4.Final]
[JBossINF] at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) [jgroups-3.0.4.Final.jar:3.0.4.Final]
[JBossINF] at org.jgroups.protocols.Discovery.up(Discovery.java:354) [jgroups-3.0.4.Final.jar:3.0.4.Final]
[JBossINF] at org.jgroups.stack.Protocol.up(Protocol.java:358) [jgroups-3.0.4.Final.jar:3.0.4.Final]
[JBossINF] at org.jgroups.protocols.TP.passMessageUp(TP.java:1174) [jgroups-3.0.4.Final.jar:3.0.4.Final]
[JBossINF] at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1709) [jgroups-3.0.4.Final.jar:3.0.4.Final]
[JBossINF] at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1691) [jgroups-3.0.4.Final.jar:3.0.4.Final]
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30]
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30]
[JBossINF] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]
{noformat}
Error recovering views from the cluster members
{noformat}
...
[JBossINF] 10:24:24,735 ERROR [org.infinispan.cacheviews.CacheViewsManagerImpl] (CacheViewTrigger,perf21/web) Error recovering views from the cluster members: java.util.concurrent.ExecutionException: org.infinispan.CacheException: java.util.concurrent.ExecutionException: org.infinispan.marshall.NotSerializableException: org.infinispan.remoting.responses.ExceptionResponse
[JBossINF] at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) [rt.jar:1.6.0_30]
[JBossINF] at java.util.concurrent.FutureTask.get(FutureTask.java:83) [rt.jar:1.6.0_30]
[JBossINF] at org.infinispan.cacheviews.CacheViewsManagerImpl.recoverViews(CacheViewsManagerImpl.java:672) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at org.infinispan.cacheviews.CacheViewsManagerImpl.access$500(CacheViewsManagerImpl.java:95) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at org.infinispan.cacheviews.CacheViewsManagerImpl$ViewTriggerThread.run(CacheViewsManagerImpl.java:812) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] Caused by: org.infinispan.CacheException: java.util.concurrent.ExecutionException: org.infinispan.marshall.NotSerializableException: org.infinispan.remoting.responses.ExceptionResponse
[JBossINF] at org.infinispan.util.Util.rewrapAsCacheException(Util.java:524) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:115) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:447) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at org.infinispan.cacheviews.CacheViewsManagerImpl$4.call(CacheViewsManagerImpl.java:663) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at org.infinispan.cacheviews.CacheViewsManagerImpl$4.call(CacheViewsManagerImpl.java:660) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_30]
[JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_30]
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30]
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30]
[JBossINF] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]
[JBossINF] Caused by: java.util.concurrent.ExecutionException: org.infinispan.marshall.NotSerializableException: org.infinispan.remoting.responses.ExceptionResponse
[JBossINF] at org.jgroups.blocks.UnicastRequest.getValue(UnicastRequest.java:165) [jgroups-3.0.4.Final.jar:3.0.4.Final]
[JBossINF] at org.jgroups.blocks.UnicastRequest.get(UnicastRequest.java:191) [jgroups-3.0.4.Final.jar:3.0.4.Final]
[JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.call(CommandAwareRpcDispatcher.java:269) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:111) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] ... 8 more
[JBossINF] Caused by: org.infinispan.marshall.NotSerializableException: org.infinispan.remoting.responses.ExceptionResponse
[JBossINF] Caused by: an exception which occurred:
[JBossINF] in object org.infinispan.remoting.responses.ExceptionResponse@58e39f23
{noformat}
{noformat}
[JBossINF] 10:24:24,849 INFO [org.jboss.as.clustering.impl.CoreGroupCommunicationService.lifecycle.web] (Incoming-11,null) JBAS010247: New cluster view for partition web (id: 7, delta: -1, merge: false) : [perf21/web, perf20/web]
[JBossINF] 10:24:24,851 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-11,null) ISPN000094: Received new cluster view: [perf21/web|7] [perf21/web, perf20/web]
[JBossINF] 10:25:22,987 INFO [org.jboss.as.clustering.impl.CoreGroupCommunicationService.lifecycle.web] (Incoming-13,null) JBAS010247: New cluster view for partition web (id: 8, delta: 1, merge: false) : [perf21/web, perf20/web, perf19/web]
[JBossINF] 10:25:22,987 INFO [org.jboss.as.clustering.impl.CoreGroupCommunicationService.lifecycle.ejb] (Incoming-14,null) JBAS010247: New cluster view for partition ejb (id: 8, delta: 1, merge: false) : [perf21/ejb, perf20/ejb, perf19/ejb]
[JBossINF] 10:25:22,988 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-13,null) ISPN000094: Received new cluster view: [perf21/web|8] [perf21/web, perf20/web, perf19/web]
[JBossINF] 10:25:22,989 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-14,null) ISPN000094: Received new cluster view: [perf21/ejb|8] [perf21/ejb, perf20/ejb, perf19/ejb]
[JBossINF] 10:26:22,401 INFO [org.jboss.as.clustering.impl.CoreGroupCommunicationService.lifecycle.web] (Incoming-4,null) JBAS010247: New cluster view for partition web (id: 9, delta: -1, merge: false) : [perf21/web, perf20/web]
[JBossINF] 10:26:22,402 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-4,null) ISPN000094: Received new cluster view: [perf21/web|9] [perf21/web, perf20/web]
[JBossINF] 10:26:23,554 ERROR [org.infinispan.cacheviews.CacheViewsManagerImpl] (CacheViewInstaller-6,perf21/ejb) ISPN000172: Failed to prepare view CacheView{viewId=11, members=[perf21/ejb, perf20/ejb, perf19/ejb]} for cache org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSBImpl, rolling back to view CacheView{viewId=10, members=[perf21/ejb, perf20/ejb]}: java.util.concurrent.TimeoutException
[JBossINF] at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:228) [rt.jar:1.6.0_30]
[JBossINF] at java.util.concurrent.FutureTask.get(FutureTask.java:91) [rt.jar:1.6.0_30]
[JBossINF] at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterPrepareView(CacheViewsManagerImpl.java:319) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterInstallView(CacheViewsManagerImpl.java:250) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at org.infinispan.cacheviews.CacheViewsManagerImpl$ViewInstallationTask.call(CacheViewsManagerImpl.java:876) [infinispan-core-5.1.1.FINAL.jar:5.1.1.FINAL]
[JBossINF] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_30]
[JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_30]
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30]
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30]
[JBossINF] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 10 months
[JBoss JIRA] (AS7-2467) TimeoutException on node joining
by Radoslav Husar (Created) (JIRA)
TimeoutException on node joining
---------------------------------
Key: AS7-2467
URL: https://issues.jboss.org/browse/AS7-2467
Project: Application Server 7
Issue Type: Bug
Components: Clustering
Affects Versions: No Release
Environment: build is current head
Reporter: Radoslav Husar
Assignee: Paul Ferraro
Priority: Critical
Fix For: 7.1.0.Alpha1
The scenario was like this (manually)
# start 1 node
# start 2nd node
# when the node is joining (kind of hard to target the right time) post a request
This is all taking place in high perf lab without load so the timeout is not not anticipated.
{code}
10:35:05,501 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-4,null) ISPN000094: Received new cluster view: [perf18/web|1] [perf18/web, perf19/web]
10:35:26,055 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (ajp-perf18-10.16.90.54-8009-24) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [15 seconds] on key [//default-host/null/Ww+5NaIAnFIoXWEuhszfGqNb] for requestor [GlobalTransaction:<perf18/web>:87:local]! Lock held by [GlobalTransaction:<perf19/web>:1:local]
at org.infinispan.util.concurrent.locks.LockManagerImpl.lock(LockManagerImpl.java:207) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.util.concurrent.locks.LockManagerImpl.acquireLockNoCheck(LockManagerImpl.java:190) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockKey(AbstractLockingInterceptor.java:111) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.locking.OptimisticLockingInterceptor$LockAquisitionVisitor.visitPutKeyValueCommand(OptimisticLockingInterceptor.java:162) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:63) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:49) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:133) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:108) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:104) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:133) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:108) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.StateTransferLockInterceptor.visitPrepareCommand(StateTransferLockInterceptor.java:62) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:133) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:108) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:104) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:64) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:108) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:133) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:108) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:77) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:108) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:133) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.jboss.as.clustering.infinispan.ClassLoaderAwareCache$ClassLoaderAwareCommandInterceptor.handleDefault(ClassLoaderAwareCache.java:111) [jboss-as-clustering-infinispan-7.1.0.Alpha2-SNAPSHOT.jar:7.1.0.Alpha2-SNAPSHOT]
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:108) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:318) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:124) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:73) [infinispan-core-5.1.0.BETA2.jar:5.1.0.BETA2]
at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:117)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:403)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:104)
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:159)
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1159)
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:119)
at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
at org.jboss.as.clustering.web.impl.TransactionBatchingManager.endBatch(TransactionBatchingManager.java:75) [jboss-as-clustering-web-spi-7.1.0.Alpha2-SNAPSHOT.jar:7.1.0.Alpha2-SNAPSHOT]
at org.jboss.as.web.session.DistributableSessionManager.processSessionRepl(DistributableSessionManager.java:1547)
at org.jboss.as.web.session.DistributableSessionManager.storeSession(DistributableSessionManager.java:877)
at org.jboss.as.web.session.InstantSnapshotManager.snapshot(InstantSnapshotManager.java:45)
at org.jboss.as.web.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:131)
at org.jboss.as.web.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:89)
at org.jboss.as.web.session.JvmRouteValve.invoke(JvmRouteValve.java:88)
at org.jboss.as.web.session.LockingValve.invoke(LockingValve.java:56)
at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:155)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:362)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505)
at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930)
at java.lang.Thread.run(Thread.java:619) [:1.6.0_20]
10:35:26,065 ERROR [org.infinispan.transaction.TransactionCoordinator] (ajp-perf18-10.16.90.54-8009-24) ISPN000097: Error while processing 1PC PrepareCommand: org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [15 seconds] on key [//default-host/null/Ww+5NaIAnFIoXWEuhszfGqNb] for requestor [GlobalTransaction:<perf18/web>:87:local]! Lock held by [GlobalTransaction:<perf19/web>:1:local]
at org.infinispan.util.concurrent.locks.LockManagerImpl.lock(LockManagerImpl.java:207)
at org.infinispan.util.concurrent.locks.LockManagerImpl.acquireLockNoCheck(LockManagerImpl.java:190)
at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockKey(AbstractLockingInterceptor.java:111)
at org.infinispan.interceptors.locking.OptimisticLockingInterceptor$LockAquisitionVisitor.visitPutKeyValueCommand(OptimisticLockingInterceptor.java:162)
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:63)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)
at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:49)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:133)
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:108)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)
at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:104)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:133)
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:108)
at org.infinispan.interceptors.StateTransferLockInterceptor.visitPrepareCommand(StateTransferLockInterceptor.java:62)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:133)
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:108)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:104)
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:64)
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:108)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:133)
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:108)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)
at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:77)
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:108)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:133)
at org.jboss.as.clustering.infinispan.ClassLoaderAwareCache$ClassLoaderAwareCommandInterceptor.handleDefault(ClassLoaderAwareCache.java:111)
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:108)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:127)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:318)
at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:124)
at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:73)
at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:117)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:403)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:104)
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:159)
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1159)
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:119)
at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
at org.jboss.as.clustering.web.impl.TransactionBatchingManager.endBatch(TransactionBatchingManager.java:75)
at org.jboss.as.web.session.DistributableSessionManager.processSessionRepl(DistributableSessionManager.java:1547) [jboss-as-web-7.1.0.Alpha2-SNAPSHOT.jar:7.1.0.Alpha2-SNAPSHOT]
at org.jboss.as.web.session.DistributableSessionManager.storeSession(DistributableSessionManager.java:877) [jboss-as-web-7.1.0.Alpha2-SNAPSHOT.jar:7.1.0.Alpha2-SNAPSHOT]
at org.jboss.as.web.session.InstantSnapshotManager.snapshot(InstantSnapshotManager.java:45) [jboss-as-web-7.1.0.Alpha2-SNAPSHOT.jar:7.1.0.Alpha2-SNAPSHOT]
at org.jboss.as.web.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:131) [jboss-as-web-7.1.0.Alpha2-SNAPSHOT.jar:7.1.0.Alpha2-SNAPSHOT]
at org.jboss.as.web.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:89) [jboss-as-web-7.1.0.Alpha2-SNAPSHOT.jar:7.1.0.Alpha2-SNAPSHOT]
at org.jboss.as.web.session.JvmRouteValve.invoke(JvmRouteValve.java:88) [jboss-as-web-7.1.0.Alpha2-SNAPSHOT.jar:7.1.0.Alpha2-SNAPSHOT]
at org.jboss.as.web.session.LockingValve.invoke(LockingValve.java:56) [jboss-as-web-7.1.0.Alpha2-SNAPSHOT.jar:7.1.0.Alpha2-SNAPSHOT]
at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:155) [jboss-as-web-7.1.0.Alpha2-SNAPSHOT.jar:7.1.0.Alpha2-SNAPSHOT]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.3.Final.jar:7.1.0.Alpha2-SNAPSHOT]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.3.Final.jar:7.1.0.Alpha2-SNAPSHOT]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.3.Final.jar:7.1.0.Alpha2-SNAPSHOT]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:362) [jbossweb-7.0.3.Final.jar:7.1.0.Alpha2-SNAPSHOT]
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.3.Final.jar:7.1.0.Alpha2-SNAPSHOT]
at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [jbossweb-7.0.3.Final.jar:7.1.0.Alpha2-SNAPSHOT]
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.3.Final.jar:7.1.0.Alpha2-SNAPSHOT]
at java.lang.Thread.run(Thread.java:619) [:1.6.0_20]
10:35:26,072 WARN [com.arjuna.ats.jta] (ajp-perf18-10.16.90.54-8009-24) ARJUNA16029: SynchronizationImple.afterCompletion - failed for org.infinispan.transaction.synchronization.SynchronizationAdapter@432398ed with exception: org.infinispan.CacheException: Could not commit.
at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:75)
at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:117)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:403)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:104)
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:159)
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1159)
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:119)
at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
at org.jboss.as.clustering.web.impl.TransactionBatchingManager.endBatch(TransactionBatchingManager.java:75)
at org.jboss.as.web.session.DistributableSessionManager.processSessionRepl(DistributableSessionManager.java:1547) [jboss-as-web-7.1.0.Alpha2-SNAPSHOT.jar:7.1.0.Alpha2-SNAPSHOT]
at org.jboss.as.web.session.DistributableSessionManager.storeSession(DistributableSessionManager.java:877) [jboss-as-web-7.1.0.Alpha2-SNAPSHOT.jar:7.1.0.Alpha2-SNAPSHOT]
at org.jboss.as.web.session.InstantSnapshotManager.snapshot(InstantSnapshotManager.java:45) [jboss-as-web-7.1.0.Alpha2-SNAPSHOT.jar:7.1.0.Alpha2-SNAPSHOT]
at org.jboss.as.web.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:131) [jboss-as-web-7.1.0.Alpha2-SNAPSHOT.jar:7.1.0.Alpha2-SNAPSHOT]
at org.jboss.as.web.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:89) [jboss-as-web-7.1.0.Alpha2-SNAPSHOT.jar:7.1.0.Alpha2-SNAPSHOT]
at org.jboss.as.web.session.JvmRouteValve.invoke(JvmRouteValve.java:88) [jboss-as-web-7.1.0.Alpha2-SNAPSHOT.jar:7.1.0.Alpha2-SNAPSHOT]
at org.jboss.as.web.session.LockingValve.invoke(LockingValve.java:56) [jboss-as-web-7.1.0.Alpha2-SNAPSHOT.jar:7.1.0.Alpha2-SNAPSHOT]
at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:155) [jboss-as-web-7.1.0.Alpha2-SNAPSHOT.jar:7.1.0.Alpha2-SNAPSHOT]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.3.Final.jar:7.1.0.Alpha2-SNAPSHOT]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.3.Final.jar:7.1.0.Alpha2-SNAPSHOT]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.3.Final.jar:7.1.0.Alpha2-SNAPSHOT]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:362) [jbossweb-7.0.3.Final.jar:7.1.0.Alpha2-SNAPSHOT]
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.3.Final.jar:7.1.0.Alpha2-SNAPSHOT]
at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [jbossweb-7.0.3.Final.jar:7.1.0.Alpha2-SNAPSHOT]
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.3.Final.jar:7.1.0.Alpha2-SNAPSHOT]
at java.lang.Thread.run(Thread.java:619) [:1.6.0_20]
Caused by: javax.transaction.xa.XAException
at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:129)
at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:73)
... 24 more
10:35:26,075 WARN [com.arjuna.ats.arjuna] (ajp-perf18-10.16.90.54-8009-24) ARJUNA12127: TwoPhaseCoordinator.afterCompletion - returned failure for SynchronizationImple< 0:ffff0a105a36:784a5ec6:4eb154d1:b6, org.infinispan.transaction.synchronization.SynchronizationAdapter@432398ed >
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 10 months
[JBoss JIRA] Created: (JGRP-1053) UNICAST: set retransmission timeout based on actual retransmission times
by Bela Ban (JIRA)
UNICAST: set retransmission timeout based on actual retransmission times
------------------------------------------------------------------------
Key: JGRP-1053
URL: https://jira.jboss.org/jira/browse/JGRP-1053
Project: JGroups
Issue Type: Feature Request
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.9
UNICAST needs to compute a rolling average of retransmission times, per sender (AckSenderWindow).
The retransmission timeout per sender can then be set based on the actual average retransmission times. The advantage is that we throttle retransmission when we have a lot of message loss, and speed it up again when there are no message drops.
The function to set the timeout should always compute the new timeot value based on (1) the old value times a decay factor and (2) a new value.
The average should go up relatively quickly if the actual retransmission values go up, but come down slowly when the actual values go down.
A potential function is shown below:
static final double SLOW_DECAY_FACTOR=0.9, FAST_DECAY_FACTOR=0.7;
static final double FAST_UP= 1 / FAST_DECAY_FACTOR, SLOW_UP= 1 / SLOW_DECAY_FACTOR;
static final double SAFETY_BUFFER=0.3;
static double avg=200;
public static void main(String[] args) {
final long[] times={200,200,400,400,500,500,500,500,500,100,100,100,100,100,100,100,100,100,100,100,100,100};
// final long[] times={200,200,200,200,200,200,200,200,200,200,200};
for(Long val: times) {
double result=avg(val);
System.out.println(val + ": " + result);
}
}
private static double avg(long val) {
double decay, up;
if(val > avg) {
decay=FAST_DECAY_FACTOR;
up=FAST_UP;
}
else {
decay=SLOW_DECAY_FACTOR;
up=SLOW_UP;
}
double old_val=avg * decay;
double result=(old_val + val * up) / 2;
avg=result;
return result * (1 + SAFETY_BUFFER);
}
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 10 months
[JBoss JIRA] Created: (JGRP-1352) STATE: queue requests during state transfer
by Bela Ban (JIRA)
STATE: queue requests during state transfer
-------------------------------------------
Key: JGRP-1352
URL: https://issues.jboss.org/browse/JGRP-1352
Project: JGroups
Issue Type: Feature Request
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 3.1
STATE should queue requests to be delivered to the application while a state transfer is in progress.
Draft design:
- On STATE-REQ:
- If busy --> add request to queue, return
- Else --> processRequest
- Close MessageQueue and deliver all messages in it to the application
- Kick off a STABLE round, wait until it completes
- If MessageQueue is not empty --> processRequests()
- processRequest():
- Close BARRIER, suspend STABLE
- Get digest
- Open MessageQueue: all new messages will be queued now, rather than delivered to the application
- (Only MSG, not VIEW !)
- Open BARRIER (leave STABLE suspended !)
- Transfer state to state requester
- Resume STABLE
Advantages:
- The state provider is not blocked, e.g. to handle JOIN requests
- Only the state *requester* is blocked, this is OK though
- Between the state requests, the state provider gets a little breathing room to process
queued-up messages, and to purge messages (through stability)
Misc:
- Should the MessageQueue use the disk to handle large message sets ?
- If modifications are idempotent, we *don't* need to queue messages (make this configurable !)
(for example Infinispan ?)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 10 months
[JBoss JIRA] Created: (JGRP-1354) Compress views
by Bela Ban (JIRA)
Compress views
--------------
Key: JGRP-1354
URL: https://issues.jboss.org/browse/JGRP-1354
Project: JGroups
Issue Type: Enhancement
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 3.1
In clusters with many members, a View can be large. See if we can compress views. Possible strategies:
- Canonicalization of addresses (replace UUIDs with shorts) see different JIRA)
- Compress digests (see different JIRA)
- Keep a list of ViewIds and Views and only send the ViewIds (not for JoinRsp !)
- Create a DiffView: send the previous ViewId V1, the new ViewId V2, and the diff between V1 and V2, for example
- V34=V33 + N1 + N2
- V34=V33 - N30 - N13 + N3 (2 nodes left, 1 node joined)
- The receiver can then construct a full view from V34, V33 and the diffs
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 10 months
[JBoss JIRA] (JGRP-1436) GMS: separate view bundling timeouts for JOIN, LEAVE, MERGE
by Bela Ban (JIRA)
Bela Ban created JGRP-1436:
------------------------------
Summary: GMS: separate view bundling timeouts for JOIN, LEAVE, MERGE
Key: JGRP-1436
URL: https://issues.jboss.org/browse/JGRP-1436
Project: JGroups
Issue Type: Feature Request
Reporter: Bela Ban
Assignee: Bela Ban
Priority: Minor
Fix For: 3.2
View bundling in GMS bundles events based on (1) whether they can be processed together and (2) whether a view bundling timeout is defined. The latter timeout is for all events, perhaps we want to define separate timeouts, e.g. the timeout for JOINs is 3000, but for LEAVE it is 500. This means that when we get multiple JOIN events, we'll wait up to 5 seconds and then process them together. When we get multiple LEAVE requests, we only wait for half a second before processing them. This causes LEAVEs to be processed almost immediately, whereas JOINs are bundled.
Also revisit the code which determines which events can be processed together. Perhaps MERGE events can be bundled, too ?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 10 months
[JBoss JIRA] (JGRP-1391) Compress MergeViews
by Bela Ban (Created) (JIRA)
Compress MergeViews
-------------------
Key: JGRP-1391
URL: https://issues.jboss.org/browse/JGRP-1391
Project: JGroups
Issue Type: Enhancement
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 3.1
The 'subgroups' field of MergeView is a list of Views, referring to the subclusters that formed the new view.
Each View in that list contains (1) the creator of the view, (2) an ID and (3) a list of member addresses. Both (1) and (3) are present in the new MergeView, so, to save memory and message sizes, 'subgroups' could be compressed: (1) and (3) could be indices into View's 'members' field.
Example: if we have MergeView A|22={A,B,C,D,E,X,Y,Z}, subgroups=A|21={A,B,C}, D|19={D,E}, X|20={X,Y,Z},
then 'subgroups' can be compressed as follows:
subgroups=0|21={0,1,2}, 3|19={3,4}, 5|20={5,6,7}
This is possible because the indices never change, as the contents of Views and MergeViews are static, e.g. we can never add or remove a member: to do this, a new View/MergeView would be created.
Substituting addresses with indices (shorts, possibly using variable-bit encoding) reduces the memory footprint of a view, and the size of a serialized view on the wire.
Getting the subgroups field, e.g. using method getSubgroups() would synthesize the List<View> just-in-time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 10 months
[JBoss JIRA] (AS7-4094) Add "connector" => "ajp": Scheme may not be null
by Michal Babacek (JIRA)
Michal Babacek created AS7-4094:
-----------------------------------
Summary: Add "connector" => "ajp": Scheme may not be null
Key: AS7-4094
URL: https://issues.jboss.org/browse/AS7-4094
Project: Application Server 7
Issue Type: Bug
Affects Versions: 7.1.1.Final
Reporter: Michal Babacek
Assignee: Paul Ferraro
I have tried to use mod_cluster with ajp on AS7 current master *89daec9* with [this configuration|http://pastebin.com/p9r3FzeV] (default with jvmRoute and mod_cluster setup added + changed port numbers, nothing else).
The result of my endeavour was this unfortunate error on startup:
{noformat}
03:56:31,995 INFO [org.jboss.modules] JBoss Modules version 1.1.1.GA
03:56:32,190 INFO [org.jboss.msc] JBoss MSC version 1.0.2.GA 03:56:32,247 INFO [org.jboss.as] JBAS015899: JBoss AS 7.1.1.Final-SNAPSHOT "Thunder" starting
03:56:33,431 ERROR [org.jboss.as.controller.management-operation] Operation ("add") failed - address: ([ ("subsystem" => "web"), ("connector" => "ajp") ]) - failure description: "JBAS014746: scheme may not be null"
2012/03/08 03:57:31:378 EST [DEBUG][Thread-17] HOST perf09.mw.lab.eng.bos.redhat.com:rootProcess:sf - JBossStartup, server started!
{noformat}
Any ideas?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 10 months