4:51 p.m.
Potential deadlock with ClusteredCacheLoader
--------------------------------------------
Key: JBCACHE-1103
URL: http://jira.jboss.com/jira/browse/JBCACHE-1103
Project: JBoss Cache
Issue Type: Bug
Security Level: Public (Everyone can see)
Components: Cache loaders, Clustering
Affects Versions: 1.4.1.SP3, 2.0.0.CR2
Reporter: Brian Stansberry
Assigned To: Manik Surtani
Can occur if
1) Thread1 on node1 is trying to get(fqnA) which is not in-VM, so the CCL tries to do a
clustered get. The CacheLoaderInterceptor lock for fqnA is held at this time.
2) Thread1 blocks in FC waiting for credits.
3) Replication message for fqnA arrives from node2.
4) IncomingPacketHandler thread will block waiting for the CacheLoaderInterceptor lock for
fqnA.
5) FC credits cannot arrive, so we deadlock.
Stack traces below show this.
Thread: TP-Processor623 : priority:5, demon:true, threadId:1637,
threadState:TIMED_WAITING,
threadLockName:EDU.oswego.cs.dl.util.concurrent.CondVar@5947cf38
java.lang.Object.wait(Native Method)
EDU.oswego.cs.dl.util.concurrent.CondVar.timedwait(CondVar.java:222)
org.jgroups.protocols.FC.handleDownMessage(FC.java:454)
org.jgroups.protocols.FC.down(FC.java:374)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.protocols.FC.receiveDownEvent(FC.java:368)
org.jgroups.stack.Protocol.passDown(Protocol.java:533)
org.jgroups.protocols.FRAG2.down(FRAG2.java:167)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.stack.Protocol.passDown(Protocol.java:533)
org.jgroups.protocols.pbcast.STATE_TRANSFER.down(STATE_TRANSFER.java:294)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:390)
org.jgroups.JChannel.down(JChannel.java:1231)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:790)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.passDown(MessageDispatcher.java:767)
org.jgroups.blocks.RequestCorrelator.sendRequest(RequestCorrelator.java:308)
org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:444)
org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:193)
org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:431)
org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:192)
org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:163)
org.jboss.cache.TreeCache.callRemoteMethodsViaReflection(TreeCache.java:4437)
org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4391)
org.jboss.cache.loader.ClusteredCacheLoader.callRemote(ClusteredCacheLoader.java:75)
org.jboss.cache.loader.ClusteredCacheLoader.get0(ClusteredCacheLoader.java:119)
org.jboss.cache.loader.ClusteredCacheLoader.get(ClusteredCacheLoader.java:113)
org.jboss.cache.interceptors.CacheLoaderInterceptor.loadData(CacheLoaderInterceptor.java:443)
org.jboss.cache.interceptors.CacheLoaderInterceptor.loadNode(CacheLoaderInterceptor.java:353)
org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:180)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:39)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:135)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:138)
org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
org.jboss.cache.TreeCache.get(TreeCache.java:3627)
org.jboss.cache.TreeCache.get(TreeCache.java:3608)
org.jboss.cache.TreeCache.get(TreeCache.java:3404)
sun.reflect.GeneratedMethodAccessor341.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
$Proxy60.get(Unknown Source)
org.jboss.web.tomcat.tc5.session.JBossCacheWrapper.get(JBossCacheWrapper.java:78)
org.jboss.web.tomcat.tc5.session.JBossCacheService.loadSession(JBossCacheService.java:247)
org.jboss.web.tomcat.tc5.session.JBossCacheManager.loadSession(JBossCacheManager.java:1006)
org.jboss.web.tomcat.tc5.session.JBossCacheManager.findSession(JBossCacheManager.java:778)
org.apache.catalina.connector.Request.doGetSession(Request.java:2196)
org.apache.catalina.connector.Request.getSession(Request.java:2024)
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:97)
org.jboss.web.tomcat.tc5.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:87)
org.jboss.web.tomcat.tc5.session.JvmRouteValve.invoke(JvmRouteValve.java:84)
org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)
org.jboss.web.tomcat.tc5.sso.ClusteredSingleSignOn.invoke(ClusteredSingleSignOn.java:637)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
java.lang.Thread.run(Thread.java:619)
Thread: IncomingPacketHandler (channel=Tomcat-LAB01) : priority:5, demon:true,
threadId:35, threadState:WAITING,
threadLockName:org.jboss.cache.interceptors.CacheLoaderInterceptor@5778e8ed
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.jboss.cache.interceptors.BaseCacheLoaderInterceptor.obtainLoaderLock(BaseCacheLoaderInterceptor.java:49)
org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:168)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:39)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:64)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:167)
org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
org.jboss.cache.TreeCache._replicate(TreeCache.java:5162)
sun.reflect.GeneratedMethodAccessor344.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jgroups.blocks.MethodCall.invoke(MethodCall.java:330)
org.jgroups.blocks.RpcDispatcher.handle(RpcDispatcher.java:281)
org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:654)
org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:544)
org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:367)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:777)
org.jgroups.JChannel.up(JChannel.java:1091)
org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:382)
org.jgroups.stack.ProtocolStack.receiveUpEvent(ProtocolStack.java:398)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:158)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FRAG2.up(FRAG2.java:197)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FC.up(FC.java:422)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.GMS.up(GMS.java:768)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.protocols.pbcast.GMS.receiveUpEvent(GMS.java:788)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:262)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.UNICAST.up(UNICAST.java:259)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:685)
org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:517)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:170)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FD.up(FD.java:300)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.MERGE2.up(MERGE2.java:162)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.Discovery.up(Discovery.java:225)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.TP.handleIncomingMessage(TP.java:909)
org.jgroups.protocols.TP.handleIncomingPacket(TP.java:851)
org.jgroups.protocols.TP.access$400(TP.java:46)
org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1297)
java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:02 p.m.
New subject: [JBoss JIRA] Updated: (JBCACHE-1103) Potential deadlock with ClusteredCacheLoader
[ http://jira.jboss.com/jira/browse/JBCACHE-1103?page=all ]
Brian Stansberry updated JBCACHE-1103:
--------------------------------------
Help Desk Ticket Reference: https://na1.salesforce.com/50030000003N8dr
Potential deadlock with ClusteredCacheLoader
--------------------------------------------
Key: JBCACHE-1103
URL: http://jira.jboss.com/jira/browse/JBCACHE-1103
Project: JBoss Cache
Issue Type: Bug
Security Level: Public(Everyone can see)
Components: Clustering, Cache loaders
Affects Versions: 1.4.1.SP3, 2.0.0.CR2
Reporter: Brian Stansberry
Assigned To: Manik Surtani
Can occur if
1) Thread1 on node1 is trying to get(fqnA) which is not in-VM, so the CCL tries to do a
clustered get. The CacheLoaderInterceptor lock for fqnA is held at this time.
2) Thread1 blocks in FC waiting for credits.
3) Replication message for fqnA arrives from node2.
4) IncomingPacketHandler thread will block waiting for the CacheLoaderInterceptor lock
for fqnA.
5) FC credits cannot arrive, so we deadlock.
Stack traces below show this.
Thread: TP-Processor623 : priority:5, demon:true, threadId:1637,
threadState:TIMED_WAITING,
threadLockName:EDU.oswego.cs.dl.util.concurrent.CondVar@5947cf38
java.lang.Object.wait(Native Method)
EDU.oswego.cs.dl.util.concurrent.CondVar.timedwait(CondVar.java:222)
org.jgroups.protocols.FC.handleDownMessage(FC.java:454)
org.jgroups.protocols.FC.down(FC.java:374)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.protocols.FC.receiveDownEvent(FC.java:368)
org.jgroups.stack.Protocol.passDown(Protocol.java:533)
org.jgroups.protocols.FRAG2.down(FRAG2.java:167)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.stack.Protocol.passDown(Protocol.java:533)
org.jgroups.protocols.pbcast.STATE_TRANSFER.down(STATE_TRANSFER.java:294)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:390)
org.jgroups.JChannel.down(JChannel.java:1231)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:790)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.passDown(MessageDispatcher.java:767)
org.jgroups.blocks.RequestCorrelator.sendRequest(RequestCorrelator.java:308)
org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:444)
org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:193)
org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:431)
org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:192)
org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:163)
org.jboss.cache.TreeCache.callRemoteMethodsViaReflection(TreeCache.java:4437)
org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4391)
org.jboss.cache.loader.ClusteredCacheLoader.callRemote(ClusteredCacheLoader.java:75)
org.jboss.cache.loader.ClusteredCacheLoader.get0(ClusteredCacheLoader.java:119)
org.jboss.cache.loader.ClusteredCacheLoader.get(ClusteredCacheLoader.java:113)
org.jboss.cache.interceptors.CacheLoaderInterceptor.loadData(CacheLoaderInterceptor.java:443)
org.jboss.cache.interceptors.CacheLoaderInterceptor.loadNode(CacheLoaderInterceptor.java:353)
org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:180)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:39)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:135)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:138)
org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
org.jboss.cache.TreeCache.get(TreeCache.java:3627)
org.jboss.cache.TreeCache.get(TreeCache.java:3608)
org.jboss.cache.TreeCache.get(TreeCache.java:3404)
sun.reflect.GeneratedMethodAccessor341.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
$Proxy60.get(Unknown Source)
org.jboss.web.tomcat.tc5.session.JBossCacheWrapper.get(JBossCacheWrapper.java:78)
org.jboss.web.tomcat.tc5.session.JBossCacheService.loadSession(JBossCacheService.java:247)
org.jboss.web.tomcat.tc5.session.JBossCacheManager.loadSession(JBossCacheManager.java:1006)
org.jboss.web.tomcat.tc5.session.JBossCacheManager.findSession(JBossCacheManager.java:778)
org.apache.catalina.connector.Request.doGetSession(Request.java:2196)
org.apache.catalina.connector.Request.getSession(Request.java:2024)
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:97)
org.jboss.web.tomcat.tc5.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:87)
org.jboss.web.tomcat.tc5.session.JvmRouteValve.invoke(JvmRouteValve.java:84)
org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)
org.jboss.web.tomcat.tc5.sso.ClusteredSingleSignOn.invoke(ClusteredSingleSignOn.java:637)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
java.lang.Thread.run(Thread.java:619)
Thread: IncomingPacketHandler (channel=Tomcat-LAB01) : priority:5, demon:true,
threadId:35, threadState:WAITING,
threadLockName:org.jboss.cache.interceptors.CacheLoaderInterceptor@5778e8ed
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.jboss.cache.interceptors.BaseCacheLoaderInterceptor.obtainLoaderLock(BaseCacheLoaderInterceptor.java:49)
org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:168)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:39)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:64)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:167)
org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
org.jboss.cache.TreeCache._replicate(TreeCache.java:5162)
sun.reflect.GeneratedMethodAccessor344.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jgroups.blocks.MethodCall.invoke(MethodCall.java:330)
org.jgroups.blocks.RpcDispatcher.handle(RpcDispatcher.java:281)
org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:654)
org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:544)
org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:367)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:777)
org.jgroups.JChannel.up(JChannel.java:1091)
org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:382)
org.jgroups.stack.ProtocolStack.receiveUpEvent(ProtocolStack.java:398)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:158)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FRAG2.up(FRAG2.java:197)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FC.up(FC.java:422)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.GMS.up(GMS.java:768)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.protocols.pbcast.GMS.receiveUpEvent(GMS.java:788)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:262)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.UNICAST.up(UNICAST.java:259)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:685)
org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:517)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:170)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FD.up(FD.java:300)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.MERGE2.up(MERGE2.java:162)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.Discovery.up(Discovery.java:225)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.TP.handleIncomingMessage(TP.java:909)
org.jgroups.protocols.TP.handleIncomingPacket(TP.java:851)
org.jgroups.protocols.TP.access$400(TP.java:46)
org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1297)
java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:40 a.m.
New subject: [JBoss JIRA] Commented: (JBCACHE-1103) Potential deadlock with ClusteredCacheLoader
[ http://jira.jboss.com/jira/browse/JBCACHE-1103?page=comments#action_12365203 ]
Manik Surtani commented on JBCACHE-1103:
----------------------------------------
Shouldn't JGroups' RpcDispatcher's timeout param timeout and not block forever
waiting for FC? And if it does, is this by design, that the timeout param is only about
comms timeout, and not for the RpcDispatcher to return on an invocation? Will ping Bela
and check on this.
Potential deadlock with ClusteredCacheLoader
--------------------------------------------
Key: JBCACHE-1103
URL: http://jira.jboss.com/jira/browse/JBCACHE-1103
Project: JBoss Cache
Issue Type: Bug
Security Level: Public(Everyone can see)
Components: Clustering, Cache loaders
Affects Versions: 1.4.1.SP3, 2.0.0.CR2
Reporter: Brian Stansberry
Assigned To: Manik Surtani
Can occur if
1) Thread1 on node1 is trying to get(fqnA) which is not in-VM, so the CCL tries to do a
clustered get. The CacheLoaderInterceptor lock for fqnA is held at this time.
2) Thread1 blocks in FC waiting for credits.
3) Replication message for fqnA arrives from node2.
4) IncomingPacketHandler thread will block waiting for the CacheLoaderInterceptor lock
for fqnA.
5) FC credits cannot arrive, so we deadlock.
Stack traces below show this.
Thread: TP-Processor623 : priority:5, demon:true, threadId:1637,
threadState:TIMED_WAITING,
threadLockName:EDU.oswego.cs.dl.util.concurrent.CondVar@5947cf38
java.lang.Object.wait(Native Method)
EDU.oswego.cs.dl.util.concurrent.CondVar.timedwait(CondVar.java:222)
org.jgroups.protocols.FC.handleDownMessage(FC.java:454)
org.jgroups.protocols.FC.down(FC.java:374)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.protocols.FC.receiveDownEvent(FC.java:368)
org.jgroups.stack.Protocol.passDown(Protocol.java:533)
org.jgroups.protocols.FRAG2.down(FRAG2.java:167)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.stack.Protocol.passDown(Protocol.java:533)
org.jgroups.protocols.pbcast.STATE_TRANSFER.down(STATE_TRANSFER.java:294)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:390)
org.jgroups.JChannel.down(JChannel.java:1231)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:790)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.passDown(MessageDispatcher.java:767)
org.jgroups.blocks.RequestCorrelator.sendRequest(RequestCorrelator.java:308)
org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:444)
org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:193)
org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:431)
org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:192)
org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:163)
org.jboss.cache.TreeCache.callRemoteMethodsViaReflection(TreeCache.java:4437)
org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4391)
org.jboss.cache.loader.ClusteredCacheLoader.callRemote(ClusteredCacheLoader.java:75)
org.jboss.cache.loader.ClusteredCacheLoader.get0(ClusteredCacheLoader.java:119)
org.jboss.cache.loader.ClusteredCacheLoader.get(ClusteredCacheLoader.java:113)
org.jboss.cache.interceptors.CacheLoaderInterceptor.loadData(CacheLoaderInterceptor.java:443)
org.jboss.cache.interceptors.CacheLoaderInterceptor.loadNode(CacheLoaderInterceptor.java:353)
org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:180)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:39)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:135)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:138)
org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
org.jboss.cache.TreeCache.get(TreeCache.java:3627)
org.jboss.cache.TreeCache.get(TreeCache.java:3608)
org.jboss.cache.TreeCache.get(TreeCache.java:3404)
sun.reflect.GeneratedMethodAccessor341.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
$Proxy60.get(Unknown Source)
org.jboss.web.tomcat.tc5.session.JBossCacheWrapper.get(JBossCacheWrapper.java:78)
org.jboss.web.tomcat.tc5.session.JBossCacheService.loadSession(JBossCacheService.java:247)
org.jboss.web.tomcat.tc5.session.JBossCacheManager.loadSession(JBossCacheManager.java:1006)
org.jboss.web.tomcat.tc5.session.JBossCacheManager.findSession(JBossCacheManager.java:778)
org.apache.catalina.connector.Request.doGetSession(Request.java:2196)
org.apache.catalina.connector.Request.getSession(Request.java:2024)
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:97)
org.jboss.web.tomcat.tc5.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:87)
org.jboss.web.tomcat.tc5.session.JvmRouteValve.invoke(JvmRouteValve.java:84)
org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)
org.jboss.web.tomcat.tc5.sso.ClusteredSingleSignOn.invoke(ClusteredSingleSignOn.java:637)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
java.lang.Thread.run(Thread.java:619)
Thread: IncomingPacketHandler (channel=Tomcat-LAB01) : priority:5, demon:true,
threadId:35, threadState:WAITING,
threadLockName:org.jboss.cache.interceptors.CacheLoaderInterceptor@5778e8ed
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.jboss.cache.interceptors.BaseCacheLoaderInterceptor.obtainLoaderLock(BaseCacheLoaderInterceptor.java:49)
org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:168)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:39)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:64)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:167)
org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
org.jboss.cache.TreeCache._replicate(TreeCache.java:5162)
sun.reflect.GeneratedMethodAccessor344.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jgroups.blocks.MethodCall.invoke(MethodCall.java:330)
org.jgroups.blocks.RpcDispatcher.handle(RpcDispatcher.java:281)
org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:654)
org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:544)
org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:367)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:777)
org.jgroups.JChannel.up(JChannel.java:1091)
org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:382)
org.jgroups.stack.ProtocolStack.receiveUpEvent(ProtocolStack.java:398)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:158)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FRAG2.up(FRAG2.java:197)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FC.up(FC.java:422)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.GMS.up(GMS.java:768)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.protocols.pbcast.GMS.receiveUpEvent(GMS.java:788)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:262)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.UNICAST.up(UNICAST.java:259)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:685)
org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:517)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:170)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FD.up(FD.java:300)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.MERGE2.up(MERGE2.java:162)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.Discovery.up(Discovery.java:225)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.TP.handleIncomingMessage(TP.java:909)
org.jgroups.protocols.TP.handleIncomingPacket(TP.java:851)
org.jgroups.protocols.TP.access$400(TP.java:46)
org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1297)
java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:48 a.m.
New subject: [JBoss JIRA] Commented: (JBCACHE-1103) Potential deadlock with ClusteredCacheLoader
[ http://jira.jboss.com/jira/browse/JBCACHE-1103?page=comments#action_12365204 ]
Manik Surtani commented on JBCACHE-1103:
----------------------------------------
the timeout only measures time taken for the remote call; does not consider the JGroups
protocol stack. This is by design.
Potential deadlock with ClusteredCacheLoader
--------------------------------------------
Key: JBCACHE-1103
URL: http://jira.jboss.com/jira/browse/JBCACHE-1103
Project: JBoss Cache
Issue Type: Bug
Security Level: Public(Everyone can see)
Components: Clustering, Cache loaders
Affects Versions: 1.4.1.SP3, 2.0.0.CR2
Reporter: Brian Stansberry
Assigned To: Manik Surtani
Can occur if
1) Thread1 on node1 is trying to get(fqnA) which is not in-VM, so the CCL tries to do a
clustered get. The CacheLoaderInterceptor lock for fqnA is held at this time.
2) Thread1 blocks in FC waiting for credits.
3) Replication message for fqnA arrives from node2.
4) IncomingPacketHandler thread will block waiting for the CacheLoaderInterceptor lock
for fqnA.
5) FC credits cannot arrive, so we deadlock.
Stack traces below show this.
Thread: TP-Processor623 : priority:5, demon:true, threadId:1637,
threadState:TIMED_WAITING,
threadLockName:EDU.oswego.cs.dl.util.concurrent.CondVar@5947cf38
java.lang.Object.wait(Native Method)
EDU.oswego.cs.dl.util.concurrent.CondVar.timedwait(CondVar.java:222)
org.jgroups.protocols.FC.handleDownMessage(FC.java:454)
org.jgroups.protocols.FC.down(FC.java:374)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.protocols.FC.receiveDownEvent(FC.java:368)
org.jgroups.stack.Protocol.passDown(Protocol.java:533)
org.jgroups.protocols.FRAG2.down(FRAG2.java:167)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.stack.Protocol.passDown(Protocol.java:533)
org.jgroups.protocols.pbcast.STATE_TRANSFER.down(STATE_TRANSFER.java:294)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:390)
org.jgroups.JChannel.down(JChannel.java:1231)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:790)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.passDown(MessageDispatcher.java:767)
org.jgroups.blocks.RequestCorrelator.sendRequest(RequestCorrelator.java:308)
org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:444)
org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:193)
org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:431)
org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:192)
org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:163)
org.jboss.cache.TreeCache.callRemoteMethodsViaReflection(TreeCache.java:4437)
org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4391)
org.jboss.cache.loader.ClusteredCacheLoader.callRemote(ClusteredCacheLoader.java:75)
org.jboss.cache.loader.ClusteredCacheLoader.get0(ClusteredCacheLoader.java:119)
org.jboss.cache.loader.ClusteredCacheLoader.get(ClusteredCacheLoader.java:113)
org.jboss.cache.interceptors.CacheLoaderInterceptor.loadData(CacheLoaderInterceptor.java:443)
org.jboss.cache.interceptors.CacheLoaderInterceptor.loadNode(CacheLoaderInterceptor.java:353)
org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:180)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:39)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:135)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:138)
org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
org.jboss.cache.TreeCache.get(TreeCache.java:3627)
org.jboss.cache.TreeCache.get(TreeCache.java:3608)
org.jboss.cache.TreeCache.get(TreeCache.java:3404)
sun.reflect.GeneratedMethodAccessor341.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
$Proxy60.get(Unknown Source)
org.jboss.web.tomcat.tc5.session.JBossCacheWrapper.get(JBossCacheWrapper.java:78)
org.jboss.web.tomcat.tc5.session.JBossCacheService.loadSession(JBossCacheService.java:247)
org.jboss.web.tomcat.tc5.session.JBossCacheManager.loadSession(JBossCacheManager.java:1006)
org.jboss.web.tomcat.tc5.session.JBossCacheManager.findSession(JBossCacheManager.java:778)
org.apache.catalina.connector.Request.doGetSession(Request.java:2196)
org.apache.catalina.connector.Request.getSession(Request.java:2024)
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:97)
org.jboss.web.tomcat.tc5.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:87)
org.jboss.web.tomcat.tc5.session.JvmRouteValve.invoke(JvmRouteValve.java:84)
org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)
org.jboss.web.tomcat.tc5.sso.ClusteredSingleSignOn.invoke(ClusteredSingleSignOn.java:637)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
java.lang.Thread.run(Thread.java:619)
Thread: IncomingPacketHandler (channel=Tomcat-LAB01) : priority:5, demon:true,
threadId:35, threadState:WAITING,
threadLockName:org.jboss.cache.interceptors.CacheLoaderInterceptor@5778e8ed
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.jboss.cache.interceptors.BaseCacheLoaderInterceptor.obtainLoaderLock(BaseCacheLoaderInterceptor.java:49)
org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:168)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:39)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:64)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:167)
org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
org.jboss.cache.TreeCache._replicate(TreeCache.java:5162)
sun.reflect.GeneratedMethodAccessor344.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jgroups.blocks.MethodCall.invoke(MethodCall.java:330)
org.jgroups.blocks.RpcDispatcher.handle(RpcDispatcher.java:281)
org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:654)
org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:544)
org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:367)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:777)
org.jgroups.JChannel.up(JChannel.java:1091)
org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:382)
org.jgroups.stack.ProtocolStack.receiveUpEvent(ProtocolStack.java:398)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:158)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FRAG2.up(FRAG2.java:197)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FC.up(FC.java:422)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.GMS.up(GMS.java:768)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.protocols.pbcast.GMS.receiveUpEvent(GMS.java:788)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:262)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.UNICAST.up(UNICAST.java:259)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:685)
org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:517)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:170)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FD.up(FD.java:300)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.MERGE2.up(MERGE2.java:162)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.Discovery.up(Discovery.java:225)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.TP.handleIncomingMessage(TP.java:909)
org.jgroups.protocols.TP.handleIncomingPacket(TP.java:851)
org.jgroups.protocols.TP.access$400(TP.java:46)
org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1297)
java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:34 a.m.
New subject: [JBoss JIRA] Commented: (JBCACHE-1103) Potential deadlock with ClusteredCacheLoader
[ http://jira.jboss.com/jira/browse/JBCACHE-1103?page=comments#action_12365404 ]
Manik Surtani commented on JBCACHE-1103:
----------------------------------------
This should not affect JBoss Cache 2.x since 2.x uses JGroups 2.5 (out of band message
delivery will prevent the deadlock)
Potential deadlock with ClusteredCacheLoader
--------------------------------------------
Key: JBCACHE-1103
URL: http://jira.jboss.com/jira/browse/JBCACHE-1103
Project: JBoss Cache
Issue Type: Bug
Security Level: Public(Everyone can see)
Components: Clustering, Cache loaders
Affects Versions: 1.4.1.SP3, 2.0.0.CR2
Reporter: Brian Stansberry
Assigned To: Manik Surtani
Can occur if
1) Thread1 on node1 is trying to get(fqnA) which is not in-VM, so the CCL tries to do a
clustered get. The CacheLoaderInterceptor lock for fqnA is held at this time.
2) Thread1 blocks in FC waiting for credits.
3) Replication message for fqnA arrives from node2.
4) IncomingPacketHandler thread will block waiting for the CacheLoaderInterceptor lock
for fqnA.
5) FC credits cannot arrive, so we deadlock.
Stack traces below show this.
Thread: TP-Processor623 : priority:5, demon:true, threadId:1637,
threadState:TIMED_WAITING,
threadLockName:EDU.oswego.cs.dl.util.concurrent.CondVar@5947cf38
java.lang.Object.wait(Native Method)
EDU.oswego.cs.dl.util.concurrent.CondVar.timedwait(CondVar.java:222)
org.jgroups.protocols.FC.handleDownMessage(FC.java:454)
org.jgroups.protocols.FC.down(FC.java:374)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.protocols.FC.receiveDownEvent(FC.java:368)
org.jgroups.stack.Protocol.passDown(Protocol.java:533)
org.jgroups.protocols.FRAG2.down(FRAG2.java:167)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.stack.Protocol.passDown(Protocol.java:533)
org.jgroups.protocols.pbcast.STATE_TRANSFER.down(STATE_TRANSFER.java:294)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:390)
org.jgroups.JChannel.down(JChannel.java:1231)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:790)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.passDown(MessageDispatcher.java:767)
org.jgroups.blocks.RequestCorrelator.sendRequest(RequestCorrelator.java:308)
org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:444)
org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:193)
org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:431)
org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:192)
org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:163)
org.jboss.cache.TreeCache.callRemoteMethodsViaReflection(TreeCache.java:4437)
org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4391)
org.jboss.cache.loader.ClusteredCacheLoader.callRemote(ClusteredCacheLoader.java:75)
org.jboss.cache.loader.ClusteredCacheLoader.get0(ClusteredCacheLoader.java:119)
org.jboss.cache.loader.ClusteredCacheLoader.get(ClusteredCacheLoader.java:113)
org.jboss.cache.interceptors.CacheLoaderInterceptor.loadData(CacheLoaderInterceptor.java:443)
org.jboss.cache.interceptors.CacheLoaderInterceptor.loadNode(CacheLoaderInterceptor.java:353)
org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:180)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:39)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:135)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:138)
org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
org.jboss.cache.TreeCache.get(TreeCache.java:3627)
org.jboss.cache.TreeCache.get(TreeCache.java:3608)
org.jboss.cache.TreeCache.get(TreeCache.java:3404)
sun.reflect.GeneratedMethodAccessor341.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
$Proxy60.get(Unknown Source)
org.jboss.web.tomcat.tc5.session.JBossCacheWrapper.get(JBossCacheWrapper.java:78)
org.jboss.web.tomcat.tc5.session.JBossCacheService.loadSession(JBossCacheService.java:247)
org.jboss.web.tomcat.tc5.session.JBossCacheManager.loadSession(JBossCacheManager.java:1006)
org.jboss.web.tomcat.tc5.session.JBossCacheManager.findSession(JBossCacheManager.java:778)
org.apache.catalina.connector.Request.doGetSession(Request.java:2196)
org.apache.catalina.connector.Request.getSession(Request.java:2024)
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:97)
org.jboss.web.tomcat.tc5.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:87)
org.jboss.web.tomcat.tc5.session.JvmRouteValve.invoke(JvmRouteValve.java:84)
org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)
org.jboss.web.tomcat.tc5.sso.ClusteredSingleSignOn.invoke(ClusteredSingleSignOn.java:637)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
java.lang.Thread.run(Thread.java:619)
Thread: IncomingPacketHandler (channel=Tomcat-LAB01) : priority:5, demon:true,
threadId:35, threadState:WAITING,
threadLockName:org.jboss.cache.interceptors.CacheLoaderInterceptor@5778e8ed
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.jboss.cache.interceptors.BaseCacheLoaderInterceptor.obtainLoaderLock(BaseCacheLoaderInterceptor.java:49)
org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:168)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:39)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:64)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:167)
org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
org.jboss.cache.TreeCache._replicate(TreeCache.java:5162)
sun.reflect.GeneratedMethodAccessor344.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jgroups.blocks.MethodCall.invoke(MethodCall.java:330)
org.jgroups.blocks.RpcDispatcher.handle(RpcDispatcher.java:281)
org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:654)
org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:544)
org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:367)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:777)
org.jgroups.JChannel.up(JChannel.java:1091)
org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:382)
org.jgroups.stack.ProtocolStack.receiveUpEvent(ProtocolStack.java:398)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:158)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FRAG2.up(FRAG2.java:197)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FC.up(FC.java:422)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.GMS.up(GMS.java:768)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.protocols.pbcast.GMS.receiveUpEvent(GMS.java:788)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:262)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.UNICAST.up(UNICAST.java:259)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:685)
org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:517)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:170)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FD.up(FD.java:300)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.MERGE2.up(MERGE2.java:162)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.Discovery.up(Discovery.java:225)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.TP.handleIncomingMessage(TP.java:909)
org.jgroups.protocols.TP.handleIncomingPacket(TP.java:851)
org.jgroups.protocols.TP.access$400(TP.java:46)
org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1297)
java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:36 a.m.
New subject: [JBoss JIRA] Commented: (JBCACHE-1103) Potential deadlock with ClusteredCacheLoader
[ http://jira.jboss.com/jira/browse/JBCACHE-1103?page=comments#action_12365405 ]
Manik Surtani commented on JBCACHE-1103:
----------------------------------------
Once JGRP-533 is fixed and we have JGroups 2.4.1.SP4, I can patch the CCL to use this new,
overloaded RpcDispatcher call (will use reflection here so it doesn't break with older
JGroups releases) and this should sort this problem.
Potential deadlock with ClusteredCacheLoader
--------------------------------------------
Key: JBCACHE-1103
URL: http://jira.jboss.com/jira/browse/JBCACHE-1103
Project: JBoss Cache
Issue Type: Bug
Security Level: Public(Everyone can see)
Components: Clustering, Cache loaders
Affects Versions: 1.4.1.SP3
Reporter: Brian Stansberry
Assigned To: Manik Surtani
Fix For: 1.4.1.SP4
Can occur if
1) Thread1 on node1 is trying to get(fqnA) which is not in-VM, so the CCL tries to do a
clustered get. The CacheLoaderInterceptor lock for fqnA is held at this time.
2) Thread1 blocks in FC waiting for credits.
3) Replication message for fqnA arrives from node2.
4) IncomingPacketHandler thread will block waiting for the CacheLoaderInterceptor lock
for fqnA.
5) FC credits cannot arrive, so we deadlock.
Stack traces below show this.
Thread: TP-Processor623 : priority:5, demon:true, threadId:1637,
threadState:TIMED_WAITING,
threadLockName:EDU.oswego.cs.dl.util.concurrent.CondVar@5947cf38
java.lang.Object.wait(Native Method)
EDU.oswego.cs.dl.util.concurrent.CondVar.timedwait(CondVar.java:222)
org.jgroups.protocols.FC.handleDownMessage(FC.java:454)
org.jgroups.protocols.FC.down(FC.java:374)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.protocols.FC.receiveDownEvent(FC.java:368)
org.jgroups.stack.Protocol.passDown(Protocol.java:533)
org.jgroups.protocols.FRAG2.down(FRAG2.java:167)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.stack.Protocol.passDown(Protocol.java:533)
org.jgroups.protocols.pbcast.STATE_TRANSFER.down(STATE_TRANSFER.java:294)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:390)
org.jgroups.JChannel.down(JChannel.java:1231)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:790)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.passDown(MessageDispatcher.java:767)
org.jgroups.blocks.RequestCorrelator.sendRequest(RequestCorrelator.java:308)
org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:444)
org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:193)
org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:431)
org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:192)
org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:163)
org.jboss.cache.TreeCache.callRemoteMethodsViaReflection(TreeCache.java:4437)
org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4391)
org.jboss.cache.loader.ClusteredCacheLoader.callRemote(ClusteredCacheLoader.java:75)
org.jboss.cache.loader.ClusteredCacheLoader.get0(ClusteredCacheLoader.java:119)
org.jboss.cache.loader.ClusteredCacheLoader.get(ClusteredCacheLoader.java:113)
org.jboss.cache.interceptors.CacheLoaderInterceptor.loadData(CacheLoaderInterceptor.java:443)
org.jboss.cache.interceptors.CacheLoaderInterceptor.loadNode(CacheLoaderInterceptor.java:353)
org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:180)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:39)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:135)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:138)
org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
org.jboss.cache.TreeCache.get(TreeCache.java:3627)
org.jboss.cache.TreeCache.get(TreeCache.java:3608)
org.jboss.cache.TreeCache.get(TreeCache.java:3404)
sun.reflect.GeneratedMethodAccessor341.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
$Proxy60.get(Unknown Source)
org.jboss.web.tomcat.tc5.session.JBossCacheWrapper.get(JBossCacheWrapper.java:78)
org.jboss.web.tomcat.tc5.session.JBossCacheService.loadSession(JBossCacheService.java:247)
org.jboss.web.tomcat.tc5.session.JBossCacheManager.loadSession(JBossCacheManager.java:1006)
org.jboss.web.tomcat.tc5.session.JBossCacheManager.findSession(JBossCacheManager.java:778)
org.apache.catalina.connector.Request.doGetSession(Request.java:2196)
org.apache.catalina.connector.Request.getSession(Request.java:2024)
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:97)
org.jboss.web.tomcat.tc5.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:87)
org.jboss.web.tomcat.tc5.session.JvmRouteValve.invoke(JvmRouteValve.java:84)
org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)
org.jboss.web.tomcat.tc5.sso.ClusteredSingleSignOn.invoke(ClusteredSingleSignOn.java:637)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
java.lang.Thread.run(Thread.java:619)
Thread: IncomingPacketHandler (channel=Tomcat-LAB01) : priority:5, demon:true,
threadId:35, threadState:WAITING,
threadLockName:org.jboss.cache.interceptors.CacheLoaderInterceptor@5778e8ed
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.jboss.cache.interceptors.BaseCacheLoaderInterceptor.obtainLoaderLock(BaseCacheLoaderInterceptor.java:49)
org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:168)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:39)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:64)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:167)
org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
org.jboss.cache.TreeCache._replicate(TreeCache.java:5162)
sun.reflect.GeneratedMethodAccessor344.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jgroups.blocks.MethodCall.invoke(MethodCall.java:330)
org.jgroups.blocks.RpcDispatcher.handle(RpcDispatcher.java:281)
org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:654)
org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:544)
org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:367)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:777)
org.jgroups.JChannel.up(JChannel.java:1091)
org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:382)
org.jgroups.stack.ProtocolStack.receiveUpEvent(ProtocolStack.java:398)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:158)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FRAG2.up(FRAG2.java:197)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FC.up(FC.java:422)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.GMS.up(GMS.java:768)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.protocols.pbcast.GMS.receiveUpEvent(GMS.java:788)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:262)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.UNICAST.up(UNICAST.java:259)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:685)
org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:517)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:170)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FD.up(FD.java:300)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.MERGE2.up(MERGE2.java:162)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.Discovery.up(Discovery.java:225)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.TP.handleIncomingMessage(TP.java:909)
org.jgroups.protocols.TP.handleIncomingPacket(TP.java:851)
org.jgroups.protocols.TP.access$400(TP.java:46)
org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1297)
java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:36 a.m.
New subject: [JBoss JIRA] Updated: (JBCACHE-1103) Potential deadlock with ClusteredCacheLoader
[ http://jira.jboss.com/jira/browse/JBCACHE-1103?page=all ]
Manik Surtani updated JBCACHE-1103:
-----------------------------------
Fix Version/s: 1.4.1.SP4
Affects Version/s: (was: 2.0.0.CR2)
Potential deadlock with ClusteredCacheLoader
--------------------------------------------
Key: JBCACHE-1103
URL: http://jira.jboss.com/jira/browse/JBCACHE-1103
Project: JBoss Cache
Issue Type: Bug
Security Level: Public(Everyone can see)
Components: Clustering, Cache loaders
Affects Versions: 1.4.1.SP3
Reporter: Brian Stansberry
Assigned To: Manik Surtani
Fix For: 1.4.1.SP4
Can occur if
1) Thread1 on node1 is trying to get(fqnA) which is not in-VM, so the CCL tries to do a
clustered get. The CacheLoaderInterceptor lock for fqnA is held at this time.
2) Thread1 blocks in FC waiting for credits.
3) Replication message for fqnA arrives from node2.
4) IncomingPacketHandler thread will block waiting for the CacheLoaderInterceptor lock
for fqnA.
5) FC credits cannot arrive, so we deadlock.
Stack traces below show this.
Thread: TP-Processor623 : priority:5, demon:true, threadId:1637,
threadState:TIMED_WAITING,
threadLockName:EDU.oswego.cs.dl.util.concurrent.CondVar@5947cf38
java.lang.Object.wait(Native Method)
EDU.oswego.cs.dl.util.concurrent.CondVar.timedwait(CondVar.java:222)
org.jgroups.protocols.FC.handleDownMessage(FC.java:454)
org.jgroups.protocols.FC.down(FC.java:374)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.protocols.FC.receiveDownEvent(FC.java:368)
org.jgroups.stack.Protocol.passDown(Protocol.java:533)
org.jgroups.protocols.FRAG2.down(FRAG2.java:167)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.stack.Protocol.passDown(Protocol.java:533)
org.jgroups.protocols.pbcast.STATE_TRANSFER.down(STATE_TRANSFER.java:294)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:390)
org.jgroups.JChannel.down(JChannel.java:1231)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:790)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.passDown(MessageDispatcher.java:767)
org.jgroups.blocks.RequestCorrelator.sendRequest(RequestCorrelator.java:308)
org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:444)
org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:193)
org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:431)
org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:192)
org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:163)
org.jboss.cache.TreeCache.callRemoteMethodsViaReflection(TreeCache.java:4437)
org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4391)
org.jboss.cache.loader.ClusteredCacheLoader.callRemote(ClusteredCacheLoader.java:75)
org.jboss.cache.loader.ClusteredCacheLoader.get0(ClusteredCacheLoader.java:119)
org.jboss.cache.loader.ClusteredCacheLoader.get(ClusteredCacheLoader.java:113)
org.jboss.cache.interceptors.CacheLoaderInterceptor.loadData(CacheLoaderInterceptor.java:443)
org.jboss.cache.interceptors.CacheLoaderInterceptor.loadNode(CacheLoaderInterceptor.java:353)
org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:180)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:39)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:135)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:138)
org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
org.jboss.cache.TreeCache.get(TreeCache.java:3627)
org.jboss.cache.TreeCache.get(TreeCache.java:3608)
org.jboss.cache.TreeCache.get(TreeCache.java:3404)
sun.reflect.GeneratedMethodAccessor341.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
$Proxy60.get(Unknown Source)
org.jboss.web.tomcat.tc5.session.JBossCacheWrapper.get(JBossCacheWrapper.java:78)
org.jboss.web.tomcat.tc5.session.JBossCacheService.loadSession(JBossCacheService.java:247)
org.jboss.web.tomcat.tc5.session.JBossCacheManager.loadSession(JBossCacheManager.java:1006)
org.jboss.web.tomcat.tc5.session.JBossCacheManager.findSession(JBossCacheManager.java:778)
org.apache.catalina.connector.Request.doGetSession(Request.java:2196)
org.apache.catalina.connector.Request.getSession(Request.java:2024)
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:97)
org.jboss.web.tomcat.tc5.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:87)
org.jboss.web.tomcat.tc5.session.JvmRouteValve.invoke(JvmRouteValve.java:84)
org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)
org.jboss.web.tomcat.tc5.sso.ClusteredSingleSignOn.invoke(ClusteredSingleSignOn.java:637)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
java.lang.Thread.run(Thread.java:619)
Thread: IncomingPacketHandler (channel=Tomcat-LAB01) : priority:5, demon:true,
threadId:35, threadState:WAITING,
threadLockName:org.jboss.cache.interceptors.CacheLoaderInterceptor@5778e8ed
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.jboss.cache.interceptors.BaseCacheLoaderInterceptor.obtainLoaderLock(BaseCacheLoaderInterceptor.java:49)
org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:168)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:39)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:64)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:167)
org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
org.jboss.cache.TreeCache._replicate(TreeCache.java:5162)
sun.reflect.GeneratedMethodAccessor344.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jgroups.blocks.MethodCall.invoke(MethodCall.java:330)
org.jgroups.blocks.RpcDispatcher.handle(RpcDispatcher.java:281)
org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:654)
org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:544)
org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:367)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:777)
org.jgroups.JChannel.up(JChannel.java:1091)
org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:382)
org.jgroups.stack.ProtocolStack.receiveUpEvent(ProtocolStack.java:398)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:158)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FRAG2.up(FRAG2.java:197)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FC.up(FC.java:422)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.GMS.up(GMS.java:768)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.protocols.pbcast.GMS.receiveUpEvent(GMS.java:788)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:262)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.UNICAST.up(UNICAST.java:259)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:685)
org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:517)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:170)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FD.up(FD.java:300)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.MERGE2.up(MERGE2.java:162)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.Discovery.up(Discovery.java:225)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.TP.handleIncomingMessage(TP.java:909)
org.jgroups.protocols.TP.handleIncomingPacket(TP.java:851)
org.jgroups.protocols.TP.access$400(TP.java:46)
org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1297)
java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:36 p.m.
New subject: [JBoss JIRA] Closed: (JBCACHE-1103) Potential deadlock with ClusteredCacheLoader
[ http://jira.jboss.com/jira/browse/JBCACHE-1103?page=all ]
Manik Surtani closed JBCACHE-1103.
----------------------------------
Resolution: Done
Potential deadlock with ClusteredCacheLoader
--------------------------------------------
Key: JBCACHE-1103
URL: http://jira.jboss.com/jira/browse/JBCACHE-1103
Project: JBoss Cache
Issue Type: Bug
Security Level: Public(Everyone can see)
Components: Clustering, Cache loaders
Affects Versions: 1.4.1.SP3
Reporter: Brian Stansberry
Assigned To: Manik Surtani
Fix For: 1.4.1.SP4
Can occur if
1) Thread1 on node1 is trying to get(fqnA) which is not in-VM, so the CCL tries to do a
clustered get. The CacheLoaderInterceptor lock for fqnA is held at this time.
2) Thread1 blocks in FC waiting for credits.
3) Replication message for fqnA arrives from node2.
4) IncomingPacketHandler thread will block waiting for the CacheLoaderInterceptor lock
for fqnA.
5) FC credits cannot arrive, so we deadlock.
Stack traces below show this.
Thread: TP-Processor623 : priority:5, demon:true, threadId:1637,
threadState:TIMED_WAITING,
threadLockName:EDU.oswego.cs.dl.util.concurrent.CondVar@5947cf38
java.lang.Object.wait(Native Method)
EDU.oswego.cs.dl.util.concurrent.CondVar.timedwait(CondVar.java:222)
org.jgroups.protocols.FC.handleDownMessage(FC.java:454)
org.jgroups.protocols.FC.down(FC.java:374)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.protocols.FC.receiveDownEvent(FC.java:368)
org.jgroups.stack.Protocol.passDown(Protocol.java:533)
org.jgroups.protocols.FRAG2.down(FRAG2.java:167)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.stack.Protocol.passDown(Protocol.java:533)
org.jgroups.protocols.pbcast.STATE_TRANSFER.down(STATE_TRANSFER.java:294)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:390)
org.jgroups.JChannel.down(JChannel.java:1231)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:790)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.passDown(MessageDispatcher.java:767)
org.jgroups.blocks.RequestCorrelator.sendRequest(RequestCorrelator.java:308)
org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:444)
org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:193)
org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:431)
org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:192)
org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:163)
org.jboss.cache.TreeCache.callRemoteMethodsViaReflection(TreeCache.java:4437)
org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4391)
org.jboss.cache.loader.ClusteredCacheLoader.callRemote(ClusteredCacheLoader.java:75)
org.jboss.cache.loader.ClusteredCacheLoader.get0(ClusteredCacheLoader.java:119)
org.jboss.cache.loader.ClusteredCacheLoader.get(ClusteredCacheLoader.java:113)
org.jboss.cache.interceptors.CacheLoaderInterceptor.loadData(CacheLoaderInterceptor.java:443)
org.jboss.cache.interceptors.CacheLoaderInterceptor.loadNode(CacheLoaderInterceptor.java:353)
org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:180)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:39)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:135)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:138)
org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
org.jboss.cache.TreeCache.get(TreeCache.java:3627)
org.jboss.cache.TreeCache.get(TreeCache.java:3608)
org.jboss.cache.TreeCache.get(TreeCache.java:3404)
sun.reflect.GeneratedMethodAccessor341.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
$Proxy60.get(Unknown Source)
org.jboss.web.tomcat.tc5.session.JBossCacheWrapper.get(JBossCacheWrapper.java:78)
org.jboss.web.tomcat.tc5.session.JBossCacheService.loadSession(JBossCacheService.java:247)
org.jboss.web.tomcat.tc5.session.JBossCacheManager.loadSession(JBossCacheManager.java:1006)
org.jboss.web.tomcat.tc5.session.JBossCacheManager.findSession(JBossCacheManager.java:778)
org.apache.catalina.connector.Request.doGetSession(Request.java:2196)
org.apache.catalina.connector.Request.getSession(Request.java:2024)
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:97)
org.jboss.web.tomcat.tc5.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:87)
org.jboss.web.tomcat.tc5.session.JvmRouteValve.invoke(JvmRouteValve.java:84)
org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)
org.jboss.web.tomcat.tc5.sso.ClusteredSingleSignOn.invoke(ClusteredSingleSignOn.java:637)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
java.lang.Thread.run(Thread.java:619)
Thread: IncomingPacketHandler (channel=Tomcat-LAB01) : priority:5, demon:true,
threadId:35, threadState:WAITING,
threadLockName:org.jboss.cache.interceptors.CacheLoaderInterceptor@5778e8ed
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.jboss.cache.interceptors.BaseCacheLoaderInterceptor.obtainLoaderLock(BaseCacheLoaderInterceptor.java:49)
org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:168)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:39)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:64)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:167)
org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
org.jboss.cache.TreeCache._replicate(TreeCache.java:5162)
sun.reflect.GeneratedMethodAccessor344.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jgroups.blocks.MethodCall.invoke(MethodCall.java:330)
org.jgroups.blocks.RpcDispatcher.handle(RpcDispatcher.java:281)
org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:654)
org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:544)
org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:367)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:777)
org.jgroups.JChannel.up(JChannel.java:1091)
org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:382)
org.jgroups.stack.ProtocolStack.receiveUpEvent(ProtocolStack.java:398)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:158)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FRAG2.up(FRAG2.java:197)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FC.up(FC.java:422)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.GMS.up(GMS.java:768)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.protocols.pbcast.GMS.receiveUpEvent(GMS.java:788)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:262)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.UNICAST.up(UNICAST.java:259)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:685)
org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:517)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:170)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FD.up(FD.java:300)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.MERGE2.up(MERGE2.java:162)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.Discovery.up(Discovery.java:225)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.TP.handleIncomingMessage(TP.java:909)
org.jgroups.protocols.TP.handleIncomingPacket(TP.java:851)
org.jgroups.protocols.TP.access$400(TP.java:46)
org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1297)
java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:38 p.m.
New subject: [JBoss JIRA] Reopened: (JBCACHE-1103) Potential deadlock with ClusteredCacheLoader
[ http://jira.jboss.com/jira/browse/JBCACHE-1103?page=all ]
Manik Surtani reopened JBCACHE-1103:
------------------------------------
Potential deadlock with ClusteredCacheLoader
--------------------------------------------
Key: JBCACHE-1103
URL: http://jira.jboss.com/jira/browse/JBCACHE-1103
Project: JBoss Cache
Issue Type: Bug
Security Level: Public(Everyone can see)
Components: Clustering, Cache loaders
Affects Versions: 1.4.1.SP3
Reporter: Brian Stansberry
Assigned To: Manik Surtani
Fix For: 1.4.1.SP4
Can occur if
1) Thread1 on node1 is trying to get(fqnA) which is not in-VM, so the CCL tries to do a
clustered get. The CacheLoaderInterceptor lock for fqnA is held at this time.
2) Thread1 blocks in FC waiting for credits.
3) Replication message for fqnA arrives from node2.
4) IncomingPacketHandler thread will block waiting for the CacheLoaderInterceptor lock
for fqnA.
5) FC credits cannot arrive, so we deadlock.
Stack traces below show this.
Thread: TP-Processor623 : priority:5, demon:true, threadId:1637,
threadState:TIMED_WAITING,
threadLockName:EDU.oswego.cs.dl.util.concurrent.CondVar@5947cf38
java.lang.Object.wait(Native Method)
EDU.oswego.cs.dl.util.concurrent.CondVar.timedwait(CondVar.java:222)
org.jgroups.protocols.FC.handleDownMessage(FC.java:454)
org.jgroups.protocols.FC.down(FC.java:374)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.protocols.FC.receiveDownEvent(FC.java:368)
org.jgroups.stack.Protocol.passDown(Protocol.java:533)
org.jgroups.protocols.FRAG2.down(FRAG2.java:167)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.stack.Protocol.passDown(Protocol.java:533)
org.jgroups.protocols.pbcast.STATE_TRANSFER.down(STATE_TRANSFER.java:294)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:390)
org.jgroups.JChannel.down(JChannel.java:1231)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:790)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.passDown(MessageDispatcher.java:767)
org.jgroups.blocks.RequestCorrelator.sendRequest(RequestCorrelator.java:308)
org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:444)
org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:193)
org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:431)
org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:192)
org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:163)
org.jboss.cache.TreeCache.callRemoteMethodsViaReflection(TreeCache.java:4437)
org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4391)
org.jboss.cache.loader.ClusteredCacheLoader.callRemote(ClusteredCacheLoader.java:75)
org.jboss.cache.loader.ClusteredCacheLoader.get0(ClusteredCacheLoader.java:119)
org.jboss.cache.loader.ClusteredCacheLoader.get(ClusteredCacheLoader.java:113)
org.jboss.cache.interceptors.CacheLoaderInterceptor.loadData(CacheLoaderInterceptor.java:443)
org.jboss.cache.interceptors.CacheLoaderInterceptor.loadNode(CacheLoaderInterceptor.java:353)
org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:180)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:39)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:135)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:138)
org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
org.jboss.cache.TreeCache.get(TreeCache.java:3627)
org.jboss.cache.TreeCache.get(TreeCache.java:3608)
org.jboss.cache.TreeCache.get(TreeCache.java:3404)
sun.reflect.GeneratedMethodAccessor341.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
$Proxy60.get(Unknown Source)
org.jboss.web.tomcat.tc5.session.JBossCacheWrapper.get(JBossCacheWrapper.java:78)
org.jboss.web.tomcat.tc5.session.JBossCacheService.loadSession(JBossCacheService.java:247)
org.jboss.web.tomcat.tc5.session.JBossCacheManager.loadSession(JBossCacheManager.java:1006)
org.jboss.web.tomcat.tc5.session.JBossCacheManager.findSession(JBossCacheManager.java:778)
org.apache.catalina.connector.Request.doGetSession(Request.java:2196)
org.apache.catalina.connector.Request.getSession(Request.java:2024)
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:97)
org.jboss.web.tomcat.tc5.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:87)
org.jboss.web.tomcat.tc5.session.JvmRouteValve.invoke(JvmRouteValve.java:84)
org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)
org.jboss.web.tomcat.tc5.sso.ClusteredSingleSignOn.invoke(ClusteredSingleSignOn.java:637)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
java.lang.Thread.run(Thread.java:619)
Thread: IncomingPacketHandler (channel=Tomcat-LAB01) : priority:5, demon:true,
threadId:35, threadState:WAITING,
threadLockName:org.jboss.cache.interceptors.CacheLoaderInterceptor@5778e8ed
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.jboss.cache.interceptors.BaseCacheLoaderInterceptor.obtainLoaderLock(BaseCacheLoaderInterceptor.java:49)
org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:168)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:39)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:64)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:167)
org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
org.jboss.cache.TreeCache._replicate(TreeCache.java:5162)
sun.reflect.GeneratedMethodAccessor344.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jgroups.blocks.MethodCall.invoke(MethodCall.java:330)
org.jgroups.blocks.RpcDispatcher.handle(RpcDispatcher.java:281)
org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:654)
org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:544)
org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:367)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:777)
org.jgroups.JChannel.up(JChannel.java:1091)
org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:382)
org.jgroups.stack.ProtocolStack.receiveUpEvent(ProtocolStack.java:398)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:158)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FRAG2.up(FRAG2.java:197)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FC.up(FC.java:422)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.GMS.up(GMS.java:768)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.protocols.pbcast.GMS.receiveUpEvent(GMS.java:788)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:262)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.UNICAST.up(UNICAST.java:259)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:685)
org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:517)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:170)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FD.up(FD.java:300)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.MERGE2.up(MERGE2.java:162)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.Discovery.up(Discovery.java:225)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.TP.handleIncomingMessage(TP.java:909)
org.jgroups.protocols.TP.handleIncomingPacket(TP.java:851)
org.jgroups.protocols.TP.access$400(TP.java:46)
org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1297)
java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:40 p.m.
New subject: [JBoss JIRA] Updated: (JBCACHE-1103) Make cache loader implementations thread safe and remove synchronization in CacheLoaderInterceptor and CacheStoreInterceptor
[ http://jira.jboss.com/jira/browse/JBCACHE-1103?page=all ]
Manik Surtani updated JBCACHE-1103:
-----------------------------------
Summary: Make cache loader implementations thread safe and remove synchronization in
CacheLoaderInterceptor and CacheStoreInterceptor (was: Potential deadlock with
ClusteredCacheLoader)
The title of this JIRA used to be "Potential deadlock with
ClusteredCacheLoader"
Renaming to be more in line with the solution, which is that CacheLoaderInterceptor and
CacheStoreInterceptor should not attempt to deal with concurrent access to Fqns in the
cche loader - the cache loader implementations themselves should be thread safe.
And as a part of solving the problem in the main description, the ClusteredCacheLoader
does not synchronize on Fqn.
Make cache loader implementations thread safe and remove
synchronization in CacheLoaderInterceptor and CacheStoreInterceptor
----------------------------------------------------------------------------------------------------------------------------
Key: JBCACHE-1103
URL: http://jira.jboss.com/jira/browse/JBCACHE-1103
Project: JBoss Cache
Issue Type: Bug
Security Level: Public(Everyone can see)
Components: Clustering, Cache loaders
Affects Versions: 1.4.1.SP3
Reporter: Brian Stansberry
Assigned To: Manik Surtani
Fix For: 1.4.1.SP4
Can occur if
1) Thread1 on node1 is trying to get(fqnA) which is not in-VM, so the CCL tries to do a
clustered get. The CacheLoaderInterceptor lock for fqnA is held at this time.
2) Thread1 blocks in FC waiting for credits.
3) Replication message for fqnA arrives from node2.
4) IncomingPacketHandler thread will block waiting for the CacheLoaderInterceptor lock
for fqnA.
5) FC credits cannot arrive, so we deadlock.
Stack traces below show this.
Thread: TP-Processor623 : priority:5, demon:true, threadId:1637,
threadState:TIMED_WAITING,
threadLockName:EDU.oswego.cs.dl.util.concurrent.CondVar@5947cf38
java.lang.Object.wait(Native Method)
EDU.oswego.cs.dl.util.concurrent.CondVar.timedwait(CondVar.java:222)
org.jgroups.protocols.FC.handleDownMessage(FC.java:454)
org.jgroups.protocols.FC.down(FC.java:374)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.protocols.FC.receiveDownEvent(FC.java:368)
org.jgroups.stack.Protocol.passDown(Protocol.java:533)
org.jgroups.protocols.FRAG2.down(FRAG2.java:167)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.stack.Protocol.passDown(Protocol.java:533)
org.jgroups.protocols.pbcast.STATE_TRANSFER.down(STATE_TRANSFER.java:294)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:390)
org.jgroups.JChannel.down(JChannel.java:1231)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:790)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.passDown(MessageDispatcher.java:767)
org.jgroups.blocks.RequestCorrelator.sendRequest(RequestCorrelator.java:308)
org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:444)
org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:193)
org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:431)
org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:192)
org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:163)
org.jboss.cache.TreeCache.callRemoteMethodsViaReflection(TreeCache.java:4437)
org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4391)
org.jboss.cache.loader.ClusteredCacheLoader.callRemote(ClusteredCacheLoader.java:75)
org.jboss.cache.loader.ClusteredCacheLoader.get0(ClusteredCacheLoader.java:119)
org.jboss.cache.loader.ClusteredCacheLoader.get(ClusteredCacheLoader.java:113)
org.jboss.cache.interceptors.CacheLoaderInterceptor.loadData(CacheLoaderInterceptor.java:443)
org.jboss.cache.interceptors.CacheLoaderInterceptor.loadNode(CacheLoaderInterceptor.java:353)
org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:180)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:39)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:135)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:138)
org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
org.jboss.cache.TreeCache.get(TreeCache.java:3627)
org.jboss.cache.TreeCache.get(TreeCache.java:3608)
org.jboss.cache.TreeCache.get(TreeCache.java:3404)
sun.reflect.GeneratedMethodAccessor341.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
$Proxy60.get(Unknown Source)
org.jboss.web.tomcat.tc5.session.JBossCacheWrapper.get(JBossCacheWrapper.java:78)
org.jboss.web.tomcat.tc5.session.JBossCacheService.loadSession(JBossCacheService.java:247)
org.jboss.web.tomcat.tc5.session.JBossCacheManager.loadSession(JBossCacheManager.java:1006)
org.jboss.web.tomcat.tc5.session.JBossCacheManager.findSession(JBossCacheManager.java:778)
org.apache.catalina.connector.Request.doGetSession(Request.java:2196)
org.apache.catalina.connector.Request.getSession(Request.java:2024)
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:97)
org.jboss.web.tomcat.tc5.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:87)
org.jboss.web.tomcat.tc5.session.JvmRouteValve.invoke(JvmRouteValve.java:84)
org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)
org.jboss.web.tomcat.tc5.sso.ClusteredSingleSignOn.invoke(ClusteredSingleSignOn.java:637)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
java.lang.Thread.run(Thread.java:619)
Thread: IncomingPacketHandler (channel=Tomcat-LAB01) : priority:5, demon:true,
threadId:35, threadState:WAITING,
threadLockName:org.jboss.cache.interceptors.CacheLoaderInterceptor@5778e8ed
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.jboss.cache.interceptors.BaseCacheLoaderInterceptor.obtainLoaderLock(BaseCacheLoaderInterceptor.java:49)
org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:168)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:39)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:64)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:167)
org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
org.jboss.cache.TreeCache._replicate(TreeCache.java:5162)
sun.reflect.GeneratedMethodAccessor344.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jgroups.blocks.MethodCall.invoke(MethodCall.java:330)
org.jgroups.blocks.RpcDispatcher.handle(RpcDispatcher.java:281)
org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:654)
org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:544)
org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:367)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:777)
org.jgroups.JChannel.up(JChannel.java:1091)
org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:382)
org.jgroups.stack.ProtocolStack.receiveUpEvent(ProtocolStack.java:398)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:158)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FRAG2.up(FRAG2.java:197)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FC.up(FC.java:422)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.GMS.up(GMS.java:768)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.protocols.pbcast.GMS.receiveUpEvent(GMS.java:788)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:262)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.UNICAST.up(UNICAST.java:259)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:685)
org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:517)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:170)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FD.up(FD.java:300)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.MERGE2.up(MERGE2.java:162)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.Discovery.up(Discovery.java:225)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.TP.handleIncomingMessage(TP.java:909)
org.jgroups.protocols.TP.handleIncomingPacket(TP.java:851)
org.jgroups.protocols.TP.access$400(TP.java:46)
org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1297)
java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:42 p.m.
New subject: [JBoss JIRA] Commented: (JBCACHE-1103) Make cache loader implementations thread safe and remove synchronization in CacheLoaderInterceptor and CacheStoreInterceptor
[ http://jira.jboss.com/jira/browse/JBCACHE-1103?page=comments#action_12369367 ]
Manik Surtani commented on JBCACHE-1103:
----------------------------------------
See discussion thread on the jbosscache-dev mail list to see how we arrived at this
solution:
http://lists.jboss.org/pipermail/jbosscache-dev/2007-June/001657.html
Make cache loader implementations thread safe and remove
synchronization in CacheLoaderInterceptor and CacheStoreInterceptor
----------------------------------------------------------------------------------------------------------------------------
Key: JBCACHE-1103
URL: http://jira.jboss.com/jira/browse/JBCACHE-1103
Project: JBoss Cache
Issue Type: Bug
Security Level: Public(Everyone can see)
Components: Clustering, Cache loaders
Affects Versions: 1.4.1.SP3
Reporter: Brian Stansberry
Assigned To: Manik Surtani
Fix For: 1.4.1.SP4
Can occur if
1) Thread1 on node1 is trying to get(fqnA) which is not in-VM, so the CCL tries to do a
clustered get. The CacheLoaderInterceptor lock for fqnA is held at this time.
2) Thread1 blocks in FC waiting for credits.
3) Replication message for fqnA arrives from node2.
4) IncomingPacketHandler thread will block waiting for the CacheLoaderInterceptor lock
for fqnA.
5) FC credits cannot arrive, so we deadlock.
Stack traces below show this.
Thread: TP-Processor623 : priority:5, demon:true, threadId:1637,
threadState:TIMED_WAITING,
threadLockName:EDU.oswego.cs.dl.util.concurrent.CondVar@5947cf38
java.lang.Object.wait(Native Method)
EDU.oswego.cs.dl.util.concurrent.CondVar.timedwait(CondVar.java:222)
org.jgroups.protocols.FC.handleDownMessage(FC.java:454)
org.jgroups.protocols.FC.down(FC.java:374)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.protocols.FC.receiveDownEvent(FC.java:368)
org.jgroups.stack.Protocol.passDown(Protocol.java:533)
org.jgroups.protocols.FRAG2.down(FRAG2.java:167)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.stack.Protocol.passDown(Protocol.java:533)
org.jgroups.protocols.pbcast.STATE_TRANSFER.down(STATE_TRANSFER.java:294)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:390)
org.jgroups.JChannel.down(JChannel.java:1231)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:790)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.passDown(MessageDispatcher.java:767)
org.jgroups.blocks.RequestCorrelator.sendRequest(RequestCorrelator.java:308)
org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:444)
org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:193)
org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:431)
org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:192)
org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:163)
org.jboss.cache.TreeCache.callRemoteMethodsViaReflection(TreeCache.java:4437)
org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4391)
org.jboss.cache.loader.ClusteredCacheLoader.callRemote(ClusteredCacheLoader.java:75)
org.jboss.cache.loader.ClusteredCacheLoader.get0(ClusteredCacheLoader.java:119)
org.jboss.cache.loader.ClusteredCacheLoader.get(ClusteredCacheLoader.java:113)
org.jboss.cache.interceptors.CacheLoaderInterceptor.loadData(CacheLoaderInterceptor.java:443)
org.jboss.cache.interceptors.CacheLoaderInterceptor.loadNode(CacheLoaderInterceptor.java:353)
org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:180)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:39)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:135)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:138)
org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
org.jboss.cache.TreeCache.get(TreeCache.java:3627)
org.jboss.cache.TreeCache.get(TreeCache.java:3608)
org.jboss.cache.TreeCache.get(TreeCache.java:3404)
sun.reflect.GeneratedMethodAccessor341.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
$Proxy60.get(Unknown Source)
org.jboss.web.tomcat.tc5.session.JBossCacheWrapper.get(JBossCacheWrapper.java:78)
org.jboss.web.tomcat.tc5.session.JBossCacheService.loadSession(JBossCacheService.java:247)
org.jboss.web.tomcat.tc5.session.JBossCacheManager.loadSession(JBossCacheManager.java:1006)
org.jboss.web.tomcat.tc5.session.JBossCacheManager.findSession(JBossCacheManager.java:778)
org.apache.catalina.connector.Request.doGetSession(Request.java:2196)
org.apache.catalina.connector.Request.getSession(Request.java:2024)
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:97)
org.jboss.web.tomcat.tc5.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:87)
org.jboss.web.tomcat.tc5.session.JvmRouteValve.invoke(JvmRouteValve.java:84)
org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)
org.jboss.web.tomcat.tc5.sso.ClusteredSingleSignOn.invoke(ClusteredSingleSignOn.java:637)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
java.lang.Thread.run(Thread.java:619)
Thread: IncomingPacketHandler (channel=Tomcat-LAB01) : priority:5, demon:true,
threadId:35, threadState:WAITING,
threadLockName:org.jboss.cache.interceptors.CacheLoaderInterceptor@5778e8ed
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.jboss.cache.interceptors.BaseCacheLoaderInterceptor.obtainLoaderLock(BaseCacheLoaderInterceptor.java:49)
org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:168)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:39)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:64)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:167)
org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
org.jboss.cache.TreeCache._replicate(TreeCache.java:5162)
sun.reflect.GeneratedMethodAccessor344.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jgroups.blocks.MethodCall.invoke(MethodCall.java:330)
org.jgroups.blocks.RpcDispatcher.handle(RpcDispatcher.java:281)
org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:654)
org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:544)
org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:367)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:777)
org.jgroups.JChannel.up(JChannel.java:1091)
org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:382)
org.jgroups.stack.ProtocolStack.receiveUpEvent(ProtocolStack.java:398)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:158)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FRAG2.up(FRAG2.java:197)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FC.up(FC.java:422)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.GMS.up(GMS.java:768)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.protocols.pbcast.GMS.receiveUpEvent(GMS.java:788)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:262)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.UNICAST.up(UNICAST.java:259)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:685)
org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:517)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:170)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FD.up(FD.java:300)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.MERGE2.up(MERGE2.java:162)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.Discovery.up(Discovery.java:225)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.TP.handleIncomingMessage(TP.java:909)
org.jgroups.protocols.TP.handleIncomingPacket(TP.java:851)
org.jgroups.protocols.TP.access$400(TP.java:46)
org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1297)
java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:42 p.m.
New subject: [JBoss JIRA] Closed: (JBCACHE-1103) Make cache loader implementations thread safe and remove synchronization in CacheLoaderInterceptor and CacheStoreInterceptor
[ http://jira.jboss.com/jira/browse/JBCACHE-1103?page=all ]
Manik Surtani closed JBCACHE-1103.
----------------------------------
Resolution: Done
Make cache loader implementations thread safe and remove
synchronization in CacheLoaderInterceptor and CacheStoreInterceptor
----------------------------------------------------------------------------------------------------------------------------
Key: JBCACHE-1103
URL: http://jira.jboss.com/jira/browse/JBCACHE-1103
Project: JBoss Cache
Issue Type: Bug
Security Level: Public(Everyone can see)
Components: Clustering, Cache loaders
Affects Versions: 1.4.1.SP3
Reporter: Brian Stansberry
Assigned To: Manik Surtani
Fix For: 1.4.1.SP4
Can occur if
1) Thread1 on node1 is trying to get(fqnA) which is not in-VM, so the CCL tries to do a
clustered get. The CacheLoaderInterceptor lock for fqnA is held at this time.
2) Thread1 blocks in FC waiting for credits.
3) Replication message for fqnA arrives from node2.
4) IncomingPacketHandler thread will block waiting for the CacheLoaderInterceptor lock
for fqnA.
5) FC credits cannot arrive, so we deadlock.
Stack traces below show this.
Thread: TP-Processor623 : priority:5, demon:true, threadId:1637,
threadState:TIMED_WAITING,
threadLockName:EDU.oswego.cs.dl.util.concurrent.CondVar@5947cf38
java.lang.Object.wait(Native Method)
EDU.oswego.cs.dl.util.concurrent.CondVar.timedwait(CondVar.java:222)
org.jgroups.protocols.FC.handleDownMessage(FC.java:454)
org.jgroups.protocols.FC.down(FC.java:374)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.protocols.FC.receiveDownEvent(FC.java:368)
org.jgroups.stack.Protocol.passDown(Protocol.java:533)
org.jgroups.protocols.FRAG2.down(FRAG2.java:167)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.stack.Protocol.passDown(Protocol.java:533)
org.jgroups.protocols.pbcast.STATE_TRANSFER.down(STATE_TRANSFER.java:294)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:390)
org.jgroups.JChannel.down(JChannel.java:1231)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:790)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.passDown(MessageDispatcher.java:767)
org.jgroups.blocks.RequestCorrelator.sendRequest(RequestCorrelator.java:308)
org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:444)
org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:193)
org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:431)
org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:192)
org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:163)
org.jboss.cache.TreeCache.callRemoteMethodsViaReflection(TreeCache.java:4437)
org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4391)
org.jboss.cache.loader.ClusteredCacheLoader.callRemote(ClusteredCacheLoader.java:75)
org.jboss.cache.loader.ClusteredCacheLoader.get0(ClusteredCacheLoader.java:119)
org.jboss.cache.loader.ClusteredCacheLoader.get(ClusteredCacheLoader.java:113)
org.jboss.cache.interceptors.CacheLoaderInterceptor.loadData(CacheLoaderInterceptor.java:443)
org.jboss.cache.interceptors.CacheLoaderInterceptor.loadNode(CacheLoaderInterceptor.java:353)
org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:180)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:39)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:135)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:138)
org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
org.jboss.cache.TreeCache.get(TreeCache.java:3627)
org.jboss.cache.TreeCache.get(TreeCache.java:3608)
org.jboss.cache.TreeCache.get(TreeCache.java:3404)
sun.reflect.GeneratedMethodAccessor341.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
$Proxy60.get(Unknown Source)
org.jboss.web.tomcat.tc5.session.JBossCacheWrapper.get(JBossCacheWrapper.java:78)
org.jboss.web.tomcat.tc5.session.JBossCacheService.loadSession(JBossCacheService.java:247)
org.jboss.web.tomcat.tc5.session.JBossCacheManager.loadSession(JBossCacheManager.java:1006)
org.jboss.web.tomcat.tc5.session.JBossCacheManager.findSession(JBossCacheManager.java:778)
org.apache.catalina.connector.Request.doGetSession(Request.java:2196)
org.apache.catalina.connector.Request.getSession(Request.java:2024)
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:97)
org.jboss.web.tomcat.tc5.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:87)
org.jboss.web.tomcat.tc5.session.JvmRouteValve.invoke(JvmRouteValve.java:84)
org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)
org.jboss.web.tomcat.tc5.sso.ClusteredSingleSignOn.invoke(ClusteredSingleSignOn.java:637)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
java.lang.Thread.run(Thread.java:619)
Thread: IncomingPacketHandler (channel=Tomcat-LAB01) : priority:5, demon:true,
threadId:35, threadState:WAITING,
threadLockName:org.jboss.cache.interceptors.CacheLoaderInterceptor@5778e8ed
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.jboss.cache.interceptors.BaseCacheLoaderInterceptor.obtainLoaderLock(BaseCacheLoaderInterceptor.java:49)
org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:168)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:39)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:64)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:167)
org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
org.jboss.cache.TreeCache._replicate(TreeCache.java:5162)
sun.reflect.GeneratedMethodAccessor344.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jgroups.blocks.MethodCall.invoke(MethodCall.java:330)
org.jgroups.blocks.RpcDispatcher.handle(RpcDispatcher.java:281)
org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:654)
org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:544)
org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:367)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:777)
org.jgroups.JChannel.up(JChannel.java:1091)
org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:382)
org.jgroups.stack.ProtocolStack.receiveUpEvent(ProtocolStack.java:398)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:158)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FRAG2.up(FRAG2.java:197)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FC.up(FC.java:422)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.GMS.up(GMS.java:768)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.protocols.pbcast.GMS.receiveUpEvent(GMS.java:788)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:262)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.UNICAST.up(UNICAST.java:259)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:685)
org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:517)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:170)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FD.up(FD.java:300)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.MERGE2.up(MERGE2.java:162)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.Discovery.up(Discovery.java:225)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.TP.handleIncomingMessage(TP.java:909)
org.jgroups.protocols.TP.handleIncomingPacket(TP.java:851)
org.jgroups.protocols.TP.access$400(TP.java:46)
org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1297)
java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:42 p.m.
New subject: [JBoss JIRA] Reopened: (JBCACHE-1103) Make cache loader implementations thread safe and remove synchronization in CacheLoaderInterceptor and CacheStoreInterceptor
[ http://jira.jboss.com/jira/browse/JBCACHE-1103?page=all ]
Manik Surtani reopened JBCACHE-1103:
------------------------------------
Make cache loader implementations thread safe and remove
synchronization in CacheLoaderInterceptor and CacheStoreInterceptor
----------------------------------------------------------------------------------------------------------------------------
Key: JBCACHE-1103
URL: http://jira.jboss.com/jira/browse/JBCACHE-1103
Project: JBoss Cache
Issue Type: Bug
Security Level: Public(Everyone can see)
Components: Clustering, Cache loaders
Affects Versions: 1.4.1.SP3
Reporter: Brian Stansberry
Assigned To: Manik Surtani
Fix For: 1.4.1.SP4
Can occur if
1) Thread1 on node1 is trying to get(fqnA) which is not in-VM, so the CCL tries to do a
clustered get. The CacheLoaderInterceptor lock for fqnA is held at this time.
2) Thread1 blocks in FC waiting for credits.
3) Replication message for fqnA arrives from node2.
4) IncomingPacketHandler thread will block waiting for the CacheLoaderInterceptor lock
for fqnA.
5) FC credits cannot arrive, so we deadlock.
Stack traces below show this.
Thread: TP-Processor623 : priority:5, demon:true, threadId:1637,
threadState:TIMED_WAITING,
threadLockName:EDU.oswego.cs.dl.util.concurrent.CondVar@5947cf38
java.lang.Object.wait(Native Method)
EDU.oswego.cs.dl.util.concurrent.CondVar.timedwait(CondVar.java:222)
org.jgroups.protocols.FC.handleDownMessage(FC.java:454)
org.jgroups.protocols.FC.down(FC.java:374)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.protocols.FC.receiveDownEvent(FC.java:368)
org.jgroups.stack.Protocol.passDown(Protocol.java:533)
org.jgroups.protocols.FRAG2.down(FRAG2.java:167)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.stack.Protocol.passDown(Protocol.java:533)
org.jgroups.protocols.pbcast.STATE_TRANSFER.down(STATE_TRANSFER.java:294)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:390)
org.jgroups.JChannel.down(JChannel.java:1231)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:790)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.passDown(MessageDispatcher.java:767)
org.jgroups.blocks.RequestCorrelator.sendRequest(RequestCorrelator.java:308)
org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:444)
org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:193)
org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:431)
org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:192)
org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:163)
org.jboss.cache.TreeCache.callRemoteMethodsViaReflection(TreeCache.java:4437)
org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4391)
org.jboss.cache.loader.ClusteredCacheLoader.callRemote(ClusteredCacheLoader.java:75)
org.jboss.cache.loader.ClusteredCacheLoader.get0(ClusteredCacheLoader.java:119)
org.jboss.cache.loader.ClusteredCacheLoader.get(ClusteredCacheLoader.java:113)
org.jboss.cache.interceptors.CacheLoaderInterceptor.loadData(CacheLoaderInterceptor.java:443)
org.jboss.cache.interceptors.CacheLoaderInterceptor.loadNode(CacheLoaderInterceptor.java:353)
org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:180)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:39)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:135)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:138)
org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
org.jboss.cache.TreeCache.get(TreeCache.java:3627)
org.jboss.cache.TreeCache.get(TreeCache.java:3608)
org.jboss.cache.TreeCache.get(TreeCache.java:3404)
sun.reflect.GeneratedMethodAccessor341.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
$Proxy60.get(Unknown Source)
org.jboss.web.tomcat.tc5.session.JBossCacheWrapper.get(JBossCacheWrapper.java:78)
org.jboss.web.tomcat.tc5.session.JBossCacheService.loadSession(JBossCacheService.java:247)
org.jboss.web.tomcat.tc5.session.JBossCacheManager.loadSession(JBossCacheManager.java:1006)
org.jboss.web.tomcat.tc5.session.JBossCacheManager.findSession(JBossCacheManager.java:778)
org.apache.catalina.connector.Request.doGetSession(Request.java:2196)
org.apache.catalina.connector.Request.getSession(Request.java:2024)
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:97)
org.jboss.web.tomcat.tc5.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:87)
org.jboss.web.tomcat.tc5.session.JvmRouteValve.invoke(JvmRouteValve.java:84)
org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)
org.jboss.web.tomcat.tc5.sso.ClusteredSingleSignOn.invoke(ClusteredSingleSignOn.java:637)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
java.lang.Thread.run(Thread.java:619)
Thread: IncomingPacketHandler (channel=Tomcat-LAB01) : priority:5, demon:true,
threadId:35, threadState:WAITING,
threadLockName:org.jboss.cache.interceptors.CacheLoaderInterceptor@5778e8ed
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.jboss.cache.interceptors.BaseCacheLoaderInterceptor.obtainLoaderLock(BaseCacheLoaderInterceptor.java:49)
org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:168)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:39)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:64)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:167)
org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
org.jboss.cache.TreeCache._replicate(TreeCache.java:5162)
sun.reflect.GeneratedMethodAccessor344.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jgroups.blocks.MethodCall.invoke(MethodCall.java:330)
org.jgroups.blocks.RpcDispatcher.handle(RpcDispatcher.java:281)
org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:654)
org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:544)
org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:367)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:777)
org.jgroups.JChannel.up(JChannel.java:1091)
org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:382)
org.jgroups.stack.ProtocolStack.receiveUpEvent(ProtocolStack.java:398)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:158)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FRAG2.up(FRAG2.java:197)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FC.up(FC.java:422)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.GMS.up(GMS.java:768)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.protocols.pbcast.GMS.receiveUpEvent(GMS.java:788)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:262)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.UNICAST.up(UNICAST.java:259)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:685)
org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:517)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:170)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FD.up(FD.java:300)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.MERGE2.up(MERGE2.java:162)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.Discovery.up(Discovery.java:225)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.TP.handleIncomingMessage(TP.java:909)
org.jgroups.protocols.TP.handleIncomingPacket(TP.java:851)
org.jgroups.protocols.TP.access$400(TP.java:46)
org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1297)
java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:44 p.m.
New subject: [JBoss JIRA] Closed: (JBCACHE-1103) Make cache loader implementations thread safe and remove synchronization in CacheLoaderInterceptor and CacheStoreInterceptor
[ http://jira.jboss.com/jira/browse/JBCACHE-1103?page=all ]
Manik Surtani closed JBCACHE-1103.
----------------------------------
Fix Version/s: 2.0.0.GA
Resolution: Done
Make cache loader implementations thread safe and remove
synchronization in CacheLoaderInterceptor and CacheStoreInterceptor
----------------------------------------------------------------------------------------------------------------------------
Key: JBCACHE-1103
URL: http://jira.jboss.com/jira/browse/JBCACHE-1103
Project: JBoss Cache
Issue Type: Bug
Security Level: Public(Everyone can see)
Components: Clustering, Cache loaders
Affects Versions: 1.4.1.SP3
Reporter: Brian Stansberry
Assigned To: Manik Surtani
Fix For: 2.0.0.GA, 1.4.1.SP4
Can occur if
1) Thread1 on node1 is trying to get(fqnA) which is not in-VM, so the CCL tries to do a
clustered get. The CacheLoaderInterceptor lock for fqnA is held at this time.
2) Thread1 blocks in FC waiting for credits.
3) Replication message for fqnA arrives from node2.
4) IncomingPacketHandler thread will block waiting for the CacheLoaderInterceptor lock
for fqnA.
5) FC credits cannot arrive, so we deadlock.
Stack traces below show this.
Thread: TP-Processor623 : priority:5, demon:true, threadId:1637,
threadState:TIMED_WAITING,
threadLockName:EDU.oswego.cs.dl.util.concurrent.CondVar@5947cf38
java.lang.Object.wait(Native Method)
EDU.oswego.cs.dl.util.concurrent.CondVar.timedwait(CondVar.java:222)
org.jgroups.protocols.FC.handleDownMessage(FC.java:454)
org.jgroups.protocols.FC.down(FC.java:374)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.protocols.FC.receiveDownEvent(FC.java:368)
org.jgroups.stack.Protocol.passDown(Protocol.java:533)
org.jgroups.protocols.FRAG2.down(FRAG2.java:167)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.stack.Protocol.passDown(Protocol.java:533)
org.jgroups.protocols.pbcast.STATE_TRANSFER.down(STATE_TRANSFER.java:294)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:390)
org.jgroups.JChannel.down(JChannel.java:1231)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:790)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.passDown(MessageDispatcher.java:767)
org.jgroups.blocks.RequestCorrelator.sendRequest(RequestCorrelator.java:308)
org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:444)
org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:193)
org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:431)
org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:192)
org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:163)
org.jboss.cache.TreeCache.callRemoteMethodsViaReflection(TreeCache.java:4437)
org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4391)
org.jboss.cache.loader.ClusteredCacheLoader.callRemote(ClusteredCacheLoader.java:75)
org.jboss.cache.loader.ClusteredCacheLoader.get0(ClusteredCacheLoader.java:119)
org.jboss.cache.loader.ClusteredCacheLoader.get(ClusteredCacheLoader.java:113)
org.jboss.cache.interceptors.CacheLoaderInterceptor.loadData(CacheLoaderInterceptor.java:443)
org.jboss.cache.interceptors.CacheLoaderInterceptor.loadNode(CacheLoaderInterceptor.java:353)
org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:180)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:39)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:135)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:138)
org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
org.jboss.cache.TreeCache.get(TreeCache.java:3627)
org.jboss.cache.TreeCache.get(TreeCache.java:3608)
org.jboss.cache.TreeCache.get(TreeCache.java:3404)
sun.reflect.GeneratedMethodAccessor341.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
$Proxy60.get(Unknown Source)
org.jboss.web.tomcat.tc5.session.JBossCacheWrapper.get(JBossCacheWrapper.java:78)
org.jboss.web.tomcat.tc5.session.JBossCacheService.loadSession(JBossCacheService.java:247)
org.jboss.web.tomcat.tc5.session.JBossCacheManager.loadSession(JBossCacheManager.java:1006)
org.jboss.web.tomcat.tc5.session.JBossCacheManager.findSession(JBossCacheManager.java:778)
org.apache.catalina.connector.Request.doGetSession(Request.java:2196)
org.apache.catalina.connector.Request.getSession(Request.java:2024)
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:97)
org.jboss.web.tomcat.tc5.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:87)
org.jboss.web.tomcat.tc5.session.JvmRouteValve.invoke(JvmRouteValve.java:84)
org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)
org.jboss.web.tomcat.tc5.sso.ClusteredSingleSignOn.invoke(ClusteredSingleSignOn.java:637)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
java.lang.Thread.run(Thread.java:619)
Thread: IncomingPacketHandler (channel=Tomcat-LAB01) : priority:5, demon:true,
threadId:35, threadState:WAITING,
threadLockName:org.jboss.cache.interceptors.CacheLoaderInterceptor@5778e8ed
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.jboss.cache.interceptors.BaseCacheLoaderInterceptor.obtainLoaderLock(BaseCacheLoaderInterceptor.java:49)
org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:168)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:39)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:64)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:167)
org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
org.jboss.cache.TreeCache._replicate(TreeCache.java:5162)
sun.reflect.GeneratedMethodAccessor344.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jgroups.blocks.MethodCall.invoke(MethodCall.java:330)
org.jgroups.blocks.RpcDispatcher.handle(RpcDispatcher.java:281)
org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:654)
org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:544)
org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:367)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:777)
org.jgroups.JChannel.up(JChannel.java:1091)
org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:382)
org.jgroups.stack.ProtocolStack.receiveUpEvent(ProtocolStack.java:398)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:158)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FRAG2.up(FRAG2.java:197)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FC.up(FC.java:422)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.GMS.up(GMS.java:768)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.protocols.pbcast.GMS.receiveUpEvent(GMS.java:788)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:262)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.UNICAST.up(UNICAST.java:259)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:685)
org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:517)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:170)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FD.up(FD.java:300)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.MERGE2.up(MERGE2.java:162)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.Discovery.up(Discovery.java:225)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.TP.handleIncomingMessage(TP.java:909)
org.jgroups.protocols.TP.handleIncomingPacket(TP.java:851)
org.jgroups.protocols.TP.access$400(TP.java:46)
org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1297)
java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:08 p.m.
New subject: [JBoss JIRA] Commented: (JBCACHE-1103) Make cache loader implementations thread safe and remove synchronization in CacheLoaderInterceptor and CacheStoreInterceptor
[ http://jira.jboss.com/jira/browse/JBCACHE-1103?page=comments#action_12370459 ]
David Stringer commented on JBCACHE-1103:
-----------------------------------------
We were running into this issue with SP3. After upgrading to SP4 we are now deadlocking
somewhere else. Was a upgrade of jgroups necessary as well ? Just using the one that was
released with JbossCache SP4.
(Our environment, Jboss 4.0.5, Jboss Cache SP4, RHEL3, java 1.5.0_11)
Here is the thread stack trace on a locked thread --
State: BLOCKED on org.jgroups.protocols.UNICAST$Entry@c70737 owned by:
IncomingPacketHandler (channel=ARCHLABSBK-Tomcat)
Total blocked: 102 Total waited: 1,157
Stack trace:
org.jgroups.protocols.UNICAST.down(UNICAST.java:265)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
org.jgroups.stack.Protocol.passDown(Protocol.java:551)
org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:283)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
org.jgroups.stack.Protocol.passDown(Protocol.java:551)
org.jgroups.protocols.pbcast.GMS.down(GMS.java:840)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
org.jgroups.stack.Protocol.passDown(Protocol.java:551)
org.jgroups.protocols.FC.sendCreditRequest(FC.java:567)
org.jgroups.protocols.FC.handleDownMessage(FC.java:390)
org.jgroups.protocols.FC.down(FC.java:320)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
org.jgroups.protocols.FC.receiveDownEvent(FC.java:314)
org.jgroups.stack.Protocol.passDown(Protocol.java:551)
org.jgroups.protocols.FRAG2.down(FRAG2.java:167)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
org.jgroups.stack.Protocol.passDown(Protocol.java:551)
org.jgroups.protocols.pbcast.STATE_TRANSFER.down(STATE_TRANSFER.java:294)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:385)
org.jgroups.JChannel.down(JChannel.java:1231)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:788)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.passDown(MessageDispatcher.java:765)
org.jgroups.blocks.RequestCorrelator.sendRequest(RequestCorrelator.java:295)
org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:444)
org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:193)
org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:429)
org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:192)
sun.reflect.GeneratedMethodAccessor106.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jboss.cache.TreeCache.callRemoteMethodsViaReflection(TreeCache.java:4435)
org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4391)
org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4344)
org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4455)
org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:110)
org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:88)
org.jboss.cache.interceptors.ReplicationInterceptor.handleReplicatedMethod(ReplicationInterceptor.java:119)
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:88)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:167)
org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
org.jboss.cache.TreeCache.put(TreeCache.java:3759)
sun.reflect.GeneratedMethodAccessor142.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
$Proxy54.put(Unknown Source)
org.jboss.web.tomcat.tc5.session.JBossCacheWrapper.put(JBossCacheWrapper.java:141)
org.jboss.web.tomcat.tc5.session.JBossCacheService.putSession(JBossCacheService.java:315)
org.jboss.web.tomcat.tc5.session.JBossCacheClusteredSession.processSessionRepl(JBossCacheClusteredSession.java:121)
- locked org.jboss.web.tomcat.tc5.session.SessionBasedClusteredSession@8e8c11
org.jboss.web.tomcat.tc5.session.JBossCacheManager.processSessionRepl(JBossCacheManager.java:1093)
org.jboss.web.tomcat.tc5.session.JBossCacheManager.storeSession(JBossCacheManager.java:648)
- locked org.jboss.web.tomcat.tc5.session.SessionBasedClusteredSession@8e8c11
org.jboss.web.tomcat.tc5.session.InstantSnapshotManager.snapshot(InstantSnapshotManager.java:49)
org.jboss.web.tomcat.tc5.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:98)
org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)
org.apache.catalina.valves.FastCommonAccessLogValve.invoke(FastCommonAccessLogValve.java:495)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
java.lang.Thread.run(Thread.java:619)
Make cache loader implementations thread safe and remove
synchronization in CacheLoaderInterceptor and CacheStoreInterceptor
----------------------------------------------------------------------------------------------------------------------------
Key: JBCACHE-1103
URL: http://jira.jboss.com/jira/browse/JBCACHE-1103
Project: JBoss Cache
Issue Type: Bug
Security Level: Public(Everyone can see)
Components: Clustering, Cache loaders
Affects Versions: 1.4.1.SP3
Reporter: Brian Stansberry
Assigned To: Manik Surtani
Fix For: 2.0.0.GA, 1.4.1.SP4
Can occur if
1) Thread1 on node1 is trying to get(fqnA) which is not in-VM, so the CCL tries to do a
clustered get. The CacheLoaderInterceptor lock for fqnA is held at this time.
2) Thread1 blocks in FC waiting for credits.
3) Replication message for fqnA arrives from node2.
4) IncomingPacketHandler thread will block waiting for the CacheLoaderInterceptor lock
for fqnA.
5) FC credits cannot arrive, so we deadlock.
Stack traces below show this.
Thread: TP-Processor623 : priority:5, demon:true, threadId:1637,
threadState:TIMED_WAITING,
threadLockName:EDU.oswego.cs.dl.util.concurrent.CondVar@5947cf38
java.lang.Object.wait(Native Method)
EDU.oswego.cs.dl.util.concurrent.CondVar.timedwait(CondVar.java:222)
org.jgroups.protocols.FC.handleDownMessage(FC.java:454)
org.jgroups.protocols.FC.down(FC.java:374)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.protocols.FC.receiveDownEvent(FC.java:368)
org.jgroups.stack.Protocol.passDown(Protocol.java:533)
org.jgroups.protocols.FRAG2.down(FRAG2.java:167)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.stack.Protocol.passDown(Protocol.java:533)
org.jgroups.protocols.pbcast.STATE_TRANSFER.down(STATE_TRANSFER.java:294)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:390)
org.jgroups.JChannel.down(JChannel.java:1231)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:790)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.passDown(MessageDispatcher.java:767)
org.jgroups.blocks.RequestCorrelator.sendRequest(RequestCorrelator.java:308)
org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:444)
org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:193)
org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:431)
org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:192)
org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:163)
org.jboss.cache.TreeCache.callRemoteMethodsViaReflection(TreeCache.java:4437)
org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4391)
org.jboss.cache.loader.ClusteredCacheLoader.callRemote(ClusteredCacheLoader.java:75)
org.jboss.cache.loader.ClusteredCacheLoader.get0(ClusteredCacheLoader.java:119)
org.jboss.cache.loader.ClusteredCacheLoader.get(ClusteredCacheLoader.java:113)
org.jboss.cache.interceptors.CacheLoaderInterceptor.loadData(CacheLoaderInterceptor.java:443)
org.jboss.cache.interceptors.CacheLoaderInterceptor.loadNode(CacheLoaderInterceptor.java:353)
org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:180)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:39)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:135)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:138)
org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
org.jboss.cache.TreeCache.get(TreeCache.java:3627)
org.jboss.cache.TreeCache.get(TreeCache.java:3608)
org.jboss.cache.TreeCache.get(TreeCache.java:3404)
sun.reflect.GeneratedMethodAccessor341.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
$Proxy60.get(Unknown Source)
org.jboss.web.tomcat.tc5.session.JBossCacheWrapper.get(JBossCacheWrapper.java:78)
org.jboss.web.tomcat.tc5.session.JBossCacheService.loadSession(JBossCacheService.java:247)
org.jboss.web.tomcat.tc5.session.JBossCacheManager.loadSession(JBossCacheManager.java:1006)
org.jboss.web.tomcat.tc5.session.JBossCacheManager.findSession(JBossCacheManager.java:778)
org.apache.catalina.connector.Request.doGetSession(Request.java:2196)
org.apache.catalina.connector.Request.getSession(Request.java:2024)
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:97)
org.jboss.web.tomcat.tc5.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:87)
org.jboss.web.tomcat.tc5.session.JvmRouteValve.invoke(JvmRouteValve.java:84)
org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)
org.jboss.web.tomcat.tc5.sso.ClusteredSingleSignOn.invoke(ClusteredSingleSignOn.java:637)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
java.lang.Thread.run(Thread.java:619)
Thread: IncomingPacketHandler (channel=Tomcat-LAB01) : priority:5, demon:true,
threadId:35, threadState:WAITING,
threadLockName:org.jboss.cache.interceptors.CacheLoaderInterceptor@5778e8ed
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.jboss.cache.interceptors.BaseCacheLoaderInterceptor.obtainLoaderLock(BaseCacheLoaderInterceptor.java:49)
org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:168)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:39)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:64)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:167)
org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
org.jboss.cache.TreeCache._replicate(TreeCache.java:5162)
sun.reflect.GeneratedMethodAccessor344.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jgroups.blocks.MethodCall.invoke(MethodCall.java:330)
org.jgroups.blocks.RpcDispatcher.handle(RpcDispatcher.java:281)
org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:654)
org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:544)
org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:367)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:777)
org.jgroups.JChannel.up(JChannel.java:1091)
org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:382)
org.jgroups.stack.ProtocolStack.receiveUpEvent(ProtocolStack.java:398)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:158)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FRAG2.up(FRAG2.java:197)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FC.up(FC.java:422)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.GMS.up(GMS.java:768)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.protocols.pbcast.GMS.receiveUpEvent(GMS.java:788)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:262)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.UNICAST.up(UNICAST.java:259)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:685)
org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:517)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:170)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FD.up(FD.java:300)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.MERGE2.up(MERGE2.java:162)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.Discovery.up(Discovery.java:225)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.TP.handleIncomingMessage(TP.java:909)
org.jgroups.protocols.TP.handleIncomingPacket(TP.java:851)
org.jgroups.protocols.TP.access$400(TP.java:46)
org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1297)
java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:35 p.m.
New subject: [JBoss JIRA] Commented: (JBCACHE-1103) Make cache loader implementations thread safe and remove synchronization in CacheLoaderInterceptor and CacheStoreInterceptor
[ http://jira.jboss.com/jira/browse/JBCACHE-1103?page=comments#action_12370464 ]
Manik Surtani commented on JBCACHE-1103:
----------------------------------------
Does your app use the ClusteredCacheLoader? Do you have a thread dump when the deadlock
occurs?
Make cache loader implementations thread safe and remove
synchronization in CacheLoaderInterceptor and CacheStoreInterceptor
----------------------------------------------------------------------------------------------------------------------------
Key: JBCACHE-1103
URL: http://jira.jboss.com/jira/browse/JBCACHE-1103
Project: JBoss Cache
Issue Type: Bug
Security Level: Public(Everyone can see)
Components: Clustering, Cache loaders
Affects Versions: 1.4.1.SP3
Reporter: Brian Stansberry
Assigned To: Manik Surtani
Fix For: 2.0.0.GA, 1.4.1.SP4
Can occur if
1) Thread1 on node1 is trying to get(fqnA) which is not in-VM, so the CCL tries to do a
clustered get. The CacheLoaderInterceptor lock for fqnA is held at this time.
2) Thread1 blocks in FC waiting for credits.
3) Replication message for fqnA arrives from node2.
4) IncomingPacketHandler thread will block waiting for the CacheLoaderInterceptor lock
for fqnA.
5) FC credits cannot arrive, so we deadlock.
Stack traces below show this.
Thread: TP-Processor623 : priority:5, demon:true, threadId:1637,
threadState:TIMED_WAITING,
threadLockName:EDU.oswego.cs.dl.util.concurrent.CondVar@5947cf38
java.lang.Object.wait(Native Method)
EDU.oswego.cs.dl.util.concurrent.CondVar.timedwait(CondVar.java:222)
org.jgroups.protocols.FC.handleDownMessage(FC.java:454)
org.jgroups.protocols.FC.down(FC.java:374)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.protocols.FC.receiveDownEvent(FC.java:368)
org.jgroups.stack.Protocol.passDown(Protocol.java:533)
org.jgroups.protocols.FRAG2.down(FRAG2.java:167)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.stack.Protocol.passDown(Protocol.java:533)
org.jgroups.protocols.pbcast.STATE_TRANSFER.down(STATE_TRANSFER.java:294)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:390)
org.jgroups.JChannel.down(JChannel.java:1231)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:790)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.passDown(MessageDispatcher.java:767)
org.jgroups.blocks.RequestCorrelator.sendRequest(RequestCorrelator.java:308)
org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:444)
org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:193)
org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:431)
org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:192)
org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:163)
org.jboss.cache.TreeCache.callRemoteMethodsViaReflection(TreeCache.java:4437)
org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4391)
org.jboss.cache.loader.ClusteredCacheLoader.callRemote(ClusteredCacheLoader.java:75)
org.jboss.cache.loader.ClusteredCacheLoader.get0(ClusteredCacheLoader.java:119)
org.jboss.cache.loader.ClusteredCacheLoader.get(ClusteredCacheLoader.java:113)
org.jboss.cache.interceptors.CacheLoaderInterceptor.loadData(CacheLoaderInterceptor.java:443)
org.jboss.cache.interceptors.CacheLoaderInterceptor.loadNode(CacheLoaderInterceptor.java:353)
org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:180)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:39)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:135)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:138)
org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
org.jboss.cache.TreeCache.get(TreeCache.java:3627)
org.jboss.cache.TreeCache.get(TreeCache.java:3608)
org.jboss.cache.TreeCache.get(TreeCache.java:3404)
sun.reflect.GeneratedMethodAccessor341.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
$Proxy60.get(Unknown Source)
org.jboss.web.tomcat.tc5.session.JBossCacheWrapper.get(JBossCacheWrapper.java:78)
org.jboss.web.tomcat.tc5.session.JBossCacheService.loadSession(JBossCacheService.java:247)
org.jboss.web.tomcat.tc5.session.JBossCacheManager.loadSession(JBossCacheManager.java:1006)
org.jboss.web.tomcat.tc5.session.JBossCacheManager.findSession(JBossCacheManager.java:778)
org.apache.catalina.connector.Request.doGetSession(Request.java:2196)
org.apache.catalina.connector.Request.getSession(Request.java:2024)
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:97)
org.jboss.web.tomcat.tc5.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:87)
org.jboss.web.tomcat.tc5.session.JvmRouteValve.invoke(JvmRouteValve.java:84)
org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)
org.jboss.web.tomcat.tc5.sso.ClusteredSingleSignOn.invoke(ClusteredSingleSignOn.java:637)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
java.lang.Thread.run(Thread.java:619)
Thread: IncomingPacketHandler (channel=Tomcat-LAB01) : priority:5, demon:true,
threadId:35, threadState:WAITING,
threadLockName:org.jboss.cache.interceptors.CacheLoaderInterceptor@5778e8ed
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.jboss.cache.interceptors.BaseCacheLoaderInterceptor.obtainLoaderLock(BaseCacheLoaderInterceptor.java:49)
org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:168)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:39)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:64)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:167)
org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
org.jboss.cache.TreeCache._replicate(TreeCache.java:5162)
sun.reflect.GeneratedMethodAccessor344.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jgroups.blocks.MethodCall.invoke(MethodCall.java:330)
org.jgroups.blocks.RpcDispatcher.handle(RpcDispatcher.java:281)
org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:654)
org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:544)
org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:367)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:777)
org.jgroups.JChannel.up(JChannel.java:1091)
org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:382)
org.jgroups.stack.ProtocolStack.receiveUpEvent(ProtocolStack.java:398)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:158)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FRAG2.up(FRAG2.java:197)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FC.up(FC.java:422)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.GMS.up(GMS.java:768)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.protocols.pbcast.GMS.receiveUpEvent(GMS.java:788)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:262)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.UNICAST.up(UNICAST.java:259)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:685)
org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:517)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:170)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FD.up(FD.java:300)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.MERGE2.up(MERGE2.java:162)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.Discovery.up(Discovery.java:225)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.TP.handleIncomingMessage(TP.java:909)
org.jgroups.protocols.TP.handleIncomingPacket(TP.java:851)
org.jgroups.protocols.TP.access$400(TP.java:46)
org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1297)
java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:02 p.m.
New subject: [JBoss JIRA] Commented: (JBCACHE-1103) Make cache loader implementations thread safe and remove synchronization in CacheLoaderInterceptor and CacheStoreInterceptor
[ http://jira.jboss.com/jira/browse/JBCACHE-1103?page=comments#action_12370467 ]
David Stringer commented on JBCACHE-1103:
-----------------------------------------
We are just using the Cache in buddy mode for session replication. I don't see
anything in the jboss-service.xml to indicate that we are using the ClusteredCacheLoader.
Do you have a recommendation for getting the threaddump ? I could run hprof and dump
everything.. ? The stack trace above was from jconsole, and looking at the HTTP thread...
Make cache loader implementations thread safe and remove
synchronization in CacheLoaderInterceptor and CacheStoreInterceptor
----------------------------------------------------------------------------------------------------------------------------
Key: JBCACHE-1103
URL: http://jira.jboss.com/jira/browse/JBCACHE-1103
Project: JBoss Cache
Issue Type: Bug
Security Level: Public(Everyone can see)
Components: Clustering, Cache loaders
Affects Versions: 1.4.1.SP3
Reporter: Brian Stansberry
Assigned To: Manik Surtani
Fix For: 2.0.0.GA, 1.4.1.SP4
Can occur if
1) Thread1 on node1 is trying to get(fqnA) which is not in-VM, so the CCL tries to do a
clustered get. The CacheLoaderInterceptor lock for fqnA is held at this time.
2) Thread1 blocks in FC waiting for credits.
3) Replication message for fqnA arrives from node2.
4) IncomingPacketHandler thread will block waiting for the CacheLoaderInterceptor lock
for fqnA.
5) FC credits cannot arrive, so we deadlock.
Stack traces below show this.
Thread: TP-Processor623 : priority:5, demon:true, threadId:1637,
threadState:TIMED_WAITING,
threadLockName:EDU.oswego.cs.dl.util.concurrent.CondVar@5947cf38
java.lang.Object.wait(Native Method)
EDU.oswego.cs.dl.util.concurrent.CondVar.timedwait(CondVar.java:222)
org.jgroups.protocols.FC.handleDownMessage(FC.java:454)
org.jgroups.protocols.FC.down(FC.java:374)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.protocols.FC.receiveDownEvent(FC.java:368)
org.jgroups.stack.Protocol.passDown(Protocol.java:533)
org.jgroups.protocols.FRAG2.down(FRAG2.java:167)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.stack.Protocol.passDown(Protocol.java:533)
org.jgroups.protocols.pbcast.STATE_TRANSFER.down(STATE_TRANSFER.java:294)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:390)
org.jgroups.JChannel.down(JChannel.java:1231)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:790)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.passDown(MessageDispatcher.java:767)
org.jgroups.blocks.RequestCorrelator.sendRequest(RequestCorrelator.java:308)
org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:444)
org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:193)
org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:431)
org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:192)
org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:163)
org.jboss.cache.TreeCache.callRemoteMethodsViaReflection(TreeCache.java:4437)
org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4391)
org.jboss.cache.loader.ClusteredCacheLoader.callRemote(ClusteredCacheLoader.java:75)
org.jboss.cache.loader.ClusteredCacheLoader.get0(ClusteredCacheLoader.java:119)
org.jboss.cache.loader.ClusteredCacheLoader.get(ClusteredCacheLoader.java:113)
org.jboss.cache.interceptors.CacheLoaderInterceptor.loadData(CacheLoaderInterceptor.java:443)
org.jboss.cache.interceptors.CacheLoaderInterceptor.loadNode(CacheLoaderInterceptor.java:353)
org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:180)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:39)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:135)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:138)
org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
org.jboss.cache.TreeCache.get(TreeCache.java:3627)
org.jboss.cache.TreeCache.get(TreeCache.java:3608)
org.jboss.cache.TreeCache.get(TreeCache.java:3404)
sun.reflect.GeneratedMethodAccessor341.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
$Proxy60.get(Unknown Source)
org.jboss.web.tomcat.tc5.session.JBossCacheWrapper.get(JBossCacheWrapper.java:78)
org.jboss.web.tomcat.tc5.session.JBossCacheService.loadSession(JBossCacheService.java:247)
org.jboss.web.tomcat.tc5.session.JBossCacheManager.loadSession(JBossCacheManager.java:1006)
org.jboss.web.tomcat.tc5.session.JBossCacheManager.findSession(JBossCacheManager.java:778)
org.apache.catalina.connector.Request.doGetSession(Request.java:2196)
org.apache.catalina.connector.Request.getSession(Request.java:2024)
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:97)
org.jboss.web.tomcat.tc5.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:87)
org.jboss.web.tomcat.tc5.session.JvmRouteValve.invoke(JvmRouteValve.java:84)
org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)
org.jboss.web.tomcat.tc5.sso.ClusteredSingleSignOn.invoke(ClusteredSingleSignOn.java:637)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
java.lang.Thread.run(Thread.java:619)
Thread: IncomingPacketHandler (channel=Tomcat-LAB01) : priority:5, demon:true,
threadId:35, threadState:WAITING,
threadLockName:org.jboss.cache.interceptors.CacheLoaderInterceptor@5778e8ed
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.jboss.cache.interceptors.BaseCacheLoaderInterceptor.obtainLoaderLock(BaseCacheLoaderInterceptor.java:49)
org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:168)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:39)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:64)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:167)
org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
org.jboss.cache.TreeCache._replicate(TreeCache.java:5162)
sun.reflect.GeneratedMethodAccessor344.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jgroups.blocks.MethodCall.invoke(MethodCall.java:330)
org.jgroups.blocks.RpcDispatcher.handle(RpcDispatcher.java:281)
org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:654)
org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:544)
org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:367)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:777)
org.jgroups.JChannel.up(JChannel.java:1091)
org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:382)
org.jgroups.stack.ProtocolStack.receiveUpEvent(ProtocolStack.java:398)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:158)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FRAG2.up(FRAG2.java:197)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FC.up(FC.java:422)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.GMS.up(GMS.java:768)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.protocols.pbcast.GMS.receiveUpEvent(GMS.java:788)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:262)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.UNICAST.up(UNICAST.java:259)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:685)
org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:517)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:170)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FD.up(FD.java:300)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.MERGE2.up(MERGE2.java:162)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.Discovery.up(Discovery.java:225)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.TP.handleIncomingMessage(TP.java:909)
org.jgroups.protocols.TP.handleIncomingPacket(TP.java:851)
org.jgroups.protocols.TP.access$400(TP.java:46)
org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1297)
java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:19 p.m.
New subject: [JBoss JIRA] Commented: (JBCACHE-1103) Make cache loader implementations thread safe and remove synchronization in CacheLoaderInterceptor and CacheStoreInterceptor
[ http://jira.jboss.com/jira/browse/JBCACHE-1103?page=comments#action_12370472 ]
Manik Surtani commented on JBCACHE-1103:
----------------------------------------
Are you running any cache loader at all? I fear your issue is not related to this bug at
all. Start a thread on the user forums. Get a thread dump using kill -3
Make cache loader implementations thread safe and remove
synchronization in CacheLoaderInterceptor and CacheStoreInterceptor
----------------------------------------------------------------------------------------------------------------------------
Key: JBCACHE-1103
URL: http://jira.jboss.com/jira/browse/JBCACHE-1103
Project: JBoss Cache
Issue Type: Bug
Security Level: Public(Everyone can see)
Components: Clustering, Cache loaders
Affects Versions: 1.4.1.SP3
Reporter: Brian Stansberry
Assigned To: Manik Surtani
Fix For: 2.0.0.GA, 1.4.1.SP4
Can occur if
1) Thread1 on node1 is trying to get(fqnA) which is not in-VM, so the CCL tries to do a
clustered get. The CacheLoaderInterceptor lock for fqnA is held at this time.
2) Thread1 blocks in FC waiting for credits.
3) Replication message for fqnA arrives from node2.
4) IncomingPacketHandler thread will block waiting for the CacheLoaderInterceptor lock
for fqnA.
5) FC credits cannot arrive, so we deadlock.
Stack traces below show this.
Thread: TP-Processor623 : priority:5, demon:true, threadId:1637,
threadState:TIMED_WAITING,
threadLockName:EDU.oswego.cs.dl.util.concurrent.CondVar@5947cf38
java.lang.Object.wait(Native Method)
EDU.oswego.cs.dl.util.concurrent.CondVar.timedwait(CondVar.java:222)
org.jgroups.protocols.FC.handleDownMessage(FC.java:454)
org.jgroups.protocols.FC.down(FC.java:374)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.protocols.FC.receiveDownEvent(FC.java:368)
org.jgroups.stack.Protocol.passDown(Protocol.java:533)
org.jgroups.protocols.FRAG2.down(FRAG2.java:167)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.stack.Protocol.passDown(Protocol.java:533)
org.jgroups.protocols.pbcast.STATE_TRANSFER.down(STATE_TRANSFER.java:294)
org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:390)
org.jgroups.JChannel.down(JChannel.java:1231)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:790)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.passDown(MessageDispatcher.java:767)
org.jgroups.blocks.RequestCorrelator.sendRequest(RequestCorrelator.java:308)
org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:444)
org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:193)
org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:431)
org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:192)
org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:163)
org.jboss.cache.TreeCache.callRemoteMethodsViaReflection(TreeCache.java:4437)
org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4391)
org.jboss.cache.loader.ClusteredCacheLoader.callRemote(ClusteredCacheLoader.java:75)
org.jboss.cache.loader.ClusteredCacheLoader.get0(ClusteredCacheLoader.java:119)
org.jboss.cache.loader.ClusteredCacheLoader.get(ClusteredCacheLoader.java:113)
org.jboss.cache.interceptors.CacheLoaderInterceptor.loadData(CacheLoaderInterceptor.java:443)
org.jboss.cache.interceptors.CacheLoaderInterceptor.loadNode(CacheLoaderInterceptor.java:353)
org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:180)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:39)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:135)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:138)
org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
org.jboss.cache.TreeCache.get(TreeCache.java:3627)
org.jboss.cache.TreeCache.get(TreeCache.java:3608)
org.jboss.cache.TreeCache.get(TreeCache.java:3404)
sun.reflect.GeneratedMethodAccessor341.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
$Proxy60.get(Unknown Source)
org.jboss.web.tomcat.tc5.session.JBossCacheWrapper.get(JBossCacheWrapper.java:78)
org.jboss.web.tomcat.tc5.session.JBossCacheService.loadSession(JBossCacheService.java:247)
org.jboss.web.tomcat.tc5.session.JBossCacheManager.loadSession(JBossCacheManager.java:1006)
org.jboss.web.tomcat.tc5.session.JBossCacheManager.findSession(JBossCacheManager.java:778)
org.apache.catalina.connector.Request.doGetSession(Request.java:2196)
org.apache.catalina.connector.Request.getSession(Request.java:2024)
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:97)
org.jboss.web.tomcat.tc5.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:87)
org.jboss.web.tomcat.tc5.session.JvmRouteValve.invoke(JvmRouteValve.java:84)
org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)
org.jboss.web.tomcat.tc5.sso.ClusteredSingleSignOn.invoke(ClusteredSingleSignOn.java:637)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
java.lang.Thread.run(Thread.java:619)
Thread: IncomingPacketHandler (channel=Tomcat-LAB01) : priority:5, demon:true,
threadId:35, threadState:WAITING,
threadLockName:org.jboss.cache.interceptors.CacheLoaderInterceptor@5778e8ed
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.jboss.cache.interceptors.BaseCacheLoaderInterceptor.obtainLoaderLock(BaseCacheLoaderInterceptor.java:49)
org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:168)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:39)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:64)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:167)
org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
org.jboss.cache.TreeCache._replicate(TreeCache.java:5162)
sun.reflect.GeneratedMethodAccessor344.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.jgroups.blocks.MethodCall.invoke(MethodCall.java:330)
org.jgroups.blocks.RpcDispatcher.handle(RpcDispatcher.java:281)
org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:654)
org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:544)
org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:367)
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:777)
org.jgroups.JChannel.up(JChannel.java:1091)
org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:382)
org.jgroups.stack.ProtocolStack.receiveUpEvent(ProtocolStack.java:398)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:158)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FRAG2.up(FRAG2.java:197)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FC.up(FC.java:422)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.GMS.up(GMS.java:768)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.protocols.pbcast.GMS.receiveUpEvent(GMS.java:788)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:262)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.UNICAST.up(UNICAST.java:259)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:685)
org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:517)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:170)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FD.up(FD.java:300)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.MERGE2.up(MERGE2.java:162)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.Discovery.up(Discovery.java:225)
org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470)
org.jgroups.stack.Protocol.passUp(Protocol.java:520)
org.jgroups.protocols.TP.handleIncomingMessage(TP.java:909)
org.jgroups.protocols.TP.handleIncomingPacket(TP.java:851)
org.jgroups.protocols.TP.access$400(TP.java:46)
org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1297)
java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
6876
days inactive
6924
days old
17 comments
3 participants
participants (3)
-
Brian Stansberry (JIRA) -
David Stringer (JIRA)
-
Manik Surtani (JIRA)