[jboss-user] [Clustering/JBoss] - Session Replication deadlock

stringaling do-not-reply at jboss.com
Tue Jul 31 16:11:54 EDT 2007 

Hi,

We are using Jboss 4.0.5 in a cluster. We are experiencing deadlock within the Jboss Cache, which is causing out of memory problems on the server. The Jboss Cache Version is 1.4.1SP3 (bundled Cache for jboss 4.0.5).  We only expirience this issue when BUDDY replication is enabled. 

The thread dump reveils a lot of threads waiting on a lock for jgroups..

"http-0.0.0.0-9006-11" daemon prio=1 tid=0x086eb710 nid=0x719 waiting for monitor entry [0x87d44000..0x87d45600]
        at org.jgroups.protocols.UNICAST.down(UNICAST.java:265)
        - waiting to lock <0x9246a478> (a org.jgroups.protocols.UNICAST$Entry)
        at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
        at org.jgroups.stack.Protocol.passDown(Protocol.java:551)
        at org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:283)
        at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
        at org.jgroups.stack.Protocol.passDown(Protocol.java:551)
        at org.jgroups.protocols.pbcast.GMS.down(GMS.java:840)
        at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
        at org.jgroups.stack.Protocol.passDown(Protocol.java:551)
        at org.jgroups.protocols.FC.sendCreditRequest(FC.java:567)
        at org.jgroups.protocols.FC.handleDownMessage(FC.java:390)
        at org.jgroups.protocols.FC.down(FC.java:320)
        at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
        at org.jgroups.protocols.FC.receiveDownEvent(FC.java:314)
        at org.jgroups.stack.Protocol.passDown(Protocol.java:551)
        at org.jgroups.protocols.FRAG2.down(FRAG2.java:167)
        at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
        at org.jgroups.stack.Protocol.passDown(Protocol.java:551)
        at org.jgroups.protocols.pbcast.STATE_TRANSFER.down(STATE_TRANSFER.java:294)
        at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
        at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:385)
        at org.jgroups.JChannel.down(JChannel.java:1231)
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:788)
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.passDown(MessageDispatcher.java:765)
        at org.jgroups.blocks.RequestCorrelator.sendRequest(RequestCorrelator.java:295)
        at org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:444)
        at org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:193)
        at org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:429)
        at org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:192)
        at sun.reflect.GeneratedMethodAccessor177.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.jboss.cache.TreeCache.callRemoteMethodsViaReflection(TreeCache.java:4435)
        at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4391)
        at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4344)
        at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4455)
        at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:110)
        at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:88)
        at org.jboss.cache.interceptors.ReplicationInterceptor.handleReplicatedMethod(ReplicationInterceptor.java:119)
        at org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:88)
        at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
        at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
        at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
        at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
        at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:167)
        at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
        at org.jboss.cache.TreeCache.put(TreeCache.java:3759)
        at sun.reflect.GeneratedMethodAccessor186.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
        at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
        at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
        at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
        at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
        at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
        at $Proxy57.put(Unknown Source)
        at org.jboss.web.tomcat.tc5.session.JBossCacheWrapper.put(JBossCacheWrapper.java:141)
        at org.jboss.web.tomcat.tc5.session.JBossCacheService.putSession(JBossCacheService.java:315)
        at org.jboss.web.tomcat.tc5.session.JBossCacheClusteredSession.processSessionRepl(JBossCacheClusteredSession.java:121)
        - locked <0x93ffa170> (a org.jboss.web.tomcat.tc5.session.SessionBasedClusteredSession)
        at org.jboss.web.tomcat.tc5.session.JBossCacheManager.processSessionRepl(JBossCacheManager.java:1093)
        at org.jboss.web.tomcat.tc5.session.JBossCacheManager.storeSession(JBossCacheManager.java:648)
        - locked <0x93ffa170> (a org.jboss.web.tomcat.tc5.session.SessionBasedClusteredSession)
        at org.jboss.web.tomcat.tc5.session.InstantSnapshotManager.snapshot(InstantSnapshotManager.java:49)
        at org.jboss.web.tomcat.tc5.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:98)
        at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
        at org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)
        at org.apache.catalina.valves.FastCommonAccessLogValve.invoke(FastCommonAccessLogValve.java:495)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
        at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
        at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)


Here is the object that has the lock: 

http-0.0.0.0-9006-12" daemon prio=1 tid=0x8a563918 nid=0x71a waiting for monitor entry [0x87dc5000..0x87dc6480]
        at org.jgroups.stack.Retransmitter$Entry.(Retransmitter.java:272)
        at org.jgroups.stack.Retransmitter.add(Retransmitter.java:104)
        - locked <0x9246a670> (a java.util.LinkedList)
        at org.jgroups.stack.AckSenderWindow.add(AckSenderWindow.java:100)
        at org.jgroups.protocols.UNICAST.down(UNICAST.java:279)
        - locked <0x9246a478> (a org.jgroups.protocols.UNICAST$Entry)
        at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
        at org.jgroups.stack.Protocol.passDown(Protocol.java:551)
        at org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:283)
        at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
        at org.jgroups.stack.Protocol.passDown(Protocol.java:551)
        at org.jgroups.protocols.pbcast.GMS.down(GMS.java:840)
        at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
        at org.jgroups.stack.Protocol.passDown(Protocol.java:551)
        at org.jgroups.protocols.FC.sendCreditRequest(FC.java:567)
        at org.jgroups.protocols.FC.handleDownMessage(FC.java:390)
        at org.jgroups.protocols.FC.down(FC.java:320)
        at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
        at org.jgroups.protocols.FC.receiveDownEvent(FC.java:314)
        at org.jgroups.stack.Protocol.passDown(Protocol.java:551)
        at org.jgroups.protocols.FRAG2.down(FRAG2.java:167)
        at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
        at org.jgroups.stack.Protocol.passDown(Protocol.java:551)
        at org.jgroups.protocols.pbcast.STATE_TRANSFER.down(STATE_TRANSFER.java:294)
        at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
        at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:385)
        at org.jgroups.JChannel.down(JChannel.java:1231)
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:788)
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.passDown(MessageDispatcher.java:765)
        at org.jgroups.blocks.RequestCorrelator.sendRequest(RequestCorrelator.java:295)
        at org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:444)
        at org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:193)
        at org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:429)
        at org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:192)
        at sun.reflect.GeneratedMethodAccessor177.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.jboss.cache.TreeCache.callRemoteMethodsViaReflection(TreeCache.java:4435)
        at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4391)
        at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4344)
        at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4455)
        at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:110)
        at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:88)
        at org.jboss.cache.interceptors.ReplicationInterceptor.handleReplicatedMethod(ReplicationInterceptor.java:119)
        at org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:88)
        at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
        at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
        at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
        at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
        at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:167)
        at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
        at org.jboss.cache.TreeCache.put(TreeCache.java:3759)
...


There are some JIRA's for Jboss Cache that appear to have a similar stack trace (533, 1103), but they are Caceloader related ?

Any help ?!?!

Thanks,
David

View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4069357#4069357

Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4069357

More information about the jboss-user mailing list