We are running our application in JBoss 4.2.0 GA and Hibernate 3.x. Two JBoss application servers are running in cluster mode.
All of sudden we got this below exception and App server 1 (node1) went down:
2012-12-04 14:23:18,568 WARN org.jboss.web.tomcat.service.session.InstantSnapshotManager./et Failed to replicate session BvlehVnGPW2uuH09kn+Prw**.node1
java.lang.RuntimeException: JBossCacheService: exception occurred in cache put after retry ...
at org.jboss.web.tomcat.service.session.JBossCacheWrapper.put(JBossCacheWrapper.java:153)
at org.jboss.web.tomcat.service.session.JBossCacheService.putSession(JBossCacheService.java:319)
at org.jboss.web.tomcat.service.session.JBossCacheClusteredSession.processSessionRepl(JBossCacheClusteredSession.java:121)
at org.jboss.web.tomcat.service.session.JBossCacheManager.processSessionRepl(JBossCacheManager.java:1097)
at org.jboss.web.tomcat.service.session.JBossCacheManager.storeSession(JBossCacheManager.java:652)
at org.jboss.web.tomcat.service.session.InstantSnapshotManager.snapshot(InstantSnapshotManager.java:49)
at org.jboss.web.tomcat.service.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:98)
at org.jboss.web.tomcat.service.session.JvmRouteValve.invoke(JvmRouteValve.java:84)
at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:241)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:437)
at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:381)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:595)
Caused by: org.jboss.cache.lock.TimeoutException: failure acquiring lock: fqn=/JSESSION/localhost/et/BvlehVnGPW2uuH09kn+Prw**, caller=Thread[ajp-192.168.129.36-8009-15,5,jboss], lock=write owner=Thread[IncomingPacketHandler (channel=Tomcat-ETProd),5,JGroups threads] (activeReaders=0, activeWriter=Thread[IncomingPacketHandler (channel=Tomcat-ETProd),5,JGroups threads], waitingReaders=4, waitingWriters=0, waitingUpgrader=0)
at org.jboss.cache.Node.acquire(Node.java:500)
at org.jboss.cache.interceptors.PessimisticLockInterceptor.acquireNodeLock(PessimisticLockInterceptor.java:379)
at org.jboss.cache.interceptors.PessimisticLockInterceptor.lock(PessimisticLockInterceptor.java:307)
at org.jboss.cache.interceptors.PessimisticLockInterceptor.invoke(PessimisticLockInterceptor.java:175)
at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
at org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
at org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:39)
at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:167)
at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
at org.jboss.cache.TreeCache.put(TreeCache.java:3759)
at sun.reflect.GeneratedMethodAccessor155.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
at $Proxy90.put(Unknown Source)
at org.jboss.web.tomcat.service.session.JBossCacheWrapper.put(JBossCacheWrapper.java:141)
... 17 more
Caused by: org.jboss.cache.lock.TimeoutException: write lock for /JSESSION/localhost/et/BvlehVnGPW2uuH09kn+Prw** could not be acquired after 15000 ms. Locks: Read lock owners: []
Write lock owner: Thread[IncomingPacketHandler (channel=Tomcat-ETProd),5,JGroups threads]
(caller=Thread[ajp-192.168.129.36-8009-15,5,jboss], lock info: write owner=Thread[IncomingPacketHandler (channel=Tomcat-ETProd),5,JGroups threads] (activeReaders=0, activeWriter=Thread[IncomingPacketHandler (channel=Tomcat-ETProd),5,JGroups threads], waitingReaders=4, waitingWriters=0, waitingUpgrader=0))
at org.jboss.cache.lock.IdentityLock.acquireWriteLock(IdentityLock.java:206)
at org.jboss.cache.Node.acquireWriteLock(Node.java:529)
at org.jboss.cache.Node.acquire(Node.java:476)
... 42 more
Also hereby attached the ejb3-clustered-sfsbcache-service.xml for configuration reference.
Appreciate any help.