<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
<body link="#355491" alink="#4262a1" vlink="#355491" style="background: #e2e2e2; margin: 0; padding: 20px;">
<div>
<table cellpadding="0" bgcolor="#FFFFFF" border="0" cellspacing="0" style="border: 1px solid #dadada; margin-bottom: 30px; width: 100%; -moz-border-radius: 6px; -webkit-border-radius: 6px;">
<tbody>
<tr>
<td>
<table border="0" cellpadding="0" cellspacing="0" bgcolor="#FFFFFF" style="border: solid 2px #ccc; background: #dadada; width: 100%; -moz-border-radius: 6px; -webkit-border-radius: 6px;">
<tbody>
<tr>
<td bgcolor="#000000" valign="middle" height="58px" style="border-bottom: 1px solid #ccc; padding: 20px; -moz-border-radius-topleft: 3px; -moz-border-radius-topright: 3px; -webkit-border-top-right-radius: 5px; -webkit-border-top-left-radius: 5px;">
<h1 style="color: #333333; font: bold 22px Arial, Helvetica, sans-serif; margin: 0; display: block !important;">
<!-- To have a header image/logo replace the name below with your img tag -->
<!-- Email clients will render the images when the message is read so any image -->
<!-- must be made available on a public server, so that all recipients can load the image. -->
<a href="http://community.jboss.org/index.jspa" style="text-decoration: none; color: #E1E1E1">JBoss Community</a></h1>
</td>
</tr>
<tr>
<td bgcolor="#FFFFFF" style="font: normal 12px Arial, Helvetica, sans-serif; color:#333333; padding: 20px; -moz-border-radius-bottomleft: 4px; -moz-border-radius-bottomright: 4px; -webkit-border-bottom-right-radius: 5px; -webkit-border-bottom-left-radius: 5px;"><h3 style="margin: 10px 0 5px; font-size: 17px; font-weight: normal;">
Jboss node hangs frequently
</h3>
<span style="margin-bottom: 10px;">
created by <a href="http://community.jboss.org/people/manohar.parelly">manohar Parelly</a> in <i>JBoss Cache</i> - <a href="http://community.jboss.org/message/572561#572561">View the full discussion</a>
</span>
<hr style="margin: 20px 0; border: none; background-color: #dadada; height: 1px;">
<div class="jive-rendered-content"><p>Dear All,</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>We have a problem in our production environment. One of our 4 jboss nodes is getting hanged frequently.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>We have 4 jboss nodes clustered in 2 physical servers. They are running perfectly till 4 months back for almost 2 years.</p><p>After starting up 4 nodes one after the other they are running fine till for some period of time (sometimes 10 days, sometimes 2 days).<br/>All of a sudden one of the jboss node is hanging and not receiving any request while other nodes are working fine. If we identify the hanging node immediately and restart the node everything works fine. Otherwise it is impacting other 3 nodes slowly and making them also to hang.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>In normal case no.of threads count is not exceeding 200 in any node. But when it hangs, it is showing more than 500 threads in Thread dump.<br/><strong>Thread dump I have attached. Please check.</strong></p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>We found below exception in our server.log file just before jboss node goes down. Please check below error:<br/>Please let me know if you need any more information. Please give your suggestions to resolve the issue.</p><p><br/>java.lang.RuntimeException: JBossCacheService: exception occurred in cache get after retry ... <br/>at org.jboss.web.tomcat.service.session.JBossCacheWrapper.get(JBossCacheWrapper.java:94)<br/>at org.jboss.web.tomcat.service.session.JBossCacheService.loadSession(JBossCacheService.java:251)<br/>at org.jboss.web.tomcat.service.session.JBossCacheManager.loadSession(JBossCacheManager.java:1010)<br/>at org.jboss.web.tomcat.service.session.JBossCacheManager.findSession(JBossCacheManager.java:782)<br/>at org.apache.catalina.connector.Request.doGetSession(Request.java:2283)<br/>at org.apache.catalina.connector.Request.getSession(Request.java:2075)<br/>at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:99)<br/>at org.jboss.web.tomcat.service.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:87)<br/>at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)<br/>at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)<br/>at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)<br/>at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)<br/>at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)<br/>at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)<br/>at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)<br/>at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)<br/>at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)<br/>at java.lang.Thread.run(Thread.java:595)<br/>Caused by: org.jboss.cache.lock.TimeoutException: failure acquiring lock: fqn=/JSESSION/localhost/atheebweb/MhPmOOs88X4+gQY0qxzIHA**, caller=GlobalTransaction:<172.16.64.21:7811>:678309, lock=write owner=Thread[IncomingMessageHandler (channel=Tomcat-AtheebWeb),5,JGroups threads] (activeReaders=0, activeWriter=Thread[IncomingMessageHandler (channel=Tomcat-AtheebWeb),5,JGroups threads], waitingReaders=0, waitingWriters=0, waitingUpgrader=0)<br/>at org.jboss.cache.Node.acquire(Node.java:500)<br/>at org.jboss.cache.interceptors.PessimisticLockInterceptor.acquireNodeLock(PessimisticLockInterceptor.java:381)<br/>at org.jboss.cache.interceptors.PessimisticLockInterceptor.lock(PessimisticLockInterceptor.java:309)<br/>at org.jboss.cache.interceptors.PessimisticLockInterceptor.invoke(PessimisticLockInterceptor.java:183)<br/>at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)<br/>at org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)<br/>at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)<br/>at org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:39)<br/>at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)<br/>at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)<br/>at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)<br/>at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)<br/>at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:138)<br/>at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5877)<br/>at org.jboss.cache.TreeCache.get(TreeCache.java:3641)<br/>at org.jboss.cache.TreeCache.get(TreeCache.java:3622)<br/>at org.jboss.cache.TreeCache.get(TreeCache.java:3418)<br/>at sun.reflect.GeneratedMethodAccessor143.invoke(Unknown Source)<br/>at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)<br/>at java.lang.reflect.Method.invoke(Method.java:585)<br/>at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)<br/>at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)<br/>at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)<br/>at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)<br/>at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)<br/>at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)<br/>at $Proxy443.get(Unknown Source)<br/>at org.jboss.web.tomcat.service.session.JBossCacheWrapper.get(JBossCacheWrapper.java:78)<br/>... 17 more<br/>Caused by: org.jboss.cache.lock.TimeoutException: read lock for /JSESSION/localhost/atheebweb/MhPmOOs88X4+gQY0qxzIHA** could not be acquired by GlobalTransaction:<172.16.64.21:7811>:678309 after 15000 ms. Locks: Read lock owners: []<br/>Write lock owner: Thread[IncomingMessageHandler (channel=Tomcat-AtheebWeb),5,JGroups threads]<br/>, lock info: write owner=Thread[IncomingMessageHandler (channel=Tomcat-AtheebWeb),5,JGroups threads] (activeReaders=0, activeWriter=Thread[IncomingMessageHandler (channel=Tomcat-AtheebWeb),5,JGroups threads], waitingReaders=0, waitingWriters=0, waitingUpgrader=0)<br/>at org.jboss.cache.lock.IdentityLock.acquireReadLock(IdentityLock.java:262)<br/>at org.jboss.cache.Node.acquireReadLock(Node.java:512)<br/>at org.jboss.cache.Node.acquire(Node.java:474)</p></div>
<div style="background-color: #f4f4f4; padding: 10px; margin-top: 20px;">
<p style="margin: 0;">Reply to this message by <a href="http://community.jboss.org/message/572561#572561">going to Community</a></p>
<p style="margin: 0;">Start a new discussion in JBoss Cache at <a href="http://community.jboss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2052">Community</a></p>
</div></td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</div>
</body>
</html>