Re: [jboss-user] [Performance Tuning] - A question about JVM activ thread count

Richard Boesch [http://community.jboss.org/people/vb_rich] replied to the discussion "A question about JVM activ thread count" To view the discussion, visit: http://community.jboss.org/message/552023#552023 -------------------------------------------------------------- I have a similar issue, but more details to provide. I’m hoping someone may have an idea to help me triage. Looking for any/all ideas on how to research this. *Environment:* * AIX ?? * JBoss 3.2.7 (yes, very old) * JRE 1.3.1 (yes, very old) * J2EE application (Swing Applet, JSP, EJB, Oracle) *Thread Monitoring Solutions:* * Hyperic agent * Twiddle (JBoss command line tool to access JMX) *Historic:* * For the past 7 years our application has hovered around 150-200 active threads. * No new code has been released recently *Issue That Started Last Month:* * Operations person “accidently restarted jboss” in the middle of the day. They said they misread an SOP and made a mistake. No other information can be found regarding what else may have happened. I believe something was deleted/configured/moved/etc from past experience with our support group. * Per monitoring, thread counts jumped up dramatically the moment JBoss was restarted and we have had a recurring problem since. We are forced to restart our system every 4-5 days. * Since the incident, as soon as we restart jboss, threads start rising at about 17k threads a day (never going downward). * It has reached up to 90,000 threads * Our JVM begins to produce heap dumps as thread counts rise past 60,000 * The heap dumps become so large/frequent that is causes our system to crash *Research:* * Heap dump analyzers show no memory/thread issues in the JVM, but our java version is so old that we are not sure how accurate the analyzer is. * Per Hyperic/Twiddle, thread counts seem to rise in coorelation with applications coming into the system. When we put up a maintenance page, thread counts stop rising (do not go downward either) * Twiddle allows us to invoke “listThreadDump” on the “jboss.system:type=ServerInfo” mbean, which shows the thread count of all thread groups, but it does not break the individual thread groups down to see how many threads are in each group (so we can’t narrow it down) * JBoss3.2.7 is so old that listThreadDump (on ServerInfo) is the only function we can execute for analysis at the JBoss level. * Our QA system has received a heap dump, but all load testing we perform does not raise the thread counts past 150 in that environment. Also, the heap dump can’t be recreated consistently. Our system is “infinitely configurable” so we may not be hitting the correct scenario as many production clients may be. *Next Steps:* * Our next step is to failover to a secondary application server to rule out a hardware failure on the prod app server. We doubt this will help, but it will eliminate a variable. Any suggestions for triage are welcome (other than the obvious advice of upgrading our versions). Ideas anyone???? -------------------------------------------------------------- Reply to this message by going to Community [http://community.jboss.org/message/552023#552023] Start a new discussion in Performance Tuning at Community [http://community.jboss.org/choose-container!input.jspa?contentType=1&...]