[jboss-user] [Performance Tuning] - Our Active Thread Count is over 20 000. Surely this is a pro

[ksaloojee]

We are doing performance testing in our UAT environment and our active thread count keeps on rising and has hit over 20 000. It never comes down, even when we stop putting load on the server. This appears to be too high in my opinion.

We are migrating an application from Weblogic to JBoss. The application uses EJB 2.x session beans, entity beans, DAO's and JMS. The client is a Swing UI (i.e. fat client) that communicates with the server via RMI and JMS.

I have dumped the threads on the server and what I have noticed is that the majority of them are:

"UIL2.SocketManager.WriteTask#385 client=10.198.8.23:8093" daemon prio=10 tid=0x00002aaecc161400 nid=0x3054 in Object.wait() [0x000000007a143000..0x000000007a143d90]
  |    java.lang.Thread.State: WAITING (on object monitor)
  | 	at java.lang.Object.wait(Native Method)
  | 	- waiting on <0x00002aae94e14fc0> (a java.lang.Object)
  | 	at java.lang.Object.wait(Object.java:485)
  | 	at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
  | 	- locked <0x00002aae94e14fc0> (a java.lang.Object)
  | 	at org.jboss.mq.il.uil2.SocketManager$WriteTask.run(SocketManager.java:518)
  | 	at java.lang.Thread.run(Thread.java:619)

That tells me that something JMS related is hogging threads. In addition to that the logs also display a lot of JMS warnings:

2009-04-30 15:15:36,102 WARN  [org.jboss.mq.Connection] Connection failure, use javax.jms.Connection.setExceptionListener() to handle this error and reconnect
  | org.jboss.mq.SpyJMSException: No pong received; - nested throwable: (java.io.IOException: ping timeout.)
  | 	at org.jboss.mq.Connection$PingTask.run(Connection.java:1313)
  | 	at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
  | 	at java.lang.Thread.run(Thread.java:619)
  | Caused by: java.io.IOException: ping timeout.

That I know how to fix thanks to http://www.jboss.org/community/docs/DOC-10000 and is also JMS related.

Is our thread count too high? Has anyone experienced anything similar with JMS where it hogs threads? Do you think that JMS is the problem or is it a red herring that is taking away attention from something else?

TIA

View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4227865#4227865

Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4227865