[jboss-user] [JBoss Messaging] - Threads not being cleaned up when clustered

[chip_schoch]

Clustered App servers.  JBossAS 4.2.2.GA, JBM 1.4.0SP3
 --------------
  | |  Appserver1  |  <====== ConvServer1
  | |  Appserver2  |  <====== ConvServer2
  |  --------------
  |  
 My configuration is a shown above.  I have 2 linux JBoss servers clustered and two windows jboss servers not clustered that act as clients to the clustered queues on the app servers.  The conv servers connect using the ClusteredXAConnection factory.  When I run a bunch of messages through in this configuration the thread count on both appservers continually increases until eventually I run out of memory.
 
 When I shut down one of the app servers and perform the same test the thread count on the other app server decreases initially then remains steady .  It does not increase when I run a bunch of messages through it.
  --------------
  | |              |  <====== ConvServer1
  | |  Appserver2  |  <====== ConvServer2
  |  --------------
  |  	                                   DevApp1 ThreadCount	DevApp2 ThreadCount
  | Cluster with queuing started on devapp2		
  | Start	                                   158	                200
  | After 4 usign packages	                   167	                206
  | After 4 usign packages	                   173	                208
  | After 4 usign packages	                   179	                208
  | After hundred events	                   177	                212
  | After 10 WPS packages	                   178	                212
  | After 10 WPS packages	                   180	                212
  | 
  | DevApp2 Only		
  | 
  | Before shutting down Devapp1		                       212
  | After shutting down DevApp1		                       187
  | After 4 usign packages	 	                               187
  | After 6 usign packages		                               187
  | 
  | Restarted with each conv server 
  | connecting to different appserver (no message sucking)
  | 
  | Start	                                 155	                225
  | After 4 usign packages	                 169	                228
  | After 4 usign packages	                 174	                232
  | After 3 packages one at a time	         175	                235*
  | After 20 uSign packages	                 187	                256
  | 
  | 
 
Without belaboring the point, in our system a package causes a series of
messages to be posted to various queues as it moves through the processing chain.

The packages called usign require that a conversion be performed by a service
running on the windows (conv) servers.  These are the ones that cause the increase
in the thread count.  I am convinced that it is not my code that is leaving the threads
around because it only happens when the cluster has more than one server running.

* I observed that appserver one processed the message, but the thread count increased
  on appserver2 anyway.
  
  
  Here are a couple of the stranded threads from the jmx-console view.
  
  Thread: Thread-1562 : priority:5, demon:true, threadId:4470, threadState:WAITING, lockName:java.lang.Object at 11bad13
  | 
  |     java.lang.Object.wait(Native Method)
  |     java.lang.Object.wait(Object.java:474)
  |     EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
  |     EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
  |     java.lang.Thread.run(Thread.java:595)
  | 
  | Thread: Thread-1568 : priority:5, demon:false, threadId:4478, threadState:WAITING, lockName:java.lang.Object at 1ffed3a
  | 
  |     java.lang.Object.wait(Native Method)
  |     java.lang.Object.wait(Object.java:474)
  |     EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
  |     EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
  |     java.lang.Thread.run(Thread.java:595)
  | 
  | Thread: Thread-1569 : priority:5, demon:true, threadId:4479, threadState:WAITING, lockName:java.lang.Object at 12f926d
  | 
  |     java.lang.Object.wait(Native Method)
  |     java.lang.Object.wait(Object.java:474)
  |     EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
  |     EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
  |     java.lang.Thread.run(Thread.java:595)
  | 
  | 
  | 
  | 

Has anyone seen anything like this before?

View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4125664#4125664

Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4125664