[jboss-user] [JBoss Messaging] - Deadlocks with ClusterRoundRobinDistributor

[travisb]

During volume testing with 2 nodes I now see these deadlocks in the thread dumps (platform AS 4.2.0.GA_CP01, JBM snapshot_stable 12-Nov-2007):

Node 1:

Found one Java-level deadlock:
  | =============================
  | "Messaging Timeout(3)-266":
  |   waiting to lock monitor 0x7548e274 (object 0x83f80fa0, a java.lang.Object),
  |   which is held by "Thread-114"
  | "Thread-114":
  |   waiting to lock monitor 0x7a7dc6e4 (object 0x83f80f70, a org.jboss.messaging.core.impl.ClusterRoundRobinDistributor),
  |   which is held by "Messaging Timeout(3)-266"
  | 
  | Java stack information for the threads listed above:
  | ===================================================
  | "Messaging Timeout(3)-266":
  |         at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.getNumberOfReceivers(MessagingQueue.java:621)
  |         - waiting to lock <0x83f80fa0> (a java.lang.Object)
  |         at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.getNumberOfReceivers(ClusterRoundRobinDistributor.java:129)
  |         - locked <0x83f80f70> (a org.jboss.messaging.core.impl.ClusterRoundRobinDistributor)
  |         at org.jboss.messaging.core.impl.ChannelSupport$DeliverRefTimeoutTarget.timedOut(ChannelSupport.java:956)
  |         at org.jboss.util.timeout.TimeoutFactory$TimeoutWorker.run(TimeoutFactory.java:265)
  |         at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)
  |         at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743)
  |         at java.lang.Thread.run(Thread.java:595)
  | "Thread-114":
  |         at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.getNumberOfReceivers(ClusterRoundRobinDistributor.java:129)
  |         - waiting to lock <0x83f80f70> (a org.jboss.messaging.core.impl.ClusterRoundRobinDistributor)
  |         at org.jboss.messaging.core.impl.ChannelSupport.deliver(ChannelSupport.java:222)
  |         - locked <0x83f80fa0> (a java.lang.Object)
  |         at org.jboss.jms.server.endpoint.ServerSessionEndpoint$2.run(ServerSessionEndpoint.java:1598)
  |         at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:89)
  |         at java.lang.Thread.run(Thread.java:595)
  | 
  | Found one Java-level deadlock:
  | =============================
  | "Thread-424":
  |   waiting to lock monitor 0x7548e2f4 (object 0x8409cda0, a org.jboss.messaging.core.impl.ClusterRoundRobinDistributor),
  |   which is held by "Messaging Timeout(3)-281"
  | "Messaging Timeout(3)-281":
  |   waiting to lock monitor 0x7548e874 (object 0x8409cdd0, a java.lang.Object),
  |   which is held by "Thread-424"
  | 
  | Java stack information for the threads listed above:
  | ===================================================
  | "Thread-424":
  |         at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.getNumberOfReceivers(ClusterRoundRobinDistributor.java:129)
  |         - waiting to lock <0x8409cda0> (a org.jboss.messaging.core.impl.ClusterRoundRobinDistributor)
  |         at org.jboss.messaging.core.impl.ChannelSupport.deliver(ChannelSupport.java:222)
  |         - locked <0x8409cdd0> (a java.lang.Object)
  |         at org.jboss.jms.server.endpoint.ServerSessionEndpoint$2.run(ServerSessionEndpoint.java:1598)
  |         at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:89)
  |         at java.lang.Thread.run(Thread.java:595)
  | "Messaging Timeout(3)-281":
  |         at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.getNumberOfReceivers(MessagingQueue.java:621)
  |         - waiting to lock <0x8409cdd0> (a java.lang.Object)
  |         at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.getNumberOfReceivers(ClusterRoundRobinDistributor.java:129)
  |         - locked <0x8409cda0> (a org.jboss.messaging.core.impl.ClusterRoundRobinDistributor)
  |         at org.jboss.messaging.core.impl.ChannelSupport$DeliverRefTimeoutTarget.timedOut(ChannelSupport.java:956)
  |         at org.jboss.util.timeout.TimeoutFactory$TimeoutWorker.run(TimeoutFactory.java:265)
  |         at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)
  |         at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743)
  |         at java.lang.Thread.run(Thread.java:595)
  | 
  | Found 2 deadlocks.
  | 

Node 2:
Found one Java-level deadlock:
  | =============================
  | "Thread-107":
  |   waiting to lock monitor 0x08355a8c (object 0x841ad4d8, a org.jboss.messaging.core.impl.ClusterRoundRobinDistributor),
  |   which is held by "Messaging Timeout(3)-13"
  | "Messaging Timeout(3)-13":
  |   waiting to lock monitor 0x087f6f54 (object 0x841ad508, a java.lang.Object),
  |   which is held by "Thread-107"
  | 
  | Java stack information for the threads listed above:
  | ===================================================
  | "Thread-107":
  |         at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.getNumberOfReceivers(ClusterRoundRobinDistributor.java:129)
  |         - waiting to lock <0x841ad4d8> (a org.jboss.messaging.core.impl.ClusterRoundRobinDistributor)
  |         at org.jboss.messaging.core.impl.ChannelSupport.deliver(ChannelSupport.java:222)
  |         - locked <0x841ad508> (a java.lang.Object)
  |         at org.jboss.jms.server.endpoint.ServerSessionEndpoint$2.run(ServerSessionEndpoint.java:1598)
  |         at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:89)
  |         at java.lang.Thread.run(Thread.java:595)
  | "Messaging Timeout(3)-13":
  |         at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.getNumberOfReceivers(MessagingQueue.java:621)
  |         - waiting to lock <0x841ad508> (a java.lang.Object)
  |         at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.getNumberOfReceivers(ClusterRoundRobinDistributor.java:129)
  |         - locked <0x841ad4d8> (a org.jboss.messaging.core.impl.ClusterRoundRobinDistributor)
  |         at org.jboss.messaging.core.impl.ChannelSupport$DeliverRefTimeoutTarget.timedOut(ChannelSupport.java:956)
  |         at org.jboss.util.timeout.TimeoutFactory$TimeoutWorker.run(TimeoutFactory.java:265)
  |         at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)
  |         at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743)
  |         at java.lang.Thread.run(Thread.java:595)
  | 
  | Found 1 deadlock.
  | 

My message processing comes to a halt, but I doubt it's fully related to these 6 threads.

View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4106604#4106604

Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4106604