Hi Mathias
The issue reappeared again this morning. Later investigations led me to the conclusion that the class which does the blocking seems to be the MessageSucker, when it tries to move messages between the nodes.
I have the following thread dump:
"Thread-28596" daemon prio=10 tid=0x00007f96e3736800 nid=0x3b1c in Object.wait() [0x0000000050c28000..0x0000000050c28ba0]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.jboss.messaging.util.Future.getResult(Future.java:51)
- locked <0x00007f982519ef50> (a org.jboss.messaging.util.Future)
at org.jboss.jms.client.container.ClientConsumer.waitForOnMessageToComplete(ClientConsumer.java:779)
at org.jboss.jms.client.container.ClientConsumer.close(ClientConsumer.java:442)
at org.jboss.jms.client.container.ConsumerAspect.handleClosing(ConsumerAspect.java:150)
at org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect55.invoke(ConsumerAspect55.java)
at org.jboss.jms.client.delegate.ClientConsumerDelegate$closing_2473194355759371067.invokeNext(ClientConsumerDelegate$closing_2473194355759371067.java)
at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
at org.jboss.jms.client.delegate.ClientConsumerDelegate$closing_2473194355759371067.invokeNext(ClientConsumerDelegate$closing_2473194355759371067.java)
at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
at org.jboss.jms.client.delegate.ClientConsumerDelegate$closing_2473194355759371067.invokeNext(ClientConsumerDelegate$closing_2473194355759371067.java)
at org.jboss.jms.client.delegate.ClientConsumerDelegate.closing(ClientConsumerDelegate.java)
at org.jboss.messaging.core.impl.clusterconnection.MessageSucker.suspend(MessageSucker.java:217)
- locked <0x00007f972efdd2a8> (a org.jboss.messaging.core.impl.clusterconnection.MessageSucker)
at org.jboss.messaging.core.impl.clusterconnection.ClusterConnectionManager$ConnectionInfo.cleanupConnection(ClusterConnectionManager.java:804)
- locked <0x00007f972efdd988> (a org.jboss.messaging.core.impl.clusterconnection.ClusterConnectionManager$ConnectionInfo)
at org.jboss.messaging.core.impl.clusterconnection.ClusterConnectionManager$ConnectionInfo.onException(ClusterConnectionManager.java:785)
at org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:120)
at org.jboss.remoting.ConnectionValidator$1.run(ConnectionValidator.java:498)
which it correlates to the following warn inside server.log:
2012-03-27 00:47:24,258 WARN [org.jboss.messaging.core.impl.clusterconnection.ClusterConnectionManager] Connection failure detected. Clean up and retry connection. maxRetry: -1 retryInterval: 5000
It seems that the MessageSucker had problems to release the locks after a connection failure between nodes.... and from here a whole cascade of events triggered, because all the other connections that tried to access the targeted queue (which is clustered) are waiting for the instance of it to be released by MessageSucker.
Daniel