[jboss-user] [JBoss Messaging] - Running inot a deadlock because of messageListener handling

[sameer saurabh]

sameer saurabh [https://community.jboss.org/people/nomind] created the discussion

"Running inot a deadlock because of messageListener handling"

To view the discussion, visit: https://community.jboss.org/message/773764#773764

--------------------------------------------------------------
The scenario for deadlock in JMS MessageListener while closing resources in onException() method of ExceptionListener:
 •     When an exception occurs in a MessageListener thread, it calls the asynchFailure() method of Session class which in turns calls the asynchFailure() method of Connection class. In the Connection class, a lock is taken on ‘elLock’ object and if there is no ExceptionListenerRunnable thread running, a ExceptionListenerRunnable thread is created and started. In the run() method of ExceptionListenerRunnable, a lock is again taken on ‘elLock’ object and the onException() method of ExceptionListener class is invoked.
 •     In onException() method of ExceptionListener class we try to recreate the JMS Connection. Before reconnecting we close all existing resources (Consumer, Session and Connection). In the close() API of SpyMessageConsumer class, the Message Consumer checks for any MessageListener thread and if there is any listener thread it calls the join() API of Thread class and wait for the listener thread to die.
 •     In the meanwhile if one more exception occurs in MessageListener thread, it again invokes asynchFailure() method of Connection class. If the earlier ExceptionListenerRunnable thread has not released the lock on the ‘elLock’ object, the MessageListener thread waits outside the synchronized block in asynchFailure() method of Connection class.
 This leads to a deadlock situation where ExceptionListener thread is waiting for the MessageListener thread to die and MessageListener thread is waiting for the ExceptionListener class to release lock on ‘elLock’ object.
 Though in asynchFailure() method of Connection class, a check is there to verify and log a warning message (“Connection failure, already in the exception listener”) if the ExceptionListenerRunnable thread is already running but this check is also inside the synchronized block. So even if ExceptionListenerRunnable thread is already running, the other thread will not be able to take the lock on ‘elLock’ object and the warning message will never be displayed. This seems to be a bug in JMS code as the check to verify an already running ExceptionListenerRunnable thread should be outside the synchronized block and if the ExceptionListenerRunnable thread is already running, the method should return from there.

Following is my thread dump - 
"ExceptionListener Connection at 206865307[token=ConnectionToken:ID:44/ee177c7bd4386f174c32bc563cb9b67a rcvstate=STARTED]"Id=331 in WAITING on lock=java.lang.Thread at 13f1442e
    at java.lang.Object.wait(Native Method)
    at java.lang.Thread.join(Thread.java:1258)
    at java.lang.Thread.join(Thread.java:1332)
    at org.jboss.mq.SpyMessageConsumer.close(SpyMessageConsumer.java:554)
    at com.tribalfusion.jms.JMSReceiver.closeResources(JMSReceiver.java:88)
    at com.tribalfusion.jms.JMSTopicReceiver.setJMSConfig(JMSTopicReceiver.java:32)
      - locked com.tribalfusion.jms.JMSTopicNonDurableSubscriber at 600877d4
    at com.tribalfusion.jms.JMSReceiver.reconfigureJMS(JMSReceiver.java:140)
    at com.tribalfusion.jms.JMSReceiver.reconnect(JMSReceiver.java:128)
    at com.tribalfusion.jms.JMSReceiver$2.onException(JMSReceiver.java:77)
    at org.jboss.mq.Connection$ExceptionListenerRunnable.run(Connection.java:1320)
      - locked java.lang.Object at 4bf3308d
    at java.lang.Thread.run(Thread.java:722)

| MessageListenerThread - dbSaveAckTopic"Id=45 in BLOCKED on lock=java.lang.Object at 4bf3308d |  owned by ExceptionListener Connection at 206865307[token=ConnectionToken:ID:44/ee177c7bd4386f174c32bc563cb9b67a rcvstate=STARTED] Id=331 |
|  | at org.jboss.mq.Connection.asynchFailure(Connection.java:424) |
|  | at org.jboss.mq.SpySession.asynchFailure(SpySession.java:1063) |
|  | at org.jboss.mq.SpyMessageConsumer.run(SpyMessageConsumer.java:731) |
|  | at java.lang.Thread.run(Thread.java:722) |



Full thread dump is attached.
--------------------------------------------------------------

Reply to this message by going to Community
[https://community.jboss.org/message/773764#773764]

Start a new discussion in JBoss Messaging at Community
[https://community.jboss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2042]

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/jboss-user/attachments/20121103/1199effd/attachment-0001.html