[jboss-jira] [JBoss JIRA] (JBMESSAGING-1947) Deadlock may happen when JBM shuts down a JGroups channel

[Yong Hao Gao (JIRA)]

Yong Hao Gao created JBMESSAGING-1947:
-----------------------------------------

             Summary: Deadlock may happen when JBM shuts down a JGroups channel
                 Key: JBMESSAGING-1947
                 URL: https://issues.jboss.org/browse/JBMESSAGING-1947
             Project: JBoss Messaging
          Issue Type: Feature Request
          Components: JMS Clustering
    Affects Versions: 1.4.8.SP9, 1.4.0.SP3.CP14
            Reporter: Yong Hao Gao
            Assignee: Yong Hao Gao
             Fix For: 1.4.0.SP3.CP15, 1.4.8.SP10


from Dennis:

On 2013-09-16 20:37:51, Reed, Dennis commented:
"The latest issue is a deadlock caused by JBoss Messaging.

JBoss Messaging is closing the JGroups channel.

This thread is waiting for a DISCONNECT_OK message from JGroups.
This thread is holding a lock on a MessagingPostOffice$StateMonitor.

  "Thread-29" prio=10 tid=0x00007f2e70f90000 nid=0xdc3 in Object.wait() [0x00007f2e590ce000]
       ...
       at org.jgroups.util.Promise.getResult(Promise.java:77)
       at org.jgroups.JChannel.disconnect(JChannel.java:473)
       ...
       at org.jgroups.JChannel.close(JChannel.java:498)
       at org.jboss.messaging.core.impl.postoffice.GroupMember.stop(GroupMember.java:250)
       ...
       at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice$StateMonitor.run(MessagingPostOffice.java:4284)
       - locked <0x00000000e2f1d3e8> (a org.jboss.messaging.core.impl.postoffice.MessagingPostOffice$StateMonitor)

Another thread is blocking while processing a new view.  (This is BAD and should never be done!!!).

This thread is waiting for the MessagingPostOffice$StateMonitor locked in the thread above.
This thread is holding a lock on the view object in JGroups.

   "IncomingMessageHandler (channel=MessagingPostOffice-CTRL)" daemon prio=10 tid=0x00007f2e70f66800 nid=0xdad waiting for monitor entry [0x00007f2e596d3000]
      java.lang.Thread.State: BLOCKED (on object monitor)
       at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice$StateMonitor.stopJBMNodeForRecovery(MessagingPostOffice.java:4264)
       - waiting to lock <0x00000000e2f1d3e8> (a org.jboss.messaging.core.impl.postoffice.MessagingPostOffice$StateMonitor)
       at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.quarantine(MessagingPostOffice.java:1765)
       at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.nodesLeft(MessagingPostOffice.java:1585)
       at org.jboss.messaging.core.impl.postoffice.GroupMember$ControlMembershipListener.viewAccepted(GroupMember.java:609)
       at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.passUp(MessageDispatcher.java:737)
       at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:372)
       at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:779)
       at org.jgroups.JChannel.up(JChannel.java:1090)
       ...
       at org.jgroups.protocols.pbcast.GMS.installView(GMS.java:508)
       - locked <0x00000000e290d740> (a org.jgroups.Membership)
       at org.jgroups.protocols.pbcast.GMS.installView(GMS.java:422)
       ...

The thread that is trying to send the DISCONNECT_OK is blocked waiting on the view object locked by the thread above:

   "ViewHandler" prio=10 tid=0x00007f2e7800e800 nid=0xe93 waiting for monitor entry [0x00007f2e58ecd000]
      java.lang.Thread.State: BLOCKED (on object monitor)
       at org.jgroups.protocols.pbcast.GMS.getNextView(GMS.java:283)
       - waiting to lock <0x00000000e290d740> (a org.jgroups.Membership)
       at org.jgroups.protocols.pbcast.CoordGmsImpl.handleMembershipChange(CoordGmsImpl.java:480)
       at org.jgroups.protocols.pbcast.GMS$ViewHandler.process(GMS.java:1418)
       at org.jgroups.protocols.pbcast.GMS$ViewHandler.run(GMS.java:1355)
       at java.lang.Thread.run(Thread.java:662)"



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira