Deadlock may happen when JBM shuts down a JGroups channel --------------------------------------------------------- Key: JBMESSAGING-1947 URL: https://issues.jboss.org/browse/JBMESSAGING-1947 Project: JBoss Messaging Issue Type: Feature Request Components: JMS Clustering Affects Versions: 1.4.0.SP3.CP14, 1.4.8.SP9 Reporter: Yong Hao Gao Assignee: Yong Hao Gao Fix For: 1.4.0.SP3.CP15, 1.4.8.SP10 from Dennis: On 2013-09-16 20:37:51, Reed, Dennis commented: "The latest issue is a deadlock caused by JBoss Messaging. JBoss Messaging is closing the JGroups channel. This thread is waiting for a DISCONNECT_OK message from JGroups. This thread is holding a lock on a MessagingPostOffice$StateMonitor. "Thread-29" prio=10 tid=0x00007f2e70f90000 nid=0xdc3 in Object.wait() [0x00007f2e590ce000] ... at org.jgroups.util.Promise.getResult(Promise.java:77) at org.jgroups.JChannel.disconnect(JChannel.java:473) ... at org.jgroups.JChannel.close(JChannel.java:498) at org.jboss.messaging.core.impl.postoffice.GroupMember.stop(GroupMember.java:250) ... at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice$StateMonitor.run(MessagingPostOffice.java:4284) - locked <0x00000000e2f1d3e8> (a org.jboss.messaging.core.impl.postoffice.MessagingPostOffice$StateMonitor) Another thread is blocking while processing a new view. (This is BAD and should never be done!!!). This thread is waiting for the MessagingPostOffice$StateMonitor locked in the thread above. This thread is holding a lock on the view object in JGroups. "IncomingMessageHandler (channel=MessagingPostOffice-CTRL)" daemon prio=10 tid=0x00007f2e70f66800 nid=0xdad waiting for monitor entry [0x00007f2e596d3000] java.lang.Thread.State: BLOCKED (on object monitor) at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice$StateMonitor.stopJBMNodeForRecovery(MessagingPostOffice.java:4264) - waiting to lock <0x00000000e2f1d3e8> (a org.jboss.messaging.core.impl.postoffice.MessagingPostOffice$StateMonitor) at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.quarantine(MessagingPostOffice.java:1765) at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.nodesLeft(MessagingPostOffice.java:1585) at org.jboss.messaging.core.impl.postoffice.GroupMember$ControlMembershipListener.viewAccepted(GroupMember.java:609) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.passUp(MessageDispatcher.java:737) at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:372) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:779) at org.jgroups.JChannel.up(JChannel.java:1090) ... at org.jgroups.protocols.pbcast.GMS.installView(GMS.java:508) - locked <0x00000000e290d740> (a org.jgroups.Membership) at org.jgroups.protocols.pbcast.GMS.installView(GMS.java:422) ... The thread that is trying to send the DISCONNECT_OK is blocked waiting on the view object locked by the thread above: "ViewHandler" prio=10 tid=0x00007f2e7800e800 nid=0xe93 waiting for monitor entry [0x00007f2e58ecd000] java.lang.Thread.State: BLOCKED (on object monitor) at org.jgroups.protocols.pbcast.GMS.getNextView(GMS.java:283) - waiting to lock <0x00000000e290d740> (a org.jgroups.Membership) at org.jgroups.protocols.pbcast.CoordGmsImpl.handleMembershipChange(CoordGmsImpl.java:480) at org.jgroups.protocols.pbcast.GMS$ViewHandler.process(GMS.java:1418) at org.jgroups.protocols.pbcast.GMS$ViewHandler.run(GMS.java:1355) at java.lang.Thread.run(Thread.java:662)"