Deadlock when a node is being shutdown by the MessagingClusterHealthMBean if the DB and Network is broken at the same time -------------------------------------------------------------------------------------------------------------------------- Key: JBMESSAGING-1901 URL: https://issues.jboss.org/browse/JBMESSAGING-1901 Project: JBoss Messaging Issue Type: Bug Components: JMS Clustering Affects Versions: 1.4.8.SP3, 1.4.0.SP3.CP14 Reporter: Yong Hao Gao Assignee: Yong Hao Gao Fix For: 1.4.0.SP3.CP15, 1.4.8.SP4 The stack trace shows the deadlock: "Thread-23": waiting to lock monitor 0x0000000001681638 (object 0x00000000d730fa60, a org.jboss.jms.server.MessagingClusterHealthMBean), which is held by "Incoming-13,192.168.1.110:55200" "Incoming-13,192.168.1.110:55200": waiting to lock monitor 0x0000000001681c20 (object 0x00000000d92e1580, a org.jboss.messaging.core.impl.postoffice.MessagingPostOffice$StateMonitor), which is held by "Thread-23" Java stack information for the threads listed above: =================================================== "Thread-23": at org.jboss.jms.server.MessagingClusterHealthMBean.stopNodeOnDBFailure(MessagingClusterHealthMBean.java:94) - waiting to lock <0x00000000d730fa60> (a org.jboss.jms.server.MessagingClusterHealthMBean) at org.jboss.jms.server.ServerPeer.stopJBMNodeForRecovery(ServerPeer.java:2115) at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice$StateMonitor.run(MessagingPostOffice.java:4256) - locked <0x00000000d92e1580> (a org.jboss.messaging.core.impl.postoffice.MessagingPostOffice$StateMonitor) "Incoming-13,192.168.1.110:55200": at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice$StateMonitor.shutdown(MessagingPostOffice.java:4280) - waiting to lock <0x00000000d92e1580> (a org.jboss.messaging.core.impl.postoffice.MessagingPostOffice$StateMonitor) at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.stop(MessagingPostOffice.java:797) at org.jboss.messaging.core.jmx.MessagingPostOfficeService.stopService(MessagingPostOfficeService.java:554) at org.jboss.system.ServiceMBeanSupport.jbossInternalStop(ServiceMBeanSupport.java:405) at org.jboss.system.ServiceMBeanSupport.stop(ServiceMBeanSupport.java:281) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96) at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:138) at org.jboss.mx.server.Invocation.invoke(Invocation.java:90) at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:140) at org.jboss.mx.server.Invocation.invoke(Invocation.java:90) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668) at org.jboss.jms.server.MessagingClusterHealthMBean.stopService(MessagingClusterHealthMBean.java:140) at org.jboss.jms.server.MessagingClusterHealthMBean.stopNodeOnDBFailure(MessagingClusterHealthMBean.java:107) - locked <0x00000000d730fa60> (a org.jboss.jms.server.MessagingClusterHealthMBean) at org.jboss.jms.server.ServerPeer.stopJBMNodeForRecovery(ServerPeer.java:2115) at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.quarantine(MessagingPostOffice.java:1751) at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.nodesLeft(MessagingPostOffice.java:1571) at org.jboss.messaging.core.impl.postoffice.GroupMember$ControlMembershipListener.viewAccepted(GroupMember.java:609) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:733) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:776) at org.jgroups.JChannel.up(JChannel.java:1336) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:454) at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:439) at org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:153) at org.jgroups.protocols.FRAG2.up(FRAG2.java:188) at org.jgroups.protocols.FC.up(FC.java:493) at org.jgroups.protocols.pbcast.GMS.installView(GMS.java:543) - locked <0x00000000d92a8ae0> (a org.jgroups.Membership) at org.jgroups.protocols.pbcast.CoordGmsImpl.handleViewChange(CoordGmsImpl.java:463) at org.jgroups.protocols.pbcast.GMS.up(GMS.java:749) at org.jgroups.protocols.VIEW_SYNC.up(VIEW_SYNC.java:192) at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:233) at org.jgroups.protocols.UNICAST.up(UNICAST.java:328) at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:895) at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:708) at org.jgroups.protocols.BARRIER.up(BARRIER.java:136) at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:167) at org.jgroups.protocols.FD.up(FD.java:284) at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:328) at org.jgroups.protocols.MERGE2.up(MERGE2.java:144) at org.jgroups.protocols.Discovery.up(Discovery.java:264) at org.jgroups.protocols.PING.up(PING.java:273) at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2319) at org.jgroups.protocols.TP.passMessageUp(TP.java:1253) at org.jgroups.protocols.TP.access$100(TP.java:50) at org.jgroups.protocols.TP$1.run(TP.java:1177) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Found 1 deadlock. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira