Deadlock when a node is being shutdown by the MessagingClusterHealthMBean if the DB and
Network is broken at the same time
--------------------------------------------------------------------------------------------------------------------------
Key: JBMESSAGING-1901
URL:
https://issues.jboss.org/browse/JBMESSAGING-1901
Project: JBoss Messaging
Issue Type: Bug
Components: JMS Clustering
Affects Versions: 1.4.8.SP3, 1.4.0.SP3.CP14
Reporter: Yong Hao Gao
Assignee: Yong Hao Gao
Fix For: 1.4.0.SP3.CP15, 1.4.8.SP4
The stack trace shows the deadlock:
"Thread-23":
waiting to lock monitor 0x0000000001681638 (object 0x00000000d730fa60, a
org.jboss.jms.server.MessagingClusterHealthMBean),
which is held by "Incoming-13,192.168.1.110:55200"
"Incoming-13,192.168.1.110:55200":
waiting to lock monitor 0x0000000001681c20 (object 0x00000000d92e1580, a
org.jboss.messaging.core.impl.postoffice.MessagingPostOffice$StateMonitor),
which is held by "Thread-23"
Java stack information for the threads listed above:
===================================================
"Thread-23":
at
org.jboss.jms.server.MessagingClusterHealthMBean.stopNodeOnDBFailure(MessagingClusterHealthMBean.java:94)
- waiting to lock <0x00000000d730fa60> (a
org.jboss.jms.server.MessagingClusterHealthMBean)
at org.jboss.jms.server.ServerPeer.stopJBMNodeForRecovery(ServerPeer.java:2115)
at
org.jboss.messaging.core.impl.postoffice.MessagingPostOffice$StateMonitor.run(MessagingPostOffice.java:4256)
- locked <0x00000000d92e1580> (a
org.jboss.messaging.core.impl.postoffice.MessagingPostOffice$StateMonitor)
"Incoming-13,192.168.1.110:55200":
at
org.jboss.messaging.core.impl.postoffice.MessagingPostOffice$StateMonitor.shutdown(MessagingPostOffice.java:4280)
- waiting to lock <0x00000000d92e1580> (a
org.jboss.messaging.core.impl.postoffice.MessagingPostOffice$StateMonitor)
at
org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.stop(MessagingPostOffice.java:797)
at
org.jboss.messaging.core.jmx.MessagingPostOfficeService.stopService(MessagingPostOfficeService.java:554)
at org.jboss.system.ServiceMBeanSupport.jbossInternalStop(ServiceMBeanSupport.java:405)
at org.jboss.system.ServiceMBeanSupport.stop(ServiceMBeanSupport.java:281)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)
at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:138)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)
at
org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:140)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
at
org.jboss.jms.server.MessagingClusterHealthMBean.stopService(MessagingClusterHealthMBean.java:140)
at
org.jboss.jms.server.MessagingClusterHealthMBean.stopNodeOnDBFailure(MessagingClusterHealthMBean.java:107)
- locked <0x00000000d730fa60> (a org.jboss.jms.server.MessagingClusterHealthMBean)
at org.jboss.jms.server.ServerPeer.stopJBMNodeForRecovery(ServerPeer.java:2115)
at
org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.quarantine(MessagingPostOffice.java:1751)
at
org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.nodesLeft(MessagingPostOffice.java:1571)
at
org.jboss.messaging.core.impl.postoffice.GroupMember$ControlMembershipListener.viewAccepted(GroupMember.java:609)
at
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:733)
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:776)
at org.jgroups.JChannel.up(JChannel.java:1336)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:454)
at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:439)
at org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:153)
at org.jgroups.protocols.FRAG2.up(FRAG2.java:188)
at org.jgroups.protocols.FC.up(FC.java:493)
at org.jgroups.protocols.pbcast.GMS.installView(GMS.java:543)
- locked <0x00000000d92a8ae0> (a org.jgroups.Membership)
at org.jgroups.protocols.pbcast.CoordGmsImpl.handleViewChange(CoordGmsImpl.java:463)
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:749)
at org.jgroups.protocols.VIEW_SYNC.up(VIEW_SYNC.java:192)
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:233)
at org.jgroups.protocols.UNICAST.up(UNICAST.java:328)
at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:895)
at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:708)
at org.jgroups.protocols.BARRIER.up(BARRIER.java:136)
at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:167)
at org.jgroups.protocols.FD.up(FD.java:284)
at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:328)
at org.jgroups.protocols.MERGE2.up(MERGE2.java:144)
at org.jgroups.protocols.Discovery.up(Discovery.java:264)
at org.jgroups.protocols.PING.up(PING.java:273)
at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2319)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1253)
at org.jgroups.protocols.TP.access$100(TP.java:50)
at org.jgroups.protocols.TP$1.run(TP.java:1177)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Found 1 deadlock.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see:
http://www.atlassian.com/software/jira