[jboss-jira] [JBoss JIRA] Assigned: (JBMESSAGING-1840) Deadlock on cluster failover

Deadlock on cluster failover ---------------------------- Key: JBMESSAGING-1840 URL: https://issues.jboss.org/browse/JBMESSAGING-1840 Project: JBoss Messaging Issue Type: Bug Components: JMS Clustering Affects Versions: 1.4.3.GA Environment: JBoss 5.1.0.GA in "all" configuration, PostOffice has FailoverOnNodeLeave = true, using MySQL Cluster 7.1.9a backend (ndb-persistence-service.xml) Reporter: Gerald Turner Assignee: Yong Hao Gao I have two JBoss 5.1.0.GA servers in a cluster, with a clustered queue named LogServ, and with an application which spawns several message consumers (not an MDB, each thread does it's own java:/ConnectionFactory lookup, and creates QueueSession, QueueReceiver, etc.). External to the JBoss VMs I have some code which is sending messages to the queue, this code uses HA-JNDI, looks up java:/ClusteredConnectionFactory, shares the QueueConnectionFactory between several threads, each thread has it's own QueueConnection/Session/Sender - taking advantage of the round-robin feature of ClusteredConnectionFactory (half the threads will be connected to one server, and the other half connected to the other server). The consumer application (in JBoss) has 32 threads on each JBoss server. The sender application (outside JBoss) has 64 threads, evenly split between the two JBoss servers. I have ensured that cluster failover is working properly by undeploying the consumer application, let the queue fill up with several tens of thousands of messages, then shutdown a server and watch the PostOffice take control of the downed servers Channel (several times back and forth). I have also witnessed failover working properly with the consumer application deployed, although with only a few hundred outstanding messages to process. Now while the consumer application is deployed, and the external sender application is running, when I shutdown a node, the running server enters a deadlock during failover. Java Thread Dump detailing the deadlock: Found one Java-level deadlock: ============================= "WorkerThread#35[208.176.159.100:27297]": waiting to lock monitor 0x00000000537fa8f8 (object 0x00002aaace27a010, a java.lang.Object), which is held by "WorkerThread#28[208.176.159.100:27288]" "WorkerThread#28[208.176.159.100:27288]": waiting to lock monitor 0x000000005340a9c0 (object 0x00002aaacd704bb0, a org.jboss.messaging.core.impl.clusterconnection.MessageSucker), which is held by "ReusableThread" "ReusableThread": waiting to lock monitor 0x00000000537fa8f8 (object 0x00002aaace27a010, a java.lang.Object), which is held by "WorkerThread#28[208.176.159.100:27288]" Java stack information for the threads listed above: =================================================== "WorkerThread#35[208.176.159.100:27297]": at org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:235) - waiting to lock <0x00002aaace27a010> (a java.lang.Object) at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2222) at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:509) at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:755) at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:399) at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87) at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeTarget(SessionAdvised$send_7280680627620114891.java) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111) at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157) at sun.reflect.GeneratedMethodAccessor320.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java) at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95) at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:891) at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:744) - locked <0x00002aaacbf53338> (a org.jboss.remoting.transport.socket.ServerThread) at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:697) at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:551) at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:232) "WorkerThread#28[208.176.159.100:27288]": at org.jboss.messaging.core.impl.clusterconnection.MessageSucker.setConsuming(MessageSucker.java:200) - waiting to lock <0x00002aaacd704bb0> (a org.jboss.messaging.core.impl.clusterconnection.MessageSucker) at org.jboss.messaging.core.impl.MessagingQueue.informSuckers(MessagingQueue.java:566) at org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:516) at org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:239) - locked <0x00002aaace27a010> (a java.lang.Object) at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2222) at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:509) at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:755) at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:399) at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87) at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeTarget(SessionAdvised$send_7280680627620114891.java) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111) at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157) at sun.reflect.GeneratedMethodAccessor320.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java) at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95) at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:891) at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:744) - locked <0x00002aaace4dc8e8> (a org.jboss.remoting.transport.socket.ServerThread) at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:697) at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:551) at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:232) "ReusableThread": at org.jboss.messaging.core.impl.MessagingQueue.unregisterSucker(MessagingQueue.java:387) - waiting to lock <0x00002aaace27a010> (a java.lang.Object) at org.jboss.messaging.core.impl.clusterconnection.MessageSucker.stop(MessageSucker.java:152) - locked <0x00002aaacd704bb0> (a org.jboss.messaging.core.impl.clusterconnection.MessageSucker) at org.jboss.messaging.core.impl.clusterconnection.ClusterConnectionManager$ConnectionInfo.closeAllSuckers(ClusterConnectionManager.java:688) - locked <0x00002aaacdf348f0> (a org.jboss.messaging.core.impl.clusterconnection.ClusterConnectionManager$ConnectionInfo) at org.jboss.messaging.core.impl.clusterconnection.ClusterConnectionManager$ConnectionInfo.close(ClusterConnectionManager.java:696) - locked <0x00002aaacdf348f0> (a org.jboss.messaging.core.impl.clusterconnection.ClusterConnectionManager$ConnectionInfo) at org.jboss.messaging.core.impl.clusterconnection.ClusterConnectionManager.notify(ClusterConnectionManager.java:279) - locked <0x00002aaaccb114b0> (a org.jboss.messaging.core.impl.clusterconnection.ClusterConnectionManager) at org.jboss.messaging.core.impl.DefaultClusterNotifier.sendNotification(DefaultClusterNotifier.java:72) at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.removeReplicantLocally(MessagingPostOffice.java:1223) at org.jboss.messaging.core.impl.postoffice.RemoveReplicantRequest.execute(RemoveReplicantRequest.java:57) at org.jboss.messaging.core.impl.postoffice.GroupMember$ControlRequestHandler.handle(GroupMember.java:622) at org.jgroups.blocks.MessageDispatcher.handle(MessageDispatcher.java:616) at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:637) at org.jgroups.blocks.RequestCorrelator$Request.run(RequestCorrelator.java:944) at org.jgroups.util.ReusableThread.run(ReusableThread.java:220) at java.lang.Thread.run(Thread.java:662) Found 1 deadlock. JBoss log of the server taking over the downed node: 2011-01-13 09:55:23,724 INFO [org.jboss.ha.framework.interfaces.HAPartition.lifecycle.DevelopmentMySQLPartition] (Incoming-9,207.88.61.168:33006:) New cluster view for partition DevelopmentMySQLPartition (id: 2, delta: -1) : [207.88.61.168:1099] 2011-01-13 09:55:23,724 INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.DevelopmentMySQLPartition] (AsynchViewChangeHandler Thread:) I am (207.88.61.168:1099) received membershipChanged event: 2011-01-13 09:55:23,724 INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.DevelopmentMySQLPartition] (AsynchViewChangeHandler Thread:) Dead members: 1 ([207.88.61.169:1099]) 2011-01-13 09:55:23,724 INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.DevelopmentMySQLPartition] (AsynchViewChangeHandler Thread:) New Members : 0 ([]) 2011-01-13 09:55:23,724 INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.DevelopmentMySQLPartition] (AsynchViewChangeHandler Thread:) All Members : 1 ([207.88.61.168:1099]) 2011-01-13 10:00:24,862 TRACE [org.jboss.messaging.core.impl.MessagingQueue] (WorkerThread#1[207.88.61.169:37674]:) Queue[1152414282/1/410-LogServ] removing receiver ConsumerEndpoint[pa-j1tmvuig-1-fctkvuig-85tfp2-o4p1g2r5] java.lang.Exception: trace at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.remove(MessagingQueue.java:623) at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.localClose(ServerConsumerEndpoint.java:524) at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.close(ServerConsumerEndpoint.java:388) at org.jboss.jms.server.endpoint.advised.ConsumerAdvised.org$jboss$jms$server$endpoint$advised$ConsumerAdvised$close$aop(ConsumerAdvised.java:59) at org.jboss.jms.server.endpoint.advised.ConsumerAdvised$close_N4742752445160157748.invokeTarget(ConsumerAdvised$close_N4742752445160157748.java) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111) at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.jms.server.endpoint.advised.ConsumerAdvised.close(ConsumerAdvised.java) at org.jboss.jms.wireformat.CloseRequest.serverInvoke(CloseRequest.java:66) at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:891) at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:744) at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:697) at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:551) at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:232) 2011-01-13 10:00:24,862 TRACE [org.jboss.messaging.core.impl.MessagingQueue] (WorkerThread#1[207.88.61.169:37674]:) Queue[1152414282/1/410-LogServ] removed = true 2011-01-13 10:00:24,862 TRACE [org.jboss.messaging.core.impl.MessagingQueue] (WorkerThread#1[207.88.61.169:37674]:) org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper@66f47f57 removed ConsumerEndpoint[pa-j1tmvuig-1-fctkvuig-85tfp2-o4p1g2r5] 2011-01-13 10:00:24,865 TRACE [org.jboss.messaging.core.impl.MessagingQueue] (WorkerThread#1[207.88.61.169:37674]:) Queue[296106912/1/411-loggingQueue] removing rece iver ConsumerEndpoint[qa-t5tmvuig-1-fctkvuig-85tfp2-o4p1g2r5] java.lang.Exception: trace at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.remove(MessagingQueue.java:623) at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.localClose(ServerConsumerEndpoint.java:524) at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.close(ServerConsumerEndpoint.java:388) at org.jboss.jms.server.endpoint.advised.ConsumerAdvised.org$jboss$jms$server$endpoint$advised$ConsumerAdvised$close$aop(ConsumerAdvised.java:59) at org.jboss.jms.server.endpoint.advised.ConsumerAdvised$close_N4742752445160157748.invokeTarget(ConsumerAdvised$close_N4742752445160157748.java) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111) at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.jms.server.endpoint.advised.ConsumerAdvised.close(ConsumerAdvised.java) at org.jboss.jms.wireformat.CloseRequest.serverInvoke(CloseRequest.java:66) at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:891) at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:744) at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:697) at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:551) at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:232) 2011-01-13 10:00:24,866 TRACE [org.jboss.messaging.core.impl.MessagingQueue] (WorkerThread#1[207.88.61.169:37674]:) Queue[296106912/1/411-loggingQueue] removed = true 2011-01-13 10:00:24,866 TRACE [org.jboss.messaging.core.impl.MessagingQueue] (WorkerThread#1[207.88.61.169:37674]:) Stopping suckers 2011-01-13 10:00:24,866 TRACE [org.jboss.messaging.core.impl.clusterconnection.MessageSucker] (WorkerThread#1[207.88.61.169:37674]:) MessageSucker:1442021912 queue:loggingQueue setConsuming false 2011-01-13 10:00:24,866 TRACE [org.jboss.messaging.core.impl.MessagingQueue] (WorkerThread#1[207.88.61.169:37674]:) Queue[296106912/1/411-loggingQueue] setReceiversReady false 2011-01-13 10:00:24,866 TRACE [org.jboss.messaging.core.impl.clusterconnection.MessageSucker] (WorkerThread#1[207.88.61.169:37674]:) MessageSucker:1442021912 queue:loggingQueue setConsuming false 2011-01-13 10:00:24,866 TRACE [org.jboss.messaging.core.impl.MessagingQueue] (WorkerThread#1[207.88.61.169:37674]:) org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper@528b641b removed ConsumerEndpoint[qa-t5tmvuig-1-fctkvuig-85tfp2-o4p1g2r5] 2011-01-13 10:00:26,872 TRACE [org.jboss.messaging.core.impl.FailoverWaiter] (WorkerThread#32[208.176.159.100:27302]:) Waiting for failover for 2 failingOverFor: -1 failedOverFor: -1 2011-01-13 10:00:26,872 DEBUG [org.jboss.messaging.core.impl.FailoverWaiter] (WorkerThread#32[208.176.159.100:27302]:) org.jboss.messaging.core.impl.FailoverWaiter@199feb94 blocking on the failover lock, waiting for failover to start 2011-01-13 10:00:27,218 TRACE [org.jboss.messaging.core.impl.FailoverWaiter] (WorkerThread#0[208.176.159.100:27434]:) Waiting for failover for 2 failingOverFor: -1 failedOverFor: -1 2011-01-13 10:00:27,221 DEBUG [org.jboss.messaging.core.impl.FailoverWaiter] (WorkerThread#0[208.176.159.100:27434]:) org.jboss.messaging.core.impl.FailoverWaiter@199feb94 blocking on the failover lock, waiting for failover to start 2011-01-13 10:00:27,278 TRACE [org.jboss.messaging.core.impl.FailoverWaiter] (WorkerThread#1[208.176.159.100:27436]:) Waiting for failover for 2 failingOverFor: -1 failedOverFor: -1 2011-01-13 10:00:27,278 DEBUG [org.jboss.messaging.core.impl.FailoverWaiter] (WorkerThread#1[208.176.159.100:27436]:) org.jboss.messaging.core.impl.FailoverWaiter@199feb94 blocking on the failover lock, waiting for failover to start 2011-01-13 10:00:27,334 TRACE [org.jboss.messaging.core.impl.FailoverWaiter] (WorkerThread#37[208.176.159.100:27438]:) Waiting for failover for 2 failingOverFor: -1 failedOverFor: -1 2011-01-13 10:00:27,334 DEBUG [org.jboss.messaging.core.impl.FailoverWaiter] (WorkerThread#37[208.176.159.100:27438]:) org.jboss.messaging.core.impl.FailoverWaiter@199feb94 blocking on the failover lock, waiting for failover to start (these pairs of TRACE/DEBUG messages repeat 32, omitting several...) 2011-01-13 10:00:29,140 TRACE [org.jboss.messaging.core.impl.FailoverWaiter] (WorkerThread#34[208.176.159.100:27496]:) Waiting for failover for 2 failingOverFor: -1 failedOverFor: -1 2011-01-13 10:00:29,140 DEBUG [org.jboss.messaging.core.impl.FailoverWaiter] (WorkerThread#34[208.176.159.100:27496]:) org.jboss.messaging.core.impl.FailoverWaiter@199feb94 blocking on the failover lock, waiting for failover to start 2011-01-13 10:00:32,072 INFO [org.jboss.messaging.core.impl.postoffice.GroupMember] (Incoming-3,207.88.61.168:33006:) org.jboss.messaging.core.impl.postoffice.GroupMember$ControlMembershipListener@5b580e04 got new view [207.88.61.168:33006|2] [207.88.61.168:33006], old view is [207.88.61.168:33006|1] [207.88.61.168:33006, 207.88.61.169:32962] 2011-01-13 10:00:32,072 INFO [org.jboss.messaging.core.impl.postoffice.GroupMember] (Incoming-3,207.88.61.168:33006:) I am (207.88.61.168:33006) 2011-01-13 10:00:32,072 TRACE [org.jboss.messaging.core.impl.postoffice.MessagingPostOffice] (Incoming-3,207.88.61.168:33006:) Nodes left 1 2011-01-13 10:00:32,072 TRACE [org.jboss.messaging.core.impl.postoffice.MessagingPostOffice] (Incoming-3,207.88.61.168:33006:) Old failover node id: 2 2011-01-13 10:00:32,072 DEBUG [org.jboss.messaging.core.impl.postoffice.MessagingPostOffice] (Incoming-3,207.88.61.168:33006:) Updated failover map: 1->1 2011-01-13 10:00:32,072 TRACE [org.jboss.messaging.core.impl.postoffice.MessagingPostOffice] (Incoming-3,207.88.61.168:33006:) First node is now true 2011-01-13 10:00:32,072 DEBUG [org.jboss.messaging.core.impl.postoffice.MessagingPostOffice] (Incoming-3,207.88.61.168:33006:) org.jboss.messaging.core.impl.postoffice.MessagingPostOffice@42f96744: 207.88.61.169:32962 left 2011-01-13 10:00:32,072 DEBUG [org.jboss.messaging.core.impl.postoffice.MessagingPostOffice] (Incoming-3,207.88.61.168:33006:) org.jboss.messaging.core.impl.postoffice.MessagingPostOffice@42f96744: node 2 has crashed 2011-01-13 10:00:32,072 DEBUG [org.jboss.messaging.core.impl.postoffice.MessagingPostOffice] (Incoming-3,207.88.61.168:33006:) org.jboss.messaging.core.impl.postoffice.MessagingPostOffice@42f96744 the failover node for the crashed node is 1 2011-01-13 10:00:32,090 ERROR [org.jgroups.blocks.ConnectionTable] (ViewHandler,MessagingPostOffice-DATA,207.88.61.168:7900:) failed sending data to 207.88.61.169:7900: java.net.SocketException: Socket closed 2011-01-13 10:00:34,074 WARN [org.jgroups.protocols.pbcast.GMS] (ViewHandler,MessagingPostOffice-CTRL,207.88.61.168:33006:) 207.88.61.168:33006 failed to collect all ACKs (1) for mcasted view [207.88.61.168:33006|2] [207.88.61.168:33006] after 2000ms, missing ACKs from [207.88.61.168:33006] (received=[]), local_addr=207.88.61.168:33006 2011-01-13 10:01:26,884 DEBUG [org.jboss.messaging.core.impl.FailoverWaiter] (WorkerThread#32[208.176.159.100:27302]:) org.jboss.messaging.core.impl.FailoverWaiter@199feb94 releasing the failover lock, checking again whether failover started ... 2011-01-13 10:01:26,884 DEBUG [org.jboss.messaging.core.impl.FailoverWaiter] (WorkerThread#32[208.176.159.100:27302]:) Timed out waiting for failover to start 2011-01-13 10:01:27,233 DEBUG [org.jboss.messaging.core.impl.FailoverWaiter] (WorkerThread#0[208.176.159.100:27434]:) org.jboss.messaging.core.impl.FailoverWaiter@199feb94 releasing the failover lock, checking again whether failover started ... 2011-01-13 10:01:27,233 DEBUG [org.jboss.messaging.core.impl.FailoverWaiter] (WorkerThread#0[208.176.159.100:27434]:) Timed out waiting for failover to start 2011-01-13 10:01:27,291 DEBUG [org.jboss.messaging.core.impl.FailoverWaiter] (WorkerThread#1[208.176.159.100:27436]:) org.jboss.messaging.core.impl.FailoverWaiter@199feb94 releasing the failover lock, checking again whether failover started ... (these pairs of DEBUG/DEBUG statements repeat 32 times total, omitting several...) 2011-01-13 10:01:29,152 DEBUG [org.jboss.messaging.core.impl.FailoverWaiter] (WorkerThread#34[208.176.159.100:27496]:) org.jboss.messaging.core.impl.FailoverWaiter@199feb94 releasing the failover lock, checking again whether failover started ... 2011-01-13 10:01:29,152 DEBUG [org.jboss.messaging.core.impl.FailoverWaiter] (WorkerThread#34[208.176.159.100:27496]:) Timed out waiting for failover to start