[jboss-jira] [JBoss JIRA] Created: (JBMESSAGING-1840) Deadlock on cluster failover
Gerald Turner (JIRA)
jira-events at lists.jboss.org
Thu Jan 13 14:19:50 EST 2011
Deadlock on cluster failover
----------------------------
Key: JBMESSAGING-1840
URL: https://issues.jboss.org/browse/JBMESSAGING-1840
Project: JBoss Messaging
Issue Type: Bug
Components: JMS Clustering
Affects Versions: 1.4.3.GA
Environment: JBoss 5.1.0.GA in "all" configuration, PostOffice has FailoverOnNodeLeave = true, using MySQL Cluster 7.1.9a backend (ndb-persistence-service.xml)
Reporter: Gerald Turner
I have two JBoss 5.1.0.GA servers in a cluster, with a clustered queue named LogServ, and with an application which spawns several message consumers (not an MDB, each thread does it's own java:/ConnectionFactory lookup, and creates QueueSession, QueueReceiver, etc.). External to the JBoss VMs I have some code which is sending messages to the queue, this code uses HA-JNDI, looks up java:/ClusteredConnectionFactory, shares the QueueConnectionFactory between several threads, each thread has it's own QueueConnection/Session/Sender - taking advantage of the round-robin feature of ClusteredConnectionFactory (half the threads will be connected to one server, and the other half connected to the other server). The consumer application (in JBoss) has 32 threads on each JBoss server. The sender application (outside JBoss) has 64 threads, evenly split between the two JBoss servers.
I have ensured that cluster failover is working properly by undeploying the consumer application, let the queue fill up with several tens of thousands of messages, then shutdown a server and watch the PostOffice take control of the downed servers Channel (several times back and forth). I have also witnessed failover working properly with the consumer application deployed, although with only a few hundred outstanding messages to process.
Now while the consumer application is deployed, and the external sender application is running, when I shutdown a node, the running server enters a deadlock during failover.
Java Thread Dump detailing the deadlock:
Found one Java-level deadlock:
=============================
"WorkerThread#35[208.176.159.100:27297]":
waiting to lock monitor 0x00000000537fa8f8 (object 0x00002aaace27a010, a java.lang.Object),
which is held by "WorkerThread#28[208.176.159.100:27288]"
"WorkerThread#28[208.176.159.100:27288]":
waiting to lock monitor 0x000000005340a9c0 (object 0x00002aaacd704bb0, a org.jboss.messaging.core.impl.clusterconnection.MessageSucker),
which is held by "ReusableThread"
"ReusableThread":
waiting to lock monitor 0x00000000537fa8f8 (object 0x00002aaace27a010, a java.lang.Object),
which is held by "WorkerThread#28[208.176.159.100:27288]"
Java stack information for the threads listed above:
===================================================
"WorkerThread#35[208.176.159.100:27297]":
at org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:235)
- waiting to lock <0x00002aaace27a010> (a java.lang.Object)
at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2222)
at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:509)
at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:755)
at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:399)
at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeTarget(SessionAdvised$send_7280680627620114891.java)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
at sun.reflect.GeneratedMethodAccessor320.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:891)
at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:744)
- locked <0x00002aaacbf53338> (a org.jboss.remoting.transport.socket.ServerThread)
at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:697)
at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:551)
at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:232)
"WorkerThread#28[208.176.159.100:27288]":
at org.jboss.messaging.core.impl.clusterconnection.MessageSucker.setConsuming(MessageSucker.java:200)
- waiting to lock <0x00002aaacd704bb0> (a org.jboss.messaging.core.impl.clusterconnection.MessageSucker)
at org.jboss.messaging.core.impl.MessagingQueue.informSuckers(MessagingQueue.java:566)
at org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:516)
at org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:239)
- locked <0x00002aaace27a010> (a java.lang.Object)
at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2222)
at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:509)
at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:755)
at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:399)
at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeTarget(SessionAdvised$send_7280680627620114891.java)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
at sun.reflect.GeneratedMethodAccessor320.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:891)
at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:744)
- locked <0x00002aaace4dc8e8> (a org.jboss.remoting.transport.socket.ServerThread)
at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:697)
at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:551)
at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:232)
"ReusableThread":
at org.jboss.messaging.core.impl.MessagingQueue.unregisterSucker(MessagingQueue.java:387)
- waiting to lock <0x00002aaace27a010> (a java.lang.Object)
at org.jboss.messaging.core.impl.clusterconnection.MessageSucker.stop(MessageSucker.java:152)
- locked <0x00002aaacd704bb0> (a org.jboss.messaging.core.impl.clusterconnection.MessageSucker)
at org.jboss.messaging.core.impl.clusterconnection.ClusterConnectionManager$ConnectionInfo.closeAllSuckers(ClusterConnectionManager.java:688)
- locked <0x00002aaacdf348f0> (a org.jboss.messaging.core.impl.clusterconnection.ClusterConnectionManager$ConnectionInfo)
at org.jboss.messaging.core.impl.clusterconnection.ClusterConnectionManager$ConnectionInfo.close(ClusterConnectionManager.java:696)
- locked <0x00002aaacdf348f0> (a org.jboss.messaging.core.impl.clusterconnection.ClusterConnectionManager$ConnectionInfo)
at org.jboss.messaging.core.impl.clusterconnection.ClusterConnectionManager.notify(ClusterConnectionManager.java:279)
- locked <0x00002aaaccb114b0> (a org.jboss.messaging.core.impl.clusterconnection.ClusterConnectionManager)
at org.jboss.messaging.core.impl.DefaultClusterNotifier.sendNotification(DefaultClusterNotifier.java:72)
at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.removeReplicantLocally(MessagingPostOffice.java:1223)
at org.jboss.messaging.core.impl.postoffice.RemoveReplicantRequest.execute(RemoveReplicantRequest.java:57)
at org.jboss.messaging.core.impl.postoffice.GroupMember$ControlRequestHandler.handle(GroupMember.java:622)
at org.jgroups.blocks.MessageDispatcher.handle(MessageDispatcher.java:616)
at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:637)
at org.jgroups.blocks.RequestCorrelator$Request.run(RequestCorrelator.java:944)
at org.jgroups.util.ReusableThread.run(ReusableThread.java:220)
at java.lang.Thread.run(Thread.java:662)
Found 1 deadlock.
JBoss log of the server taking over the downed node:
2011-01-13 09:55:23,724 INFO [org.jboss.ha.framework.interfaces.HAPartition.lifecycle.DevelopmentMySQLPartition] (Incoming-9,207.88.61.168:33006:) New cluster view for partition DevelopmentMySQLPartition (id: 2, delta: -1) : [207.88.61.168:1099]
2011-01-13 09:55:23,724 INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.DevelopmentMySQLPartition] (AsynchViewChangeHandler Thread:) I am (207.88.61.168:1099) received membershipChanged event:
2011-01-13 09:55:23,724 INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.DevelopmentMySQLPartition] (AsynchViewChangeHandler Thread:) Dead members: 1 ([207.88.61.169:1099])
2011-01-13 09:55:23,724 INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.DevelopmentMySQLPartition] (AsynchViewChangeHandler Thread:) New Members : 0 ([])
2011-01-13 09:55:23,724 INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.DevelopmentMySQLPartition] (AsynchViewChangeHandler Thread:) All Members : 1 ([207.88.61.168:1099])
2011-01-13 10:00:24,862 TRACE [org.jboss.messaging.core.impl.MessagingQueue] (WorkerThread#1[207.88.61.169:37674]:) Queue[1152414282/1/410-LogServ] removing receiver ConsumerEndpoint[pa-j1tmvuig-1-fctkvuig-85tfp2-o4p1g2r5]
java.lang.Exception: trace
at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.remove(MessagingQueue.java:623)
at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.localClose(ServerConsumerEndpoint.java:524)
at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.close(ServerConsumerEndpoint.java:388)
at org.jboss.jms.server.endpoint.advised.ConsumerAdvised.org$jboss$jms$server$endpoint$advised$ConsumerAdvised$close$aop(ConsumerAdvised.java:59)
at org.jboss.jms.server.endpoint.advised.ConsumerAdvised$close_N4742752445160157748.invokeTarget(ConsumerAdvised$close_N4742752445160157748.java)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.jms.server.endpoint.advised.ConsumerAdvised.close(ConsumerAdvised.java)
at org.jboss.jms.wireformat.CloseRequest.serverInvoke(CloseRequest.java:66)
at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:891)
at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:744)
at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:697)
at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:551)
at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:232)
2011-01-13 10:00:24,862 TRACE [org.jboss.messaging.core.impl.MessagingQueue] (WorkerThread#1[207.88.61.169:37674]:) Queue[1152414282/1/410-LogServ] removed = true
2011-01-13 10:00:24,862 TRACE [org.jboss.messaging.core.impl.MessagingQueue] (WorkerThread#1[207.88.61.169:37674]:) org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper at 66f47f57 removed ConsumerEndpoint[pa-j1tmvuig-1-fctkvuig-85tfp2-o4p1g2r5]
2011-01-13 10:00:24,865 TRACE [org.jboss.messaging.core.impl.MessagingQueue] (WorkerThread#1[207.88.61.169:37674]:) Queue[296106912/1/411-loggingQueue] removing rece
iver ConsumerEndpoint[qa-t5tmvuig-1-fctkvuig-85tfp2-o4p1g2r5]
java.lang.Exception: trace
at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.remove(MessagingQueue.java:623)
at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.localClose(ServerConsumerEndpoint.java:524)
at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.close(ServerConsumerEndpoint.java:388)
at org.jboss.jms.server.endpoint.advised.ConsumerAdvised.org$jboss$jms$server$endpoint$advised$ConsumerAdvised$close$aop(ConsumerAdvised.java:59)
at org.jboss.jms.server.endpoint.advised.ConsumerAdvised$close_N4742752445160157748.invokeTarget(ConsumerAdvised$close_N4742752445160157748.java)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.jms.server.endpoint.advised.ConsumerAdvised.close(ConsumerAdvised.java)
at org.jboss.jms.wireformat.CloseRequest.serverInvoke(CloseRequest.java:66)
at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:891)
at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:744)
at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:697)
at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:551)
at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:232)
2011-01-13 10:00:24,866 TRACE [org.jboss.messaging.core.impl.MessagingQueue] (WorkerThread#1[207.88.61.169:37674]:) Queue[296106912/1/411-loggingQueue] removed = true
2011-01-13 10:00:24,866 TRACE [org.jboss.messaging.core.impl.MessagingQueue] (WorkerThread#1[207.88.61.169:37674]:) Stopping suckers
2011-01-13 10:00:24,866 TRACE [org.jboss.messaging.core.impl.clusterconnection.MessageSucker] (WorkerThread#1[207.88.61.169:37674]:) MessageSucker:1442021912 queue:loggingQueue setConsuming false
2011-01-13 10:00:24,866 TRACE [org.jboss.messaging.core.impl.MessagingQueue] (WorkerThread#1[207.88.61.169:37674]:) Queue[296106912/1/411-loggingQueue] setReceiversReady false
2011-01-13 10:00:24,866 TRACE [org.jboss.messaging.core.impl.clusterconnection.MessageSucker] (WorkerThread#1[207.88.61.169:37674]:) MessageSucker:1442021912 queue:loggingQueue setConsuming false
2011-01-13 10:00:24,866 TRACE [org.jboss.messaging.core.impl.MessagingQueue] (WorkerThread#1[207.88.61.169:37674]:) org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper at 528b641b removed ConsumerEndpoint[qa-t5tmvuig-1-fctkvuig-85tfp2-o4p1g2r5]
2011-01-13 10:00:26,872 TRACE [org.jboss.messaging.core.impl.FailoverWaiter] (WorkerThread#32[208.176.159.100:27302]:) Waiting for failover for 2 failingOverFor: -1 failedOverFor: -1
2011-01-13 10:00:26,872 DEBUG [org.jboss.messaging.core.impl.FailoverWaiter] (WorkerThread#32[208.176.159.100:27302]:) org.jboss.messaging.core.impl.FailoverWaiter at 199feb94 blocking on the failover lock, waiting for failover to start
2011-01-13 10:00:27,218 TRACE [org.jboss.messaging.core.impl.FailoverWaiter] (WorkerThread#0[208.176.159.100:27434]:) Waiting for failover for 2 failingOverFor: -1 failedOverFor: -1
2011-01-13 10:00:27,221 DEBUG [org.jboss.messaging.core.impl.FailoverWaiter] (WorkerThread#0[208.176.159.100:27434]:) org.jboss.messaging.core.impl.FailoverWaiter at 199feb94 blocking on the failover lock, waiting for failover to start
2011-01-13 10:00:27,278 TRACE [org.jboss.messaging.core.impl.FailoverWaiter] (WorkerThread#1[208.176.159.100:27436]:) Waiting for failover for 2 failingOverFor: -1 failedOverFor: -1
2011-01-13 10:00:27,278 DEBUG [org.jboss.messaging.core.impl.FailoverWaiter] (WorkerThread#1[208.176.159.100:27436]:) org.jboss.messaging.core.impl.FailoverWaiter at 199feb94 blocking on the failover lock, waiting for failover to start
2011-01-13 10:00:27,334 TRACE [org.jboss.messaging.core.impl.FailoverWaiter] (WorkerThread#37[208.176.159.100:27438]:) Waiting for failover for 2 failingOverFor: -1 failedOverFor: -1
2011-01-13 10:00:27,334 DEBUG [org.jboss.messaging.core.impl.FailoverWaiter] (WorkerThread#37[208.176.159.100:27438]:) org.jboss.messaging.core.impl.FailoverWaiter at 199feb94 blocking on the failover lock, waiting for failover to start
(these pairs of TRACE/DEBUG messages repeat 32, omitting several...)
2011-01-13 10:00:29,140 TRACE [org.jboss.messaging.core.impl.FailoverWaiter] (WorkerThread#34[208.176.159.100:27496]:) Waiting for failover for 2 failingOverFor: -1 failedOverFor: -1
2011-01-13 10:00:29,140 DEBUG [org.jboss.messaging.core.impl.FailoverWaiter] (WorkerThread#34[208.176.159.100:27496]:) org.jboss.messaging.core.impl.FailoverWaiter at 199feb94 blocking on the failover lock, waiting for failover to start
2011-01-13 10:00:32,072 INFO [org.jboss.messaging.core.impl.postoffice.GroupMember] (Incoming-3,207.88.61.168:33006:) org.jboss.messaging.core.impl.postoffice.GroupMember$ControlMembershipListener at 5b580e04 got new view [207.88.61.168:33006|2] [207.88.61.168:33006], old view is [207.88.61.168:33006|1] [207.88.61.168:33006, 207.88.61.169:32962]
2011-01-13 10:00:32,072 INFO [org.jboss.messaging.core.impl.postoffice.GroupMember] (Incoming-3,207.88.61.168:33006:) I am (207.88.61.168:33006)
2011-01-13 10:00:32,072 TRACE [org.jboss.messaging.core.impl.postoffice.MessagingPostOffice] (Incoming-3,207.88.61.168:33006:) Nodes left 1
2011-01-13 10:00:32,072 TRACE [org.jboss.messaging.core.impl.postoffice.MessagingPostOffice] (Incoming-3,207.88.61.168:33006:) Old failover node id: 2
2011-01-13 10:00:32,072 DEBUG [org.jboss.messaging.core.impl.postoffice.MessagingPostOffice] (Incoming-3,207.88.61.168:33006:) Updated failover map:
1->1
2011-01-13 10:00:32,072 TRACE [org.jboss.messaging.core.impl.postoffice.MessagingPostOffice] (Incoming-3,207.88.61.168:33006:) First node is now true
2011-01-13 10:00:32,072 DEBUG [org.jboss.messaging.core.impl.postoffice.MessagingPostOffice] (Incoming-3,207.88.61.168:33006:) org.jboss.messaging.core.impl.postoffice.MessagingPostOffice at 42f96744: 207.88.61.169:32962 left
2011-01-13 10:00:32,072 DEBUG [org.jboss.messaging.core.impl.postoffice.MessagingPostOffice] (Incoming-3,207.88.61.168:33006:) org.jboss.messaging.core.impl.postoffice.MessagingPostOffice at 42f96744: node 2 has crashed
2011-01-13 10:00:32,072 DEBUG [org.jboss.messaging.core.impl.postoffice.MessagingPostOffice] (Incoming-3,207.88.61.168:33006:) org.jboss.messaging.core.impl.postoffice.MessagingPostOffice at 42f96744 the failover node for the crashed node is 1
2011-01-13 10:00:32,090 ERROR [org.jgroups.blocks.ConnectionTable] (ViewHandler,MessagingPostOffice-DATA,207.88.61.168:7900:) failed sending data to 207.88.61.169:7900: java.net.SocketException: Socket closed
2011-01-13 10:00:34,074 WARN [org.jgroups.protocols.pbcast.GMS] (ViewHandler,MessagingPostOffice-CTRL,207.88.61.168:33006:) 207.88.61.168:33006 failed to collect all ACKs (1) for mcasted view [207.88.61.168:33006|2] [207.88.61.168:33006] after 2000ms, missing ACKs from [207.88.61.168:33006] (received=[]), local_addr=207.88.61.168:33006
2011-01-13 10:01:26,884 DEBUG [org.jboss.messaging.core.impl.FailoverWaiter] (WorkerThread#32[208.176.159.100:27302]:) org.jboss.messaging.core.impl.FailoverWaiter at 199feb94 releasing the failover lock, checking again whether failover started ...
2011-01-13 10:01:26,884 DEBUG [org.jboss.messaging.core.impl.FailoverWaiter] (WorkerThread#32[208.176.159.100:27302]:) Timed out waiting for failover to start
2011-01-13 10:01:27,233 DEBUG [org.jboss.messaging.core.impl.FailoverWaiter] (WorkerThread#0[208.176.159.100:27434]:) org.jboss.messaging.core.impl.FailoverWaiter at 199feb94 releasing the failover lock, checking again whether failover started ...
2011-01-13 10:01:27,233 DEBUG [org.jboss.messaging.core.impl.FailoverWaiter] (WorkerThread#0[208.176.159.100:27434]:) Timed out waiting for failover to start
2011-01-13 10:01:27,291 DEBUG [org.jboss.messaging.core.impl.FailoverWaiter] (WorkerThread#1[208.176.159.100:27436]:) org.jboss.messaging.core.impl.FailoverWaiter at 199feb94 releasing the failover lock, checking again whether failover started ...
(these pairs of DEBUG/DEBUG statements repeat 32 times total, omitting several...)
2011-01-13 10:01:29,152 DEBUG [org.jboss.messaging.core.impl.FailoverWaiter] (WorkerThread#34[208.176.159.100:27496]:) org.jboss.messaging.core.impl.FailoverWaiter at 199feb94 releasing the failover lock, checking again whether failover started ...
2011-01-13 10:01:29,152 DEBUG [org.jboss.messaging.core.impl.FailoverWaiter] (WorkerThread#34[208.176.159.100:27496]:) Timed out waiting for failover to start
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
More information about the jboss-jira
mailing list