3:13 p.m.
Doug Grove created JBMESSAGING-1917:
---------------------------------------
Summary: Networking Failure Locks Post Office
Key: JBMESSAGING-1917
URL: https://issues.jboss.org/browse/JBMESSAGING-1917
Project: JBoss Messaging
Issue Type: Bug
Components: Messaging Core
Affects Versions: 1.4.8.SP5
Environment: JBoss EAP .1.2
Reporter: Doug Grove
Several network failure modes can cause the messaging post office to lock. For example,
loss of connectivity to a client can cause Messaging to stop accepting new clients and
stop delivering messages. Message can, however, still be delivered from the clients.
220 threads waiting to acquire MessagingPostOffice's ReaderLock. Likely owned by:
"WorkManager(2)-7" daemon prio=10 tid=0x25108800 nid=0x1af7 in Object.wait()
[0x1a675000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x3e6031f8> (a java.util.HashSet)
at
org.jboss.remoting.transport.bisocket.BisocketClientInvoker.createSocket(BisocketClientInvoker.java:538)
- locked <0x3e6031f8> (a java.util.HashSet)
at
org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:1165)
at
org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:816)
at
org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:470)
at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:169)
at org.jboss.remoting.Client.invoke(Client.java:2070)
at org.jboss.remoting.Client.invoke(Client.java:879)
at org.jboss.remoting.Client.invokeOneway(Client.java:928)
at
org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallback(ServerInvokerCallbackHandler.java:835)
at
org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallbackOneway(ServerInvokerCallbackHandler.java:708)
at
org.jboss.jms.server.endpoint.ServerSessionEndpoint.performDelivery(ServerSessionEndpoint.java:1610)
at
org.jboss.jms.server.endpoint.ServerSessionEndpoint.handleDelivery(ServerSessionEndpoint.java:1522)
- locked <0x3fd4bc18> (a org.jboss.jms.server.endpoint.ServerSessionEndpoint)
at
org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:353)
- locked <0x3e5960b8> (a java.lang.Object)
at
org.jboss.messaging.core.impl.RoundRobinDistributor.handle(RoundRobinDistributor.java:119)
at
org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.handle(MessagingQueue.java:617)
at
org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.handle(ClusterRoundRobinDistributor.java:79)
at org.jboss.messaging.core.impl.ChannelSupport.deliverInternal(ChannelSupport.java:677)
at org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:540)
at org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:251)
- locked <0x3e595e28> (a java.lang.Object)
at
org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:3163)
at
org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:980)
Thread looks to be in this while loop until the timeout passes:
while ((isConnected()) && ((!this.pingFailed.flag) || (pingFailedTimeRemaining
> 0L)) && ((timeout == 0) || (timeRemaining > 0L)))
{
synchronized (sockets)
{
try
{
sockets.wait(1000L); //538
}
catch (InterruptedException e)
{
log.debug(this + " unexpected interrupt");
}
if (!sockets.isEmpty())
{
Iterator it = sockets.iterator();
Socket socket = (Socket)it.next();
it.remove();
configureSocket(socket);
log.debug(this + " found socket (" + this.listenerId + "): "
+ socket);
return socket;
}
}
if (savedControlOutputStream != this.controlOutputStream)
{
savedControlOutputStream = this.controlOutputStream;
log.debug(this + " rewriting Bisocket.CREATE_ORDINARY_SOCKET on " +
this.controlOutputStream);
try
{
this.controlOutputStream.write(4);
log.debug(this + " rewrote Bisocket.CREATE_ORDINARY_SOCKET");
}
catch (IOException e)
{
log.debug(this + " unable to rewrite Bisocket.CREATE_ORDINARY_SOCKET"
+ e.getMessage());
}
}
long elapsed = System.currentTimeMillis() - start;
if (timeout > 0)
timeRemaining = timeout - elapsed;
pingFailedTimeRemaining = pingFailedWindow - elapsed;
}
so he pulls a client NIC and the thread on JBoss hangs on it per his timeout, but hangs
while holding the MessagingPostOffice lock to back everything else up. I don't see a
way around that without a messaging code overhaul. Just set a timely timeout and hope for
no NIC failures :)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira