[jboss-jira] [JBoss JIRA] (JBMESSAGING-1917) Networking Failure Locks Post Office

[Doug Grove (JIRA)]

     [ https://issues.jboss.org/browse/JBMESSAGING-1917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Grove updated JBMESSAGING-1917:
------------------------------------

    Environment: JBoss EAP 5.1.2  (was: JBoss EAP .1.2)

    
> Networking Failure Locks Post Office
> ------------------------------------
>
>                 Key: JBMESSAGING-1917
>                 URL: https://issues.jboss.org/browse/JBMESSAGING-1917
>             Project: JBoss Messaging
>          Issue Type: Bug
>          Components: Messaging Core
>    Affects Versions: 1.4.8.SP5
>         Environment: JBoss EAP 5.1.2
>            Reporter: Doug Grove
>
> Several network failure modes can cause the messaging post office to lock.  For example, loss of connectivity to a client can cause Messaging to stop accepting new clients and stop delivering messages.  Message can, however, still be delivered from the clients.
> 220 threads waiting to acquire MessagingPostOffice's ReaderLock.  Likely owned by:
> "WorkManager(2)-7" daemon prio=10 tid=0x25108800 nid=0x1af7 in Object.wait() [0x1a675000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	- waiting on <0x3e6031f8> (a java.util.HashSet)
> 	at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.createSocket(BisocketClientInvoker.java:538)
> 	- locked <0x3e6031f8> (a java.util.HashSet)
> 	at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:1165)
> 	at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:816)
> 	at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:470)
> 	at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:169)
> 	at org.jboss.remoting.Client.invoke(Client.java:2070)
> 	at org.jboss.remoting.Client.invoke(Client.java:879)
> 	at org.jboss.remoting.Client.invokeOneway(Client.java:928)
> 	at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallback(ServerInvokerCallbackHandler.java:835)
> 	at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallbackOneway(ServerInvokerCallbackHandler.java:708)
> 	at org.jboss.jms.server.endpoint.ServerSessionEndpoint.performDelivery(ServerSessionEndpoint.java:1610)
> 	at org.jboss.jms.server.endpoint.ServerSessionEndpoint.handleDelivery(ServerSessionEndpoint.java:1522)
> 	- locked <0x3fd4bc18> (a org.jboss.jms.server.endpoint.ServerSessionEndpoint)
> 	at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:353)
> 	- locked <0x3e5960b8> (a java.lang.Object)
> 	at org.jboss.messaging.core.impl.RoundRobinDistributor.handle(RoundRobinDistributor.java:119)
> 	at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.handle(MessagingQueue.java:617)
> 	at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.handle(ClusterRoundRobinDistributor.java:79)
> 	at org.jboss.messaging.core.impl.ChannelSupport.deliverInternal(ChannelSupport.java:677)
> 	at org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:540)
> 	at org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:251)
> 	- locked <0x3e595e28> (a java.lang.Object)
> 	at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:3163)
> 	at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:980)
> Thread looks to be in this while loop until the timeout passes:
>    while ((isConnected()) && ((!this.pingFailed.flag) || (pingFailedTimeRemaining > 0L)) && ((timeout == 0) || (timeRemaining > 0L)))
>     {
>       synchronized (sockets)
>       {
>         try
>         {
>           sockets.wait(1000L);   //538
>         }
>         catch (InterruptedException e)
>         {
>           log.debug(this + " unexpected interrupt");
>         }
>         if (!sockets.isEmpty())
>         {
>           Iterator it = sockets.iterator();
>           Socket socket = (Socket)it.next();
>           it.remove();
>           configureSocket(socket);
>           log.debug(this + " found socket (" + this.listenerId + "): " + socket);
>           return socket;
>         }
>       }
>       if (savedControlOutputStream != this.controlOutputStream)
>       {
>         savedControlOutputStream = this.controlOutputStream;
>         log.debug(this + " rewriting Bisocket.CREATE_ORDINARY_SOCKET on " + this.controlOutputStream);
>         try
>         {
>           this.controlOutputStream.write(4);
>           log.debug(this + " rewrote Bisocket.CREATE_ORDINARY_SOCKET");
>         }
>         catch (IOException e)
>         {
>           log.debug(this + " unable to rewrite Bisocket.CREATE_ORDINARY_SOCKET" + e.getMessage());
>         }
>       }
>       long elapsed = System.currentTimeMillis() - start;
>       if (timeout > 0)
>         timeRemaining = timeout - elapsed;
>       pingFailedTimeRemaining = pingFailedWindow - elapsed;
>     }
> so he pulls a client NIC and the thread on JBoss hangs on it per his timeout, but hangs while holding the MessagingPostOffice lock to back everything else up.  I don't see a way around that without a messaging code overhaul.  Just set a timely timeout and hope for no NIC failures :)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira