[jboss-user] [JBoss Messaging] - Possible bug with JBoss Messaging on bisocket transport

Husgaard do-not-reply at jboss.com
Wed Nov 2 18:01:12 EDT 2011 

Husgaard [http://community.jboss.org/people/Husgaard] created the discussion

"Possible bug with JBoss Messaging on bisocket transport"

To view the discussion, visit: http://community.jboss.org/message/634694#634694

--------------------------------------------------------------
Hi,

I am using JBoss Messaging 1.4.7.GA with the bisocket transport from JBoss Remoting 2.5.3.SP1 connecting to remote clients.

Ever since I started using this combination I have seen some stability issues on large and busy production systems that I have been unable to reproduce in a test environment. I have mostly seen these issues when the systems have been very busy, and I have noticed that these issues become a lot worse when the network is not good (high packet loss and/or parts of the network disconnecting at times).

Today I think I have found the cause of these issues, and it looks like a bug in the interface between JBoss Messaging and JBoss Remoting. I would like some of you JBoss Messaging experts here to comment on my observations before I open a JIRA ticket.

My findings starts with two thread dumps obtained at a production server experiencing these issues. The thread dumps were taken 8 seconds apart, and both show the same stack trace:

"WorkManager(2)-76" daemon prio=10 tid=0x00002aaae483c000 nid=0x3d40 in Object.wait() [0x000000004770a000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.createSocket(BisocketClientInvoker.java:528)
    - locked <0x000000070f9cb8f8> (a java.util.HashSet)
    at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:1165)
    at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:816)
    at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:461)
    at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:167)
    at org.jboss.remoting.Client.invoke(Client.java:2034)
    at org.jboss.remoting.Client.invoke(Client.java:877)
    at org.jboss.remoting.Client.invokeOneway(Client.java:926) - not client side, means we may block instead of executing in a new thread
    at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallback(ServerInvokerCallbackHandler.java:835)
    at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallbackOneway(ServerInvokerCallbackHandler.java:708) - serverSide=false
    at org.jboss.jms.server.endpoint.ServerSessionEndpoint.performDelivery(ServerSessionEndpoint.java:1467)
    at org.jboss.jms.server.endpoint.ServerSessionEndpoint.handleDelivery(ServerSessionEndpoint.java:1379)
    - locked <0x00000006a8c8dfd0> (a org.jboss.jms.server.endpoint.ServerSessionEndpoint)
    at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:328)
    - locked <0x0000000695caa390> (a java.lang.Object)
    at org.jboss.messaging.core.impl.RoundRobinDistributor.handle(RoundRobinDistributor.java:119)
    at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.handle(MessagingQueue.java:590)
    at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.handle(ClusterRoundRobinDistributor.java:79)
    at org.jboss.messaging.core.impl.ChannelSupport.deliverInternal(ChannelSupport.java:665)
    at org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:513)
    at org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:246)
    - locked <0x0000000695caa498> (a java.lang.Object)
    at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2504)
    at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:580)
    at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:779)
(snip)

This thread is trying to deliver a JMS message and is waiting in BisocketClientInvoker.createSocket() for a remote client to open a connection back to the server. At the same time the thread holds a read lock in MessagingPostOffice. This means that any work in MessagingPostOffice that needs a write lock is blocked.

The problem in this case is that the remote client is no longer connected to the network, so it will never connect back to the server. The wait in BisocketClientInvoker.createSocket() will time out after some time. There is a retry loop in BisocketClientInvoker.createSocket() which will retry a few times, but eventually the call will fail and the read lock in MessagingPostOffice is released. If I read the code correctly the read lock is held for 60 seconds, if the JBoss Remoting settings are default.

Having the MessagingPostOffice lock exposed to client communication problems in this way is not good, and probably not what you want.

Looking at the way JBoss Messaging calls JBoss Remoting when delivering a message to a remote client, I see that you use one-way calls where the caller does not care about a response, and the call is executed in another thread. But in JBoss Remoting there are two way of doing this:
1. Starting a thread on the caller side to handle the call. This way the new thread will take care of any communication problems, and the calling thread can return immediately so the read lock in MessagingPostOffice is immediately released.
2. Starting a new thread on the remote side to handle the call. This way the calling thread has to take care of any communication problems, and only when the invocation has been delivered to the remote side will the remote side start a new thread to handle the call.

Unfortunately it looks like JBoss Messaging is using the last way of doing one-way calls.

So my question is: Isn't this a bug? Shouldn't we use the first way of doing one-way calls instead?
--------------------------------------------------------------

Reply to this message by going to Community
[http://community.jboss.org/message/634694#634694]

Start a new discussion in JBoss Messaging at Community
[http://community.jboss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2042]

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/jboss-user/attachments/20111102/7b63a9ac/attachment.html

More information about the jboss-user mailing list