[jboss-user] [JBoss Remoting Development] - Threads Hanging in MicroSocketClientInvoker.readVersion()

[Nathan Ciliberto]

Nathan Ciliberto [http://community.jboss.org/people/snacker] created the discussion

"Threads Hanging in MicroSocketClientInvoker.readVersion()"

To view the discussion, visit: http://community.jboss.org/message/548604#548604

--------------------------------------------------------------
Note: this issue was originally reported here:
 https://community.jboss.org/message/548382#548382 https://community.jboss.org/message/548382#548382

On the web server side we see occasionally see threads with this stack:

75: [Thread[jrpp-1390,5,jrpp]]/id=49440/hc=13398717/shc=*/state=RUNNABLE/name=*
        java.net.SocketInputStream.socketRead0(Native Method)
        java.net.SocketInputStream.read(SocketInputStream.java:129)
        java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
        java.io.BufferedInputStream.read(BufferedInputStream.java:237)
        java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2249)
        java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2429)
        java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2499)
        java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2571)
        java.io.ObjectInputStream.read(ObjectInputStream.java:820)
        org.jboss.remoting.transport.socket.MicroSocketClientInvoker.*readVersion*(MicroSocketClientInvoker.java:1263)
        org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:838)
        org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:165)
        org.jboss.remoting.Client.invoke(Client.java:1724)
        org.jboss.remoting.Client.invoke(Client.java:629)
        org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:60)
        org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
        org.jboss.aspects.tx.ClientTxPropagationInterceptor.invoke(ClientTxPropagationInterceptor.java:61)
        org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
        org.jboss.ejb3.security.client.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:65)
        org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
        org.jboss.ejb3.remoting.IsLocalInterceptor.invoke(IsLocalInterceptor.java:74)
        org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
        org.jboss.aspects.remoting.PojiProxy.invoke(PojiProxy.java:62)
        $Proxy5.invoke(Unknown Source)
        org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.invoke(SessionProxyInvocationHandlerBase.java:207)
        org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.invoke(SessionProxyInvocationHandlerBase.java:164)
        $Proxy1643.completeMyOrder(Unknown Source)

        ... rest of our business process thread stack...

On the jboss server for each thread above we usually see at least 2 jboss threads with the following stack:

816: [WorkerThread#10[10.1.4.245:46454]]/id=6923/hc=1806701791/shc=*/state=RUNNABLE/name=*
         java.net.SocketInputStream.socketRead0(Native Method)
         java.net.SocketInputStream.read(SocketInputStream.java:129)
         java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
         java.io.BufferedInputStream.read(BufferedInputStream.java:237)
         java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2249)
         java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2429)
         java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2499)
         java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2571)
         java.io.ObjectInputStream.read(ObjectInputStream.java:820)
         org.jboss.remoting.transport.socket.ServerThread.*readVersion*(ServerThread.java:1038)
         org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:673)
         org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:551)
         org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:232)

I am not sure what is going on, but it looks like there is something that occasionally gets messed up with these "readVersion(...)" methods.

These threads will hang for hours and days.

The only way we can get these to stop hanging is to send a Thread.interrupt() to the threads on the web server side.

Sending the interrupt to the thread on the jboss side only moves the problem to a different thread and doesn't stop the web server threads from hanging.

We are not sure what is causing this, but it appears to be a bug in jboss.
We are using 5.1.0 GA.

It looks like the class allows for timeout and retry settings, but I am not sure how to set these on the client side.
 http://anonsvn.jboss.org/repos/jbossremoting/remoting2/tags/2.5.1/src/main/org/jboss/remoting/transport/socket/MicroSocketClientInvoker.java http://anonsvn.jboss.org/repos/jbossremoting/remoting2/tags/2.5.1/src/main/org/jboss/remoting/transport/socket/MicroSocketClientInvoker.java

The MANIFEST.MF of our jboss-remoting.jar shows version "2.5.1"

Is this related to  https://jira.jboss.org/browse/JBREM-1188 https://jira.jboss.org/browse/JBREM-1188 ?

If so, I think the default of 30 seconds would be too long for us.
Is there a way to change that via a config file on the client side?

--------------------------------------------------------------

Reply to this message by going to Community
[http://community.jboss.org/message/548604#548604]

Start a new discussion in JBoss Remoting Development at Community
[http://community.jboss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2051]

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/jboss-user/attachments/20100617/24866b74/attachment.html