Note: this issue was originally reported here:
https://community.jboss.org/message/548382#548382
On the web server side we see occasionally see threads with this stack:
75: [Thread[jrpp-1390,5,jrpp]]/id=49440/hc=13398717/shc=*/state=RUNNABLE/name=*
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:237)
java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2249)
java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2429)
java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2499)
java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2571)
java.io.ObjectInputStream.read(ObjectInputStream.java:820)
org.jboss.remoting.transport.socket.MicroSocketClientInvoker.readVersion(MicroSocketClientInvoker.java:1263)
org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:838)
org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:165)
org.jboss.remoting.Client.invoke(Client.java:1724)
org.jboss.remoting.Client.invoke(Client.java:629)
org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:60)
org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
org.jboss.aspects.tx.ClientTxPropagationInterceptor.invoke(ClientTxPropagationInterceptor.java:61)
org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
org.jboss.ejb3.security.client.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:65)
org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
org.jboss.ejb3.remoting.IsLocalInterceptor.invoke(IsLocalInterceptor.java:74)
org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
org.jboss.aspects.remoting.PojiProxy.invoke(PojiProxy.java:62)
$Proxy5.invoke(Unknown Source)
org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.invoke(SessionProxyInvocationHandlerBase.java:207)
org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.invoke(SessionProxyInvocationHandlerBase.java:164)
$Proxy1643.completeMyOrder(Unknown Source)
... rest of our business process thread stack...
On the jboss server for each thread above we usually see at least 2 jboss threads with the following stack:
816: [WorkerThread#10[10.1.4.245:46454]]/id=6923/hc=1806701791/shc=*/state=RUNNABLE/name=*
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:237)
java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2249)
java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2429)
java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2499)
java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2571)
java.io.ObjectInputStream.read(ObjectInputStream.java:820)
org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:1038)
org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:673)
org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:551)
org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:232)
I am not sure what is going on, but it looks like there is something that occasionally gets messed up with these "readVersion(...)" methods.
These threads will hang for hours and days.
The only way we can get these to stop hanging is to send a Thread.interrupt() to the threads on the web server side.
Sending the interrupt to the thread on the jboss side only moves the problem to a different thread and doesn't stop the web server threads from hanging.
We are not sure what is causing this, but it appears to be a bug in jboss.
We are using 5.1.0 GA.
It looks like the class allows for timeout and retry settings, but I am not sure how to set these on the client side.
http://anonsvn.jboss.org/repos/jbossremoting/remoting2/tags/2.5.1/src/main/org/jboss/remoting/transport/socket/MicroSocketClientInvoker.java
The MANIFEST.MF of our jboss-remoting.jar shows version "2.5.1"
Is this related to https://jira.jboss.org/browse/JBREM-1188 ?
If so, I think the default of 30 seconds would be too long for us.
Is there a way to change that via a config file on the client side?