[jboss-user] [Remoting] - EJB3/Socket invoker - connection timeouts

[javajedi]

We're running into a problem with a brand new production system that has a single remote EJB.  We are using JBoss EJB3, which uses JBoss Remoting to do the remote communication.  We are using the standard socket invoker to handle the communication.   We are using JBoss Serialization instead of Java Serialization, which, in tests, provided significantly better overall performance.  We've been testing this in a stage environment for several months, with no problems.  Once we got it into production, we started to hit a wall.  We reach a point where the socket connections between the client and the server just start timing out.  A thread dump on the client shows a bunch of threads waiting for a socket connection to the server.  A thread dump on the server shows several SeverThreads waiting to be used (via ServerThread.wakeup()).  The problem appears to be that the thread in SocketServerInvoker that is responsible for the calls to serverSocket.accept() is getting stuck, or at least severely slowed.  In the server side thread dump, here is the stack of that accept thread:

Thread: SocketServerInvoker#0-3873 : priority:5, demon:false, threadId:23, threadState:RUNNABLE, threadLockName:null
java.net.SocketInputStream.socketRead0(Native Method)
        java.net.SocketInputStream.read(SocketInputStream.java:129)
        java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
        java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
        java.io.BufferedInputStream.read(BufferedInputStream.java:317)
        java.io.FilterInputStream.read(FilterInputStream.java:90)
        org.jboss.serial.io.JBossObjectInputStream.checkSignature(JBossObjectInputStream.java:110)
        org.jboss.serial.io.JBossObjectInputStream.(JBossObjectInputStream.java:94)
        org.jboss.serial.io.JBossObjectInputStream.(JBossObjectInputStream.java:83)
        org.jboss.remoting.serialization.impl.jboss.JBossSerializationManager.createInput(JBossSerializationManager.java:57)
        org.jboss.remoting.transport.socket.ServerSocketWrapper.createInputStream(ServerSocketWrapper.java:56)
        org.jboss.remoting.transport.socket.ClientSocketWrapper.createStreams(ClientSocketWrapper.java:76)
        org.jboss.remoting.transport.socket.ClientSocketWrapper.(ClientSocketWrapper.java:54)
        org.jboss.remoting.transport.socket.ServerSocketWrapper.(ServerSocketWrapper.java:50)
        sun.reflect.GeneratedConstructorAccessor13.newInstance(Unknown Source)
        sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        org.jboss.remoting.transport.socket.ServerThread.createServerSocket(ServerThread.java:185)
        org.jboss.remoting.transport.socket.ServerThread.(ServerThread.java:87)
        org.jboss.remoting.transport.socket.SocketServerInvoker.processInvocation(SocketServerInvoker.java:429)
        org.jboss.remoting.transport.socket.SocketServerInvoker.run(SocketServerInvoker.java:391)
        java.lang.Thread.run(Thread.java:619)
        
It appears that this thread, which ought to be quickly finding a ServerThread to handle the incoming request, calling ServerThread.start(), and getting back to the call to serverSocket.accept() as quickly as possible, is instead doing some socket I/O.  ServerSocketWrapper.java has a method called createInputStream() that creates a new ObjectInputStream.  The stack trace shows that the JBossObjectInputStream constructor is reading some bytes from the input stream, but I don't think that this problem is specific to JBoss Serialization, as java.io.ObjectInputStream's constructor also reads bytes from the stream.

It seems to me that this could be the cause of the problem that we are seeing.  Any time that you're doing a blocking socket read like this, you can potentially just sit for a while until the socket times out.  When you've got a thread calling serverSocket.accept(), you can't normally afford to do something like this on that thread.

Does this analysis make sense?  Any thoughts/suggestions would be greatly appreciated.

Thanks.

--Tim

View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4057841#4057841

Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4057841