[jboss-user] [JBoss Remoting] - Remote Client.addCallbackListener hang

david malechek [http://community.jboss.org/people/dmalechek] created the discussion "Remote Client.addCallbackListener hang" To view the discussion, visit: http://community.jboss.org/message/599414#599414 -------------------------------------------------------------- Hi, I'm having an issue where a remote client call to createQueueConnection that hangs and never returns until a socket timeout. We are using JBoss 4.2.3GA, Messaging 1.4.4, Remoting 2.2.3SP3.  The system involves 500+ remote client machines sending 500K+ messages a day to a Queue with a message listener attached.  There are a few other minor JMS queue activities as well. It runs well until about once a day, usually during heavy load, all the clients hang, requiring a restart of the server.  The CPU runs at 20% with plenty of heap and a strong Oracle box. At this point I'm trying to figure out if I have a remoting configuration mistake, a client code mistake, a possible remoting bug, or just a limitation of the architecture during a high stress point.  In the client, I don't re-use any of the JMS objects from one send to the next.  I haven't been able to reproduce in a development environment yet. I did try to back out the change made in JBREM-1081 to ServerInvoker in 2.2.3 to look like the 2.5.4 version of getCallbackHandler.  It did removed the BLOCKED threads I show below, but it still fails somewhere in the ServerInvokerCallbackHandler.connect() call, probably in BisocketClientInvoker.handleConnect().  My client is reluctant to turn on trace mode in production so I'll continue to add custom logging until I track this down. My next configuration change will be trying a smaller pingFrequency. The remote client side Thread dump looks like this. 3XMTHREADINFO      "SEQMsgHandling" (TID:0x094A4800, sys_thread_t:0x094C9FB0, state:R, native ID:0x00007B14) prio=5 4XESTACKTRACE          at java/net/SocketInputStream.socketRead0(Native Method) 4XESTACKTRACE          at java/net/SocketInputStream.read(SocketInputStream.java:155(Compiled Code)) 4XESTACKTRACE          at java/io/BufferedInputStream.fill(BufferedInputStream.java:229(Compiled Code)) 4XESTACKTRACE          at java/io/BufferedInputStream.read(BufferedInputStream.java:246(Compiled Code)) 4XESTACKTRACE          at java/io/FilterInputStream.read(FilterInputStream.java:89(Compiled Code)) 4XESTACKTRACE          at org/jboss/remoting/transport/socket/MicroSocketClientInvoker.readVersion(MicroSocketClientInvoker.java:1077(Compiled Code)) 4XESTACKTRACE          at org/jboss/remoting/transport/socket/MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:682(Compiled Code)) 4XESTACKTRACE          at org/jboss/remoting/transport/bisocket/BisocketClientInvoker.transport(BisocketClientInvoker.java:458(Compiled Code)) 4XESTACKTRACE          at org/jboss/remoting/MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:141(Compiled Code)) 4XESTACKTRACE          at org/jboss/remoting/Client.invoke(Client.java:1925(Compiled Code)) 4XESTACKTRACE          at org/jboss/remoting/Client.addCallbackListener(Client.java:1994(Compiled Code)) 4XESTACKTRACE          at org/jboss/remoting/Client.addListener(Client.java:1158(Compiled Code)) 4XESTACKTRACE          at org/jboss/jms/client/remoting/JMSRemotingConnection.addInvokerCallbackHandler(JMSRemotingConnection.java:259(Compiled Code)) 4XESTACKTRACE          at org/jboss/jms/client/remoting/JMSRemotingConnection.start(JMSRemotingConnection.java:388(Compiled Code)) 4XESTACKTRACE          at org/jboss/jms/client/delegate/ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:165(Compiled Code)) 4XESTACKTRACE          at org/jboss/jms/client/delegate/ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java(Compiled Code)) 4XESTACKTRACE          at org/jboss/jms/client/container/StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:80(Compiled Code)) 4XESTACKTRACE          at org/jboss/aop/advice/org/jboss/jms/client/container/StateCreationAspect0.invoke(StateCreationAspect0.java(Compiled Code)) 4XESTACKTRACE          at org/jboss/jms/client/delegate/ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java(Compiled Code)) 4XESTACKTRACE          at org/jboss/jms/client/delegate/ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java(Compiled Code)) 4XESTACKTRACE          at org/jboss/jms/client/JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:205(Compiled Code)) 4XESTACKTRACE          at org/jboss/jms/client/JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:101(Compiled Code)) 4XESTACKTRACE          at org/jboss/jms/client/JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:95(Compiled Code)) ... The server side thread dump has all the ServerThread like on of these two: Thread: WorkerThread#0[10.31.48.175:46194] : priority:5, demon:false, threadId:494, threadState:BLOCKED, lockName:java.util.HashMap@18acff0     org.jboss.remoting.ServerInvoker.removeCallbackHandler(ServerInvoker.java:1949)     org.jboss.remoting.ServerInvoker.handleInternalInvocation(ServerInvoker.java:1514)     org.jboss.remoting.transport.bisocket.BisocketServerInvoker.handleInternalInvocation(BisocketServerInvoker.java:648)     org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:849)     org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:611)     org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:423)     org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:176) Thread: WorkerThread#4[10.26.68.175:57044] : priority:5, demon:false, threadId:598, threadState:BLOCKED, lockName:java.util.HashMap@18acff0     org.jboss.remoting.ServerInvoker.getCallbackHandler(ServerInvoker.java:1927)     org.jboss.remoting.ServerInvoker.handleInternalInvocation(ServerInvoker.java:1501)     org.jboss.remoting.transport.bisocket.BisocketServerInvoker.handleInternalInvocation(BisocketServerInvoker.java:648)     org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:849)     org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:611)     org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:423)     org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:176) My Server side remoting-bisocket-service.xml contains:    <mbean code="org.jboss.remoting.transport.Connector"           name="jboss.messaging:service=Connector,transport=bisocket"           display-name="Bisocket Transport Connector">       <attribute name="Configuration">          <config>             <invoker transport="bisocket">                  <!-- There should be no reason to change these parameters - warning!                     Changing them may stop JBoss Messaging working correctly -->                           <attribute name="marshaller" isParam="true">org.jboss.jms.wireformat.JMSWireFormat</attribute>                <attribute name="unmarshaller" isParam="true">org.jboss.jms.wireformat.JMSWireFormat</attribute>                <attribute name="dataType" isParam="true">jms</attribute>                <attribute name="socket.check_connection" isParam="true">false</attribute>                              <attribute name="serverBindAddress">${jboss.bind.address}</attribute>                <attribute name="serverBindPort">14457</attribute>                <attribute name="clientSocketClass" isParam="true">org.jboss.jms.client.remoting.ClientSocketWrapper</attribute>                <attribute name="serverSocketClass">org.jboss.jms.server.remoting.ServerSocketWrapper</attribute>                <attribute name="numberOfCallRetries" isParam="true">1</attribute>                <attribute name="pingFrequency" isParam="true">214748364</attribute>                <attribute name="pingWindowFactor" isParam="true">10</attribute>                <attribute name="onewayThreadPool">org.jboss.jms.server.remoting.DirectThreadPool</attribute>                              <!-- End immutable parameters -->                <attribute name="stopLeaseOnFailure" isParam="true">true</attribute>                <!-- Periodicity of client pings. Server window by default is twice this figure -->                                              <attribute name="clientLeasePeriod" isParam="true">10000</attribute>                <attribute name="validatorPingPeriod" isParam="true">10000</attribute>                <attribute name="validatorPingTimeout" isParam="true">5000</attribute>                <attribute name="failureDisconnectTimeout" isParam="true">0</attribute>                <attribute name="callbackErrorsAllowed">1</attribute>                <attribute name="registerCallbackListener">false</attribute>                <attribute name="useClientConnectionIdentity" isParam="true">true</attribute>                 <attribute name="timeout" isParam="true">120000</attribute>                <!-- Number of seconds to wait for a connection in the client pool to become free -->                <attribute name="numberOfRetries" isParam="true">10</attribute>                <!-- Max Number of connections in client pool. This should be significantly higher than                     the max number of sessions/consumers you expect -->                <attribute name="JBM_clientMaxPoolSize" isParam="true">200</attribute>                <!-- The maximum time to wait before timing out on trying to write a message to socket for delivery -->                <attribute name="callbackTimeout">10000</attribute>                <!-- Use these parameters to specify values for binding and connecting control connections to                     work with your firewall/NAT configuration                <attribute name="secondaryBindPort">xyz</attribute>                                          <attribute name="secondaryConnectPort">abc</attribute>                              -->            </invoker>             <handlers>                <handler subsystem="JMS">org.jboss.jms.server.remoting.JMSServerInvocationHandler</handler>             </handlers>          </config>       </attribute>    </mbean> Thanks in advance for any insights. David Malechek -------------------------------------------------------------- Reply to this message by going to Community [http://community.jboss.org/message/599414#599414] Start a new discussion in JBoss Remoting at Community [http://community.jboss.org/choose-container!input.jspa?contentType=1&...]