david malechek [
http://community.jboss.org/people/dmalechek] created the discussion
"Remote Client.addCallbackListener hang"
To view the discussion, visit:
http://community.jboss.org/message/599414#599414
--------------------------------------------------------------
Hi,
I'm having an issue where a remote client call to createQueueConnection that hangs and
never returns until a socket timeout.
We are using JBoss 4.2.3GA, Messaging 1.4.4, Remoting 2.2.3SP3. The system involves 500+
remote client machines sending 500K+ messages a day to a Queue with a message listener
attached. There are a few other minor JMS queue activities as well.
It runs well until about once a day, usually during heavy load, all the clients hang,
requiring a restart of the server. The CPU runs at 20% with plenty of heap and a strong
Oracle box.
At this point I'm trying to figure out if I have a remoting configuration mistake, a
client code mistake, a possible remoting bug, or just a limitation of the architecture
during a high stress point. In the client, I don't re-use any of the JMS objects from
one send to the next. I haven't been able to reproduce in a development environment
yet.
I did try to back out the change made in JBREM-1081 to ServerInvoker in 2.2.3 to look like
the 2.5.4 version of getCallbackHandler. It did removed the BLOCKED threads I show below,
but it still fails somewhere in the ServerInvokerCallbackHandler.connect() call, probably
in BisocketClientInvoker.handleConnect(). My client is reluctant to turn on trace mode in
production so I'll continue to add custom logging until I track this down. My next
configuration change will be trying a smaller pingFrequency.
The remote client side Thread dump looks like this.
3XMTHREADINFO "SEQMsgHandling" (TID:0x094A4800, sys_thread_t:0x094C9FB0,
state:R, native ID:0x00007B14) prio=5
4XESTACKTRACE at java/net/SocketInputStream.socketRead0(Native Method)
4XESTACKTRACE at
java/net/SocketInputStream.read(SocketInputStream.java:155(Compiled Code))
4XESTACKTRACE at
java/io/BufferedInputStream.fill(BufferedInputStream.java:229(Compiled Code))
4XESTACKTRACE at
java/io/BufferedInputStream.read(BufferedInputStream.java:246(Compiled Code))
4XESTACKTRACE at
java/io/FilterInputStream.read(FilterInputStream.java:89(Compiled Code))
4XESTACKTRACE at
org/jboss/remoting/transport/socket/MicroSocketClientInvoker.readVersion(MicroSocketClientInvoker.java:1077(Compiled
Code))
4XESTACKTRACE at
org/jboss/remoting/transport/socket/MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:682(Compiled
Code))
4XESTACKTRACE at
org/jboss/remoting/transport/bisocket/BisocketClientInvoker.transport(BisocketClientInvoker.java:458(Compiled
Code))
4XESTACKTRACE at
org/jboss/remoting/MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:141(Compiled
Code))
4XESTACKTRACE at org/jboss/remoting/Client.invoke(Client.java:1925(Compiled
Code))
4XESTACKTRACE at
org/jboss/remoting/Client.addCallbackListener(Client.java:1994(Compiled Code))
4XESTACKTRACE at org/jboss/remoting/Client.addListener(Client.java:1158(Compiled
Code))
4XESTACKTRACE at
org/jboss/jms/client/remoting/JMSRemotingConnection.addInvokerCallbackHandler(JMSRemotingConnection.java:259(Compiled
Code))
4XESTACKTRACE at
org/jboss/jms/client/remoting/JMSRemotingConnection.start(JMSRemotingConnection.java:388(Compiled
Code))
4XESTACKTRACE at
org/jboss/jms/client/delegate/ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:165(Compiled
Code))
4XESTACKTRACE at
org/jboss/jms/client/delegate/ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java(Compiled
Code))
4XESTACKTRACE at
org/jboss/jms/client/container/StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:80(Compiled
Code))
4XESTACKTRACE at
org/jboss/aop/advice/org/jboss/jms/client/container/StateCreationAspect0.invoke(StateCreationAspect0.java(Compiled
Code))
4XESTACKTRACE at
org/jboss/jms/client/delegate/ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java(Compiled
Code))
4XESTACKTRACE at
org/jboss/jms/client/delegate/ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java(Compiled
Code))
4XESTACKTRACE at
org/jboss/jms/client/JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:205(Compiled
Code))
4XESTACKTRACE at
org/jboss/jms/client/JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:101(Compiled
Code))
4XESTACKTRACE at
org/jboss/jms/client/JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:95(Compiled
Code))
...
The server side thread dump has all the ServerThread like on of these two:
Thread: WorkerThread#0[10.31.48.175:46194] : priority:5, demon:false, threadId:494,
threadState:BLOCKED, lockName:java.util.HashMap@18acff0
org.jboss.remoting.ServerInvoker.removeCallbackHandler(ServerInvoker.java:1949)
org.jboss.remoting.ServerInvoker.handleInternalInvocation(ServerInvoker.java:1514)
org.jboss.remoting.transport.bisocket.BisocketServerInvoker.handleInternalInvocation(BisocketServerInvoker.java:648)
org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:849)
org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:611)
org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:423)
org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:176)
Thread: WorkerThread#4[10.26.68.175:57044] : priority:5, demon:false, threadId:598,
threadState:BLOCKED, lockName:java.util.HashMap@18acff0
org.jboss.remoting.ServerInvoker.getCallbackHandler(ServerInvoker.java:1927)
org.jboss.remoting.ServerInvoker.handleInternalInvocation(ServerInvoker.java:1501)
org.jboss.remoting.transport.bisocket.BisocketServerInvoker.handleInternalInvocation(BisocketServerInvoker.java:648)
org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:849)
org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:611)
org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:423)
org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:176)
My Server side remoting-bisocket-service.xml contains:
<mbean code="org.jboss.remoting.transport.Connector"
name="jboss.messaging:service=Connector,transport=bisocket"
display-name="Bisocket Transport Connector">
<attribute name="Configuration">
<config>
<invoker transport="bisocket">
<!-- There should be no reason to change these parameters - warning!
Changing them may stop JBoss Messaging working correctly
-->
<attribute name="marshaller"
isParam="true">org.jboss.jms.wireformat.JMSWireFormat</attribute>
<attribute name="unmarshaller"
isParam="true">org.jboss.jms.wireformat.JMSWireFormat</attribute>
<attribute name="dataType"
isParam="true">jms</attribute>
<attribute name="socket.check_connection"
isParam="true">false</attribute>
<attribute
name="serverBindAddress">${jboss.bind.address}</attribute>
<attribute name="serverBindPort">14457</attribute>
<attribute name="clientSocketClass"
isParam="true">org.jboss.jms.client.remoting.ClientSocketWrapper</attribute>
<attribute
name="serverSocketClass">org.jboss.jms.server.remoting.ServerSocketWrapper</attribute>
<attribute name="numberOfCallRetries"
isParam="true">1</attribute>
<attribute name="pingFrequency"
isParam="true">214748364</attribute>
<attribute name="pingWindowFactor"
isParam="true">10</attribute>
<attribute
name="onewayThreadPool">org.jboss.jms.server.remoting.DirectThreadPool</attribute>
<!-- End immutable parameters -->
<attribute name="stopLeaseOnFailure"
isParam="true">true</attribute>
<!-- Periodicity of client pings. Server window by default is twice this
figure -->
<attribute name="clientLeasePeriod"
isParam="true">10000</attribute>
<attribute name="validatorPingPeriod"
isParam="true">10000</attribute>
<attribute name="validatorPingTimeout"
isParam="true">5000</attribute>
<attribute name="failureDisconnectTimeout"
isParam="true">0</attribute>
<attribute
name="callbackErrorsAllowed">1</attribute>
<attribute
name="registerCallbackListener">false</attribute>
<attribute name="useClientConnectionIdentity"
isParam="true">true</attribute>
<attribute name="timeout"
isParam="true">120000</attribute>
<!-- Number of seconds to wait for a connection in the client pool to
become free -->
<attribute name="numberOfRetries"
isParam="true">10</attribute>
<!-- Max Number of connections in client pool. This should be
significantly higher than
the max number of sessions/consumers you expect -->
<attribute name="JBM_clientMaxPoolSize"
isParam="true">200</attribute>
<!-- The maximum time to wait before timing out on trying to write a
message to socket for delivery -->
<attribute name="callbackTimeout">10000</attribute>
<!-- Use these parameters to specify values for binding and connecting
control connections to
work with your firewall/NAT configuration
<attribute
name="secondaryBindPort">xyz</attribute>
<attribute
name="secondaryConnectPort">abc</attribute>
-->
</invoker>
<handlers>
<handler
subsystem="JMS">org.jboss.jms.server.remoting.JMSServerInvocationHandler</handler>
</handlers>
</config>
</attribute>
</mbean>
Thanks in advance for any insights.
David Malechek
--------------------------------------------------------------
Reply to this message by going to Community
[
http://community.jboss.org/message/599414#599414]
Start a new discussion in JBoss Remoting at Community
[
http://community.jboss.org/choose-container!input.jspa?contentType=1&...]