[jboss-user] [JBoss Remoting] - Remote Client.addCallbackListener hang
david malechek
do-not-reply at jboss.com
Tue Apr 12 02:13:09 EDT 2011
david malechek [http://community.jboss.org/people/dmalechek] created the discussion
"Remote Client.addCallbackListener hang"
To view the discussion, visit: http://community.jboss.org/message/599414#599414
--------------------------------------------------------------
Hi,
I'm having an issue where a remote client call to createQueueConnection that hangs and never returns until a socket timeout.
We are using JBoss 4.2.3GA, Messaging 1.4.4, Remoting 2.2.3SP3. The system involves 500+ remote client machines sending 500K+ messages a day to a Queue with a message listener attached. There are a few other minor JMS queue activities as well.
It runs well until about once a day, usually during heavy load, all the clients hang, requiring a restart of the server. The CPU runs at 20% with plenty of heap and a strong Oracle box.
At this point I'm trying to figure out if I have a remoting configuration mistake, a client code mistake, a possible remoting bug, or just a limitation of the architecture during a high stress point. In the client, I don't re-use any of the JMS objects from one send to the next. I haven't been able to reproduce in a development environment yet.
I did try to back out the change made in JBREM-1081 to ServerInvoker in 2.2.3 to look like the 2.5.4 version of getCallbackHandler. It did removed the BLOCKED threads I show below, but it still fails somewhere in the ServerInvokerCallbackHandler.connect() call, probably in BisocketClientInvoker.handleConnect(). My client is reluctant to turn on trace mode in production so I'll continue to add custom logging until I track this down. My next configuration change will be trying a smaller pingFrequency.
The remote client side Thread dump looks like this.
3XMTHREADINFO "SEQMsgHandling" (TID:0x094A4800, sys_thread_t:0x094C9FB0, state:R, native ID:0x00007B14) prio=5
4XESTACKTRACE at java/net/SocketInputStream.socketRead0(Native Method)
4XESTACKTRACE at java/net/SocketInputStream.read(SocketInputStream.java:155(Compiled Code))
4XESTACKTRACE at java/io/BufferedInputStream.fill(BufferedInputStream.java:229(Compiled Code))
4XESTACKTRACE at java/io/BufferedInputStream.read(BufferedInputStream.java:246(Compiled Code))
4XESTACKTRACE at java/io/FilterInputStream.read(FilterInputStream.java:89(Compiled Code))
4XESTACKTRACE at org/jboss/remoting/transport/socket/MicroSocketClientInvoker.readVersion(MicroSocketClientInvoker.java:1077(Compiled Code))
4XESTACKTRACE at org/jboss/remoting/transport/socket/MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:682(Compiled Code))
4XESTACKTRACE at org/jboss/remoting/transport/bisocket/BisocketClientInvoker.transport(BisocketClientInvoker.java:458(Compiled Code))
4XESTACKTRACE at org/jboss/remoting/MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:141(Compiled Code))
4XESTACKTRACE at org/jboss/remoting/Client.invoke(Client.java:1925(Compiled Code))
4XESTACKTRACE at org/jboss/remoting/Client.addCallbackListener(Client.java:1994(Compiled Code))
4XESTACKTRACE at org/jboss/remoting/Client.addListener(Client.java:1158(Compiled Code))
4XESTACKTRACE at org/jboss/jms/client/remoting/JMSRemotingConnection.addInvokerCallbackHandler(JMSRemotingConnection.java:259(Compiled Code))
4XESTACKTRACE at org/jboss/jms/client/remoting/JMSRemotingConnection.start(JMSRemotingConnection.java:388(Compiled Code))
4XESTACKTRACE at org/jboss/jms/client/delegate/ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:165(Compiled Code))
4XESTACKTRACE at org/jboss/jms/client/delegate/ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java(Compiled Code))
4XESTACKTRACE at org/jboss/jms/client/container/StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:80(Compiled Code))
4XESTACKTRACE at org/jboss/aop/advice/org/jboss/jms/client/container/StateCreationAspect0.invoke(StateCreationAspect0.java(Compiled Code))
4XESTACKTRACE at org/jboss/jms/client/delegate/ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java(Compiled Code))
4XESTACKTRACE at org/jboss/jms/client/delegate/ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java(Compiled Code))
4XESTACKTRACE at org/jboss/jms/client/JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:205(Compiled Code))
4XESTACKTRACE at org/jboss/jms/client/JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:101(Compiled Code))
4XESTACKTRACE at org/jboss/jms/client/JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:95(Compiled Code))
...
The server side thread dump has all the ServerThread like on of these two:
Thread: WorkerThread#0[10.31.48.175:46194] : priority:5, demon:false, threadId:494, threadState:BLOCKED, lockName:java.util.HashMap at 18acff0
org.jboss.remoting.ServerInvoker.removeCallbackHandler(ServerInvoker.java:1949)
org.jboss.remoting.ServerInvoker.handleInternalInvocation(ServerInvoker.java:1514)
org.jboss.remoting.transport.bisocket.BisocketServerInvoker.handleInternalInvocation(BisocketServerInvoker.java:648)
org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:849)
org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:611)
org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:423)
org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:176)
Thread: WorkerThread#4[10.26.68.175:57044] : priority:5, demon:false, threadId:598, threadState:BLOCKED, lockName:java.util.HashMap at 18acff0
org.jboss.remoting.ServerInvoker.getCallbackHandler(ServerInvoker.java:1927)
org.jboss.remoting.ServerInvoker.handleInternalInvocation(ServerInvoker.java:1501)
org.jboss.remoting.transport.bisocket.BisocketServerInvoker.handleInternalInvocation(BisocketServerInvoker.java:648)
org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:849)
org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:611)
org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:423)
org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:176)
My Server side remoting-bisocket-service.xml contains:
<mbean code="org.jboss.remoting.transport.Connector"
name="jboss.messaging:service=Connector,transport=bisocket"
display-name="Bisocket Transport Connector">
<attribute name="Configuration">
<config>
<invoker transport="bisocket">
<!-- There should be no reason to change these parameters - warning!
Changing them may stop JBoss Messaging working correctly -->
<attribute name="marshaller" isParam="true">org.jboss.jms.wireformat.JMSWireFormat</attribute>
<attribute name="unmarshaller" isParam="true">org.jboss.jms.wireformat.JMSWireFormat</attribute>
<attribute name="dataType" isParam="true">jms</attribute>
<attribute name="socket.check_connection" isParam="true">false</attribute>
<attribute name="serverBindAddress">${jboss.bind.address}</attribute>
<attribute name="serverBindPort">14457</attribute>
<attribute name="clientSocketClass" isParam="true">org.jboss.jms.client.remoting.ClientSocketWrapper</attribute>
<attribute name="serverSocketClass">org.jboss.jms.server.remoting.ServerSocketWrapper</attribute>
<attribute name="numberOfCallRetries" isParam="true">1</attribute>
<attribute name="pingFrequency" isParam="true">214748364</attribute>
<attribute name="pingWindowFactor" isParam="true">10</attribute>
<attribute name="onewayThreadPool">org.jboss.jms.server.remoting.DirectThreadPool</attribute>
<!-- End immutable parameters -->
<attribute name="stopLeaseOnFailure" isParam="true">true</attribute>
<!-- Periodicity of client pings. Server window by default is twice this figure -->
<attribute name="clientLeasePeriod" isParam="true">10000</attribute>
<attribute name="validatorPingPeriod" isParam="true">10000</attribute>
<attribute name="validatorPingTimeout" isParam="true">5000</attribute>
<attribute name="failureDisconnectTimeout" isParam="true">0</attribute>
<attribute name="callbackErrorsAllowed">1</attribute>
<attribute name="registerCallbackListener">false</attribute>
<attribute name="useClientConnectionIdentity" isParam="true">true</attribute>
<attribute name="timeout" isParam="true">120000</attribute>
<!-- Number of seconds to wait for a connection in the client pool to become free -->
<attribute name="numberOfRetries" isParam="true">10</attribute>
<!-- Max Number of connections in client pool. This should be significantly higher than
the max number of sessions/consumers you expect -->
<attribute name="JBM_clientMaxPoolSize" isParam="true">200</attribute>
<!-- The maximum time to wait before timing out on trying to write a message to socket for delivery -->
<attribute name="callbackTimeout">10000</attribute>
<!-- Use these parameters to specify values for binding and connecting control connections to
work with your firewall/NAT configuration
<attribute name="secondaryBindPort">xyz</attribute>
<attribute name="secondaryConnectPort">abc</attribute>
-->
</invoker>
<handlers>
<handler subsystem="JMS">org.jboss.jms.server.remoting.JMSServerInvocationHandler</handler>
</handlers>
</config>
</attribute>
</mbean>
Thanks in advance for any insights.
David Malechek
--------------------------------------------------------------
Reply to this message by going to Community
[http://community.jboss.org/message/599414#599414]
Start a new discussion in JBoss Remoting at Community
[http://community.jboss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2050]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/jboss-user/attachments/20110412/06a50ab8/attachment-0001.html
More information about the jboss-user
mailing list