3:21 a.m.
We are using JBoss App Server 4.2.3.GA, JBoss Messaging 1.4.5 GA and JBoss Remoting 2.2.3
SP1. In our application, there are a lot of Message listeners running on the client side,
these message listeners will receive messages from queue/topic deployed in JBoss Messaging
Configuration:
We created our own JMS Connection factory which uses the default remoting connector. As
you know, the default remoting connector is configured to use the bisocket transport. We
didn't change the default value of the remoting connector
<invoker transport="bisocket">
|
| <!-- There should be no reason to change these parameters -
warning!
| Changing them may stop JBoss Messaging working correctly -->
| <attribute name="marshaller"
isParam="true">org.jboss.jms.wireformat.JMSWireFormat</attribute>
| <attribute name="unmarshaller"
isParam="true">org.jboss.jms.wireformat.JMSWireFormat</attribute>
| <attribute name="dataType"
isParam="true">jms</attribute>
| <attribute name="socket.check_connection"
isParam="true">false</attribute>
| <attribute
name="serverBindAddress">${jboss.bind.address}</attribute>
| <attribute
name="serverBindPort">4457</attribute>
| <attribute name="clientSocketClass"
isParam="true">org.jboss.jms.client.remoting.ClientSocketWrapper</attribute>
| <attribute
name="serverSocketClass">org.jboss.jms.server.remoting.ServerSocketWrapper</attribute>
|
| <attribute name="numberOfCallRetries"
isParam="true">1</attribute>
| <attribute name="pingFrequency"
isParam="true">214748364</attribute>
| <attribute name="pingWindowFactor"
isParam="true">10</attribute>
| <attribute
name="onewayThreadPool">org.jboss.jms.server.remoting.DirectThreadPool</attribute>
| <!-- End immutable parameters -->
|
| <attribute name="stopLeaseOnFailure"
isParam="true">true</attribute>
|
| <!-- Periodicity of client pings. Server window by default is twice
this figure -->
| <attribute name="clientLeasePeriod"
isParam="true">10000</attribute>
| <attribute name="validatorPingPeriod"
isParam="true">10000</attribute>
| <attribute name="validatorPingTimeout"
isParam="true">5000</attribute>
| <attribute
name="registerCallbackListener">false</attribute>
|
| <attribute name="timeout"
isParam="true">0</attribute>
|
| <!-- Number of seconds to wait for a connection in the client pool
to become free -->
| <attribute name="numberOfRetries"
isParam="true">10</attribute>
|
| <!-- Max Number of connections in client pool. This should be
significantly higher than
| the max number of sessions/consumers you expect -->
| <attribute name="JBM_clientMaxPoolSize"
isParam="true">200</attribute>
|
| <!-- The maximum time to wait before timing out on trying to write a
message to socket for delivery -
| ->
| <attribute
name="callbackTimeout">10000</attribute>
| <attribute
name="secondaryBindPort">4557</attribute>
|
| <!-- Use these parameters to specify values for binding and
connecting control connections to
| work with your firewall/NAT configuration
| <attribute
name="secondaryBindPort">xyz</attribute>
| <attribute
name="secondaryConnectPort">abc</attribute>
| -->
|
| </invoker>
|
During we run our application, we open the JBoss web console to monitor the value of
currentClientPoolSize under "Jboss.remoting" JMX MBean.
How to reproduce this issue:
1. Run 5 message listeners in the client side to receive messages from JBoss Messaging,
then we observe the value of currentClientPoolSize is 10
2. After processing several messages, we manually pull out the ethernet cable. The value
of currentClientPoolSize is still 10.
3. We run another 5 message listeners in client side, then the value of
currentClientPoolSize will become 20
4. After we do the same operations above several times, the value of currentClientPoolSize
will increase continuously. Once the value of currentClientPoolSize is equal to the
MaxPoolSize, then the subsequent incoming client requests will hang, and we will encounter
the following exception in server side
2009-10-20 18:08:09,655 ERROR [org.jboss.remoting.transport.socket.ServerThread] Worker
thread initialization failure
|
| java.net.SocketException: Connection reset
|
| at java.net.SocketInputStream.read(SocketInputStream.java:168)
|
| at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
|
| at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
|
| at java.io.FilterInputStream.read(FilterInputStream.java:66)
|
| at
org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:859)
|
| at
org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:545)
|
| at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:406)
|
| at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:173)
|
Conclusion: JBoss Messaging won't close the failure connections if they are caused by
manually pulling out ethernet cable. As a result, the value of currentClientPoolSize will
increase continuously and finally the new client requests will hang
Note: If we killed the process of message listener in client side, then the value of
currentClientPoolSize will decrease to 0 immediately, it seems that the server could
detect the failure connection and perform the corresponding resource releasing.
View the original post :
http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4266489#...
Reply to the post :
http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&a...