[jboss-user] [JBoss Remoting Users] - Re: Client hangs when get JMS connection factory

[mjjiangbhr]

Let me further elaborate this issue:

In our application, there are some Message listeners running on the client side, these message listeners will receive messages from queue/topic deployed in JBoss Messaging  

Configuration:
We created our own JMS Connection factory which uses the default remoting connector. As you know, the default remoting connector is configured to use the bisocket transport

During we run our application, we open the JBoss web console to monitor the value of currentClientPoolSize under Jboss.remoting JMX MBean


How to reproduce this issue:
1. Run 5 message listeners in the client side to receive messages from JBoss Messaging, then we monitor the value of currentClientPoolSize, we observe the value is 10
2. After processing several messages, we manually pull out the ethernet cable. After some time, we found the value of currentClientPoolSize is still 10, no any change. (If we killed the process of message listener in client side, then the value of currentClientPoolSize will decrease to 0 immediately)
3. We run another 5 message listeners in client side, then the value of currentClientPoolSize will become 20
4. After we do the same operations above several times, the value of currentClientPoolSize will increase continuously. Once the value of currentClientPoolSize is equal to the MaxPoolSize, then the subsequent incoming client requests will hang, and we will encounter the following exception in server side

  | 2009-10-20 18:08:09,655 ERROR [org.jboss.remoting.transport.socket.ServerThread] Worker thread initi
  | alization failure
  | java.net.SocketException: Connection reset
  | 	at java.net.SocketInputStream.read(SocketInputStream.java:168)
  | 	at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
  | 	at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
  | 	at java.io.FilterInputStream.read(FilterInputStream.java:66)
  | 	at org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:859)
  | 	at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:545)
  | 	at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:406)
  | at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:173)
  | 

Conclusion: JBoss Messaging will not close the failure connections if they are caused by manually pulling out ethernet cable. As a result, the value of currentClientPoolSize will increase continuously and finally the new client requests will hang

We found several similar problems in JBoss JIRA as below:
https://jira.jboss.org/jira/browse/JBMESSAGING-1268 
https://jira.jboss.org/jira/browse/JBREM-947 
https://jira.jboss.org/jira/browse/JBREM-949

In the JBREM-947 and JBREM-949, the remoting wrongly depend on the socket timeout for failure detection, the bug reporter said 'Remoting should not be dependent on the socket timeout for failure detection, the connetion validation and socket timeout should be possible to be configured separately'. According to the JIRA, this issue had been fixed in Remoting 2.2.2. SP8 and 2.4.0.CR2 (Pinto), since it is said 'JBREM-947 - is the proper fix in remoting which will allow us to configure timeouts and validation interval separately'. (stated in JBMESSAGING-1268)

However, according to our test, this issue was not fixed properly. We suspect the Remoting 2.2.3 still wrongly depend on the socket timeout for failure connection detection. So, we tried to change the timeout of bisocket transport from 0 (infinite) to a lower value e.g. 10 seconds, then we repeat the test steps again and observe the jboss web console, fortunately, we found the failure connection will be closed after 10 seconds. In JBREM-947, a daemon thread and validatorPingTimeout parameter are mentioned, we think daemon thread should use parameter validatorPingPeriod and validatorPingTimeout to close the failure connection instead of socket timeout. Thus, we post this to seek your expertise. Please correct us if we are wrong. 



View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4265001#4265001

Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4265001