[ http://jira.jboss.com/jira/browse/JBREM-821?page=comments#action_12383307 ] Ron Sigal commented on JBREM-821: --------------------------------- Copying Tim Fox's comment from JBMESSAGING-1114: I found out why this is happening (why the callback handler can't be found). : 1. An invocation fails because remoting timed out waiting for a TCP connection to be returned to the pool. (Pool size to small). 2. In this case remoting throws a java.net.SocketException: throw new SocketException("Can not obtain client socket connection from pool. " + "Have waited " + (System.currentTimeMillis() - start) + " milliseconds for available connection (" + usedPooled + "in use)"); 3. JBM catches the exception and since it is a java.net.SocketException, assumes some fatal problem has happened with the connection, and falsely initiates failover to another node. 4. Before failing over JBM closes the failed connection. 5. This results in the JBR Callback Connector getting closed. 6. All this while callbacks are still arriving from the server, since that connection is actually still fine. 7. The connector closing process removes the callbackhandler from the callback ServerInvoker so any more callbacks arrive barf with not being able to find the handler. So: After increasing the client pool size I'm not getting these exceptions any more, but imho there are two issues here: 1) Why does remoting throw SocketException? - I would say this is inappropriate since the socket is fine - I would suggest some kind of org.jboss.remoting.RemotingException should be thrown? This would allow JBM to catch it and not initiate failover in this case. In the mean-time I will have to do some kind of text comparison: if (exception.getMessage().startsWith("Can not obtain client....)) { //don't do failover } else { //do failover } This will work for now but is ugly and brittle. 2) The connector close process is not clean. I would suggest the code should be changed so the server threads are shut down at the beginning of the connector close process. I.e. a) Wait for current invocations on that server invoker to complete - and don't allow any more. b) Once all are complete shut it down. (I believe we also have a deadlock on JBoss AS that Clebert discovered that was something due to connectors closing. I don't know if this is related?) -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://jira.jboss.com/jira/browse/JBREM-821?page=comments#action_12383311 ] Ron Sigal commented on JBREM-821: --------------------------------- Regarding the deadlock that Clebert discovered (JBREM-822): I don't think it's related. The deadlock occurs on the server side while JBossAS is shutting down. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://jira.jboss.com/jira/browse/JBREM-821?page=comments#action_12384094 ] Trustin Lee commented on JBREM-821: ----------------------------------- I agree with Tim that throwing a SocketException when the connection pool is exhausted doesn't make sense. However, changing Remoting to throw RemotingException might potentially break backward compatibility although I am not sure. Ron could clarify this. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://jira.jboss.com/jira/browse/JBREM-821?page=comments#action_12389509 ] Ron Sigal commented on JBREM-821: --------------------------------- Various changes have been made to org.jboss.remoting.transport.socket.SocketServerInvoker and org.jboss.remoting.transport.socket.ServerThread in response to issue 2) above: 1. ServerThread.processInvocation() has been partiioned into processInvocation() and completeProcessInvocation(). The former reads a version number from the wire, and the latter, which is synchronized, reads and processes the invocation. 2. ServerThread.shutdown(), which is also synchronized now, closes the socket used by the ServerThread. Since completeProcessInvocation() is synchronized, shutdown can not close the socket in the middle of an invocation, and it will not return until any invocation currently being processed is complete. Since SocketServerInvoker.stop() calls ServerThread.shutdown() on all worker threads, it will not return until (a) all invocations are complete, and (b) all worker threads are guaranteed to be unable to accept any more invocations. Testing: The *ShutdownTestCase unit tests orginally created for JBREM-674 "add test case for client exiting correctly" also serve as unit tests for the current JIRA issue. The tests are: org.jboss.test.remoting.transport.bisocket.BisocketShutdownTestCase org.jboss.test.remoting.transport.bisocket.ssl.SSLBisocketShutdownTestCase org.jboss.test.remoting.transport.http.HTTPShutdownTestCase org.jboss.test.remoting.transport.http.ssl.HTTPSShutdownTestCase org.jboss.test.remoting.transport.rmi.RMIShutdownTestCase org.jboss.test.remoting.transport.rmi.ssl.SSLRMIShutdownTestCase org.jboss.test.remoting.transport.socket.SocketShutdownTestCase org.jboss.test.remoting.transport.socket.ssl.SSLSocketShutdownTestCase A new test method, testOpenClient(), has been added to the root class, org.jboss.test.remoting.shutdown.ShutdownTestParent, which verifies that the server can shut down even if the client does not. A new test method, testWithServerThreadsInAcknowledge(), has been added to SocketShutdownTestCase, which is the root of the socket and bisocket tests. testWithServerThreadsInAcknowledge() turns on the connection checking facility, which implies that ServerThreads waiting for an invocation are waiting in ServerThread.acknowledge() instead of ServerThread.readVersion(). -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://jira.jboss.com/jira/browse/JBREM-821?page=comments#action_12393293 ] Ron Sigal commented on JBREM-821: --------------------------------- Previous test renamed org.jboss.test.remoting.transport.socket.threadpool.ThreadPoolEvictionTestCase. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://jira.jboss.com/jira/browse/JBREM-821?page=comments#action_12396974 ] Ron Sigal commented on JBREM-821: --------------------------------- Of the shutdown tests mentioned about, only the bisocket and socket tests are relevant. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://jira.jboss.com/jira/browse/JBREM-821?page=all ] Ron Sigal closed JBREM-821. --------------------------- Resolution: Done The test cases have been passing, except for hudson run #39, jdk 1.4, where SSLBIsocketShutdownTestCase failed. Log output is not available. Closing the case for now, but will reopen if any tests fail again. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira