[jboss-jira] [JBoss JIRA] Updated: (JBMESSAGING-1280) Messaging client hangs forever with ClusteredConnectionFactory failover by socket hang

[ http://jira.jboss.com/jira/browse/JBMESSAGING-1280?page=all ] Takayoshi Kimura updated JBMESSAGING-1280: ------------------------------------------ Description: Tested with a QueueSession.commit() method call. Non-Clustered setup works fine and ExceptionListener is called as expected. Taking a look at client thread dump, it seems there is a deadlock on FailoverValve2. The ConnectionValidator thread is waiting the messaging client thread leaving FailoverValve2, but it won't happen because the client thread is entering the FailoverValve2 and reading dead tcp socket. ExceptionListener is never called. To reproduce this problem: 1. Apply the following patch to JBossMessaging_1_4_0_SP3_CP02: Index: src/main/org/jboss/jms/server/endpoint/ServerConnectionEndpoint.java =================================================================== --- src/main/org/jboss/jms/server/endpoint/ServerConnectionEndpoint.java (revision 4127) +++ src/main/org/jboss/jms/server/endpoint/ServerConnectionEndpoint.java (working copy) @@ -484,7 +484,10 @@ if (request.getRequestType() == TransactionRequest.ONE_PHASE_COMMIT_REQUEST) { if (trace) { log.trace(this + " received ONE_PHASE_COMMIT request"); } + System.out.println("********** Sleep 10 sec"); + Thread.sleep(10000); + Transaction tx = tr.createTransaction(); processTransaction(request.getState(), tx, checkForDuplicates); tx.commit(); Or use debugger to suspend thread at the ServerConnectionEndpoint.sendTransaction() method. 2. Call QueueSession.commit() method on the client side conn = cf.createQueueConnection(); conn.setExceptionListener(new ExceptionListener()); conn.start(); session = conn.createQueueSession(true, Session.SESSION_TRANSACTED); sender = session.createSender(queue); sender.send(session.createTextMessage("hello")); session.commit(); sender.close(); 3. Disconnect the network cable from the server machine when the client call reached to the ServerConnectionEndpoint.sendTransaction() method Or use a firewall. For example, iptables on the server side: iptables -A INPUT -s CLIENT_IPADDRESS/32 -j DROP iptables -A OUTPUT -d CLIENT_IPADDRESS/32 -j DROP was: Tested with a QueueSession.commit() method call. Taking a look at client thread dump, it seems there is a deadlock on FailoverValve2. The ConnectionValidator thread is waiting the messaging client thread leaving FailoverValve2, but it won't happen because the client thread is entering the FailoverValve2 and reading dead tcp socket. ExceptionListener is never called. To reproduce this problem: 1. Apply the following patch to JBossMessaging_1_4_0_SP3_CP02: Index: src/main/org/jboss/jms/server/endpoint/ServerConnectionEndpoint.java =================================================================== --- src/main/org/jboss/jms/server/endpoint/ServerConnectionEndpoint.java (revision 4127) +++ src/main/org/jboss/jms/server/endpoint/ServerConnectionEndpoint.java (working copy) @@ -484,7 +484,10 @@ if (request.getRequestType() == TransactionRequest.ONE_PHASE_COMMIT_REQUEST) { if (trace) { log.trace(this + " received ONE_PHASE_COMMIT request"); } + System.out.println("********** Sleep 10 sec"); + Thread.sleep(10000); + Transaction tx = tr.createTransaction(); processTransaction(request.getState(), tx, checkForDuplicates); tx.commit(); Or use debugger to suspend thread at the ServerConnectionEndpoint.sendTransaction() method. 2. Call QueueSession.commit() method on the client side conn = cf.createQueueConnection(); conn.setExceptionListener(new ExceptionListener()); conn.start(); session = conn.createQueueSession(true, Session.SESSION_TRANSACTED); sender = session.createSender(queue); sender.send(session.createTextMessage("hello")); session.commit(); sender.close(); 3. Disconnect the network cable from the server machine when the client call reached to the ServerConnectionEndpoint.sendTransaction() method Or use a firewall. For example, iptables on the server side: iptables -A INPUT -s CLIENT_IPADDRESS/32 -j DROP iptables -A OUTPUT -d CLIENT_IPADDRESS/32 -j DROP -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira