[jboss-jira] [JBoss JIRA] Updated: (JBMESSAGING-1280) Messaging client hangs forever with ClusteredConnectionFactory failover by socket hang

[Takayoshi Kimura (JIRA)]

     [ http://jira.jboss.com/jira/browse/JBMESSAGING-1280?page=all ]

Takayoshi Kimura updated JBMESSAGING-1280:
------------------------------------------

    Description: 
Tested with a QueueSession.commit() method call. Non-Clustered setup works fine and ExceptionListener is called as expected.

Taking a look at client thread dump, it seems there is a deadlock on FailoverValve2. The ConnectionValidator thread is waiting the messaging client thread leaving FailoverValve2, but it won't happen because the client thread is entering the FailoverValve2 and reading dead tcp socket. ExceptionListener is never called.

To reproduce this problem:

1. Apply the following patch to JBossMessaging_1_4_0_SP3_CP02:

Index: src/main/org/jboss/jms/server/endpoint/ServerConnectionEndpoint.java
===================================================================
--- src/main/org/jboss/jms/server/endpoint/ServerConnectionEndpoint.java	(revision 4127)
+++ src/main/org/jboss/jms/server/endpoint/ServerConnectionEndpoint.java	(working copy)
@@ -484,7 +484,10 @@
          if (request.getRequestType() == TransactionRequest.ONE_PHASE_COMMIT_REQUEST)
          {
             if (trace) { log.trace(this + " received ONE_PHASE_COMMIT request"); }
+            System.out.println("********** Sleep 10 sec");
 
+            Thread.sleep(10000);
+
             Transaction tx = tr.createTransaction();
             processTransaction(request.getState(), tx, checkForDuplicates);
             tx.commit();


Or use debugger to suspend thread at the ServerConnectionEndpoint.sendTransaction() method.

2. Call QueueSession.commit() method on the client side

conn = cf.createQueueConnection();
conn.setExceptionListener(new ExceptionListener());
conn.start();
session = conn.createQueueSession(true, Session.SESSION_TRANSACTED);
sender = session.createSender(queue);
sender.send(session.createTextMessage("hello"));
session.commit();
sender.close();

3. Disconnect the network cable from the server machine when the client call reached to the ServerConnectionEndpoint.sendTransaction() method

Or use a firewall. For example, iptables on the server side:

iptables -A INPUT -s CLIENT_IPADDRESS/32 -j DROP
iptables -A OUTPUT -d CLIENT_IPADDRESS/32 -j DROP


  was:
Tested with a QueueSession.commit() method call.

Taking a look at client thread dump, it seems there is a deadlock on FailoverValve2. The ConnectionValidator thread is waiting the messaging client thread leaving FailoverValve2, but it won't happen because the client thread is entering the FailoverValve2 and reading dead tcp socket. ExceptionListener is never called.

To reproduce this problem:

1. Apply the following patch to JBossMessaging_1_4_0_SP3_CP02:

Index: src/main/org/jboss/jms/server/endpoint/ServerConnectionEndpoint.java
===================================================================
--- src/main/org/jboss/jms/server/endpoint/ServerConnectionEndpoint.java	(revision 4127)
+++ src/main/org/jboss/jms/server/endpoint/ServerConnectionEndpoint.java	(working copy)
@@ -484,7 +484,10 @@
          if (request.getRequestType() == TransactionRequest.ONE_PHASE_COMMIT_REQUEST)
          {
             if (trace) { log.trace(this + " received ONE_PHASE_COMMIT request"); }
+            System.out.println("********** Sleep 10 sec");
 
+            Thread.sleep(10000);
+
             Transaction tx = tr.createTransaction();
             processTransaction(request.getState(), tx, checkForDuplicates);
             tx.commit();


Or use debugger to suspend thread at the ServerConnectionEndpoint.sendTransaction() method.

2. Call QueueSession.commit() method on the client side

conn = cf.createQueueConnection();
conn.setExceptionListener(new ExceptionListener());
conn.start();
session = conn.createQueueSession(true, Session.SESSION_TRANSACTED);
sender = session.createSender(queue);
sender.send(session.createTextMessage("hello"));
session.commit();
sender.close();

3. Disconnect the network cable from the server machine when the client call reached to the ServerConnectionEndpoint.sendTransaction() method

Or use a firewall. For example, iptables on the server side:

iptables -A INPUT -s CLIENT_IPADDRESS/32 -j DROP
iptables -A OUTPUT -d CLIENT_IPADDRESS/32 -j DROP



> Messaging client hangs forever with ClusteredConnectionFactory failover by socket hang
> --------------------------------------------------------------------------------------
>
>                 Key: JBMESSAGING-1280
>                 URL: http://jira.jboss.com/jira/browse/JBMESSAGING-1280
>             Project: JBoss Messaging
>          Issue Type: Bug
>    Affects Versions: 1.4.0.SP3.CP02, 1.4.0.SP3_CP01
>            Reporter: Takayoshi Kimura
>         Assigned To: Tim Fox
>         Attachments: client.console, client.log
>
>
> Tested with a QueueSession.commit() method call. Non-Clustered setup works fine and ExceptionListener is called as expected.
> Taking a look at client thread dump, it seems there is a deadlock on FailoverValve2. The ConnectionValidator thread is waiting the messaging client thread leaving FailoverValve2, but it won't happen because the client thread is entering the FailoverValve2 and reading dead tcp socket. ExceptionListener is never called.
> To reproduce this problem:
> 1. Apply the following patch to JBossMessaging_1_4_0_SP3_CP02:
> Index: src/main/org/jboss/jms/server/endpoint/ServerConnectionEndpoint.java
> ===================================================================
> --- src/main/org/jboss/jms/server/endpoint/ServerConnectionEndpoint.java	(revision 4127)
> +++ src/main/org/jboss/jms/server/endpoint/ServerConnectionEndpoint.java	(working copy)
> @@ -484,7 +484,10 @@
>           if (request.getRequestType() == TransactionRequest.ONE_PHASE_COMMIT_REQUEST)
>           {
>              if (trace) { log.trace(this + " received ONE_PHASE_COMMIT request"); }
> +            System.out.println("********** Sleep 10 sec");
>  
> +            Thread.sleep(10000);
> +
>              Transaction tx = tr.createTransaction();
>              processTransaction(request.getState(), tx, checkForDuplicates);
>              tx.commit();
> Or use debugger to suspend thread at the ServerConnectionEndpoint.sendTransaction() method.
> 2. Call QueueSession.commit() method on the client side
> conn = cf.createQueueConnection();
> conn.setExceptionListener(new ExceptionListener());
> conn.start();
> session = conn.createQueueSession(true, Session.SESSION_TRANSACTED);
> sender = session.createSender(queue);
> sender.send(session.createTextMessage("hello"));
> session.commit();
> sender.close();
> 3. Disconnect the network cable from the server machine when the client call reached to the ServerConnectionEndpoint.sendTransaction() method
> Or use a firewall. For example, iptables on the server side:
> iptables -A INPUT -s CLIENT_IPADDRESS/32 -j DROP
> iptables -A OUTPUT -d CLIENT_IPADDRESS/32 -j DROP

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira