[jboss-jira] [JBoss JIRA] Commented: (JBMESSAGING-1456) Messages stuck in being-delivered state in cluster

[Howard Gao (JIRA)]

    [ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12440142#action_12440142 ] 

Howard Gao commented on JBMESSAGING-1456:
-----------------------------------------

Thanks Zach. I configured JBOSS with 2 nodes clustered. I deployed the following queues as specified in the testClientConnectivityContext.xml file:

SSNDefaultQ
SSNExportQ
SSNGetMeterFieldStatusResultsQ
SSNEventQ
SSNODRQ
HeartBeatQ

To simplify, I also changed the config to use ClusteredXAConnectionFactory instead of SecureClusteredXAConnectionFactory (which I assume to be an indication of using secure socket transport). 

When I started the client using: 
java -Xms256m -Xmx256m -jar TestClient.jar conf/spring-contexts/testClientConnectivityContext.xml

I couldn't see anything happened instead of two line output as follows

log4j:WARN No appenders could be found for logger (org.springframework.context.support.FileSystemXmlApplicationContext).
log4j:WARN Please initialize the log4j system properly.

This is pretty much the same when I first ran it yesterday even without configure those queues in JBOSS.

I think except those classes I mentioned yesterday (which you said they should have been removed), there is some class still don't have source code, those are under com.ssn.jms.integration.test.runner package. Especially the SimpleStringThreadRunner, which I believe it the class that produces messages.

Can you help? Thanks. I'll continue to study the client code to see if i did something wrong.


> Messages stuck in being-delivered state in cluster
> --------------------------------------------------
>
>                 Key: JBMESSAGING-1456
>                 URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1456
>             Project: JBoss Messaging
>          Issue Type: Bug
>    Affects Versions: 1.4.0.SP3_CP03
>            Reporter: Justin Bertram
>            Assignee: Howard Gao
>            Priority: Critical
>         Attachments: kill3_thread_dump.txt, thread_dump.txt
>
>
> Messages become "stuck" in being-delivered state when clients use a clustered XA connection factory in a cluster of at least 2 nodes.
> JBoss setup:
>   -2 nodes of JBoss EAP 4.3 CP02
>   -commented out "ClusterPullConnectionFactory" in messaging-service.xml to prevent message redistribution and eliminate the "message suckers" as the potential culprit
>   -MySQL backend using the default mysql-persistence-service.xml (from <JBOSS_HOME>/docs/examples/jms)
> Client setup:
>   -both nodes have a client which is a separate process (i.e. not inside JBoss)
>   -clients are Spring based
>   -one client produces and consumes, the other client just consumes
>   -both clients use the ClusteredXAConnectionFactory from the default connection-factories-service.xml
>   -both clients publish to and consume from "queue/testDistributedQueue"
>   -clients are configured to send persistent messages, use AUTO_ACKNOWLEDGE, and transacted sessions
> Symptoms of the issue:
>   -when running the clients I watch the JMX-Console for the "queue/testDistributedQueue"
>   -as the consumers pull messages off the queue I can see the MessageCount and DeliveringCount go to 0 every so often
>   -after a period of time (usually a few hours) the MessageCount and DeliveringCount never go back to 0
>   -I "kill" the clients and wait for the DeliveringCount to go to 0, but it never does
>   -after the clients are killed the ConsumerCount for the queue will drop, but never to 0 when messages are "stuck"
>   -a thread dump reveals at least one JBM server session that is apparently stuck (it never goes away) - ostensibly this is the consumer that is showing in the JMX-Console for "queue/testDistributedQueue"
>   -a "killall -3 java" doesn't produce anything from the clients so I know their dead
>   -nothing is in any DLQ or expiry queue
>   -the database contains as many rows in the JBM_MSG and JBM_MSG_REF tables as the DeliveringCount in the JMX-Console
>   -rebooting the node with the stuck messages frees the messages to be consumed (i.e. un-sticks them)
> Other notes:
>   -nothing else is happening on either node but running the client and running JBoss
>   -this only appears to happen when a clustered connection factory is used.  I tested using a normal connection factory and after 24 hours couldn't reproduce a stuck message.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira