[jboss-jira] [JBoss JIRA] Commented: (JBMESSAGING-1822) MessageSucker failures cause the delivery of the failed message to stall

[Yong Hao Gao (JIRA)]

    [ https://issues.jboss.org/browse/JBMESSAGING-1822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12572344#comment-12572344 ] 

Yong Hao Gao commented on JBMESSAGING-1822:
-------------------------------------------


1. JDBCPersistenceManager.claimMessagesInSuck(final long channelID) uses two sql separate statements in a transaction, the first is a sql query and the second is an update. As JBM's DB transaction isolation level is READ_COMMITTED,
the two sql executions can result in inconsistent state. 

Fix: use the result from the first query to do the update.

2. In ServerSessionEndpoint.localClose(boolean isFromFailure), when the messages has been claimed back to the channel, the channel is not prompted for immediate delivery, this could result in those messages stuck in the queue if there is no other events that triggers the delivery (like more messages being added, consumer changes etc).

Fix: make the channel do an immediate delivery after such messages has been claimed.



> MessageSucker failures cause the delivery of the failed message to stall
> ------------------------------------------------------------------------
>
>                 Key: JBMESSAGING-1822
>                 URL: https://issues.jboss.org/browse/JBMESSAGING-1822
>             Project: JBoss Messaging
>          Issue Type: Bug
>          Components: Messaging Core
>    Affects Versions: 1.4.0.SP3.CP10, 1.4.6.GA, 1.4.6.GA.SP1, 1.4.7.GA
>            Reporter: david.boeren
>            Assignee: Yong Hao Gao
>             Fix For: 1.4.0.SP3.CP12, 1.4.8.GA
>
>         Attachments: helloworld.zip
>
>
> The MessageSucker is responsible for migrating messages between different members of a cluster, it is a consumer to the remote queue from which it receives messages destined for the queue on the local cluster member. 
> The onMessage routine, at its most basic, does the following 
> - bookkeeping for the incoming message, including expiry 
> - acknowledge the incoming message 
> - attempt to deliver to the local queue 
> When the delivery fails, the result is the *appearance* of lost messages. Those messages which are processed during the failure are not redelivered, but they still exist in the database. 
> The only way I have found to trigger the redelivery of those messages is to redeploy the queue containing the messages and/or restart that app server. Obviously neither approach is acceptable. 
> In order to trigger the error I created a SOA cluster which *only* shared the JMS database, and no other. I modified the helloworld quickstart to display a counter of messages consumed, clustered the *esb* queue, and then used byteman to trigger the faults. 
> The byteman rule is as follows, the quickstart will be attached. 
> RULE throw every fifth send 
> INTERFACE ProducerDelegate 
> METHOD send 
> AT ENTRY 
> IF callerEquals("MessageSucker.onMessage", true) && (incrementCounter("throwException") % 5 == 0) 
> DO THROW new IllegalStateException("Deliberate exception") 
> ENDRULE 
> This results in an exception being thrown for every fifth message. Once the delivery has quiesced, examine the JBM_MSG and JBM_MSG_REF tables to see the messages which have not been delivered. 
> The clusters are ports-default and ports-01, the client seeds the gateway by sending 300 messages to the default. 
> Adding up the counter from each server *plus* the message count from JBM_MSG results in 300 (or multiples thereof for more executions).

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira