[JBoss JIRA] Created: (JBMESSAGING-1822) MessageSucker failures cause the delivery of the failed message to stall
10:47 a.m.
MessageSucker failures cause the delivery of the failed message to stall
------------------------------------------------------------------------
Key: JBMESSAGING-1822
URL: https://jira.jboss.org/browse/JBMESSAGING-1822
Project: JBoss Messaging
Issue Type: Bug
Components: Messaging Core
Affects Versions: 1.4.6.GA
Reporter: david.boeren
Fix For: Unscheduled
Attachments: helloworld.zip
The MessageSucker is responsible for migrating messages between different members of a
cluster, it is a consumer to the remote queue from which it receives messages destined for
the queue on the local cluster member.
The onMessage routine, at its most basic, does the following
- bookkeeping for the incoming message, including expiry
- acknowledge the incoming message
- attempt to deliver to the local queue
When the delivery fails, the result is the *appearance* of lost messages. Those messages
which are processed during the failure are not redelivered, but they still exist in the
database.
The only way I have found to trigger the redelivery of those messages is to redeploy the
queue containing the messages and/or restart that app server. Obviously neither approach
is acceptable.
In order to trigger the error I created a SOA cluster which *only* shared the JMS
database, and no other. I modified the helloworld quickstart to display a counter of
messages consumed, clustered the *esb* queue, and then used byteman to trigger the faults.
The byteman rule is as follows, the quickstart will be attached.
RULE throw every fifth send
INTERFACE ProducerDelegate
METHOD send
AT ENTRY
IF callerEquals("MessageSucker.onMessage", true) &&
(incrementCounter("throwException") % 5 == 0)
DO THROW new IllegalStateException("Deliberate exception")
ENDRULE
This results in an exception being thrown for every fifth message. Once the delivery has
quiesced, examine the JBM_MSG and JBM_MSG_REF tables to see the messages which have not
been delivered.
The clusters are ports-default and ports-01, the client seeds the gateway by sending 300
messages to the default.
Adding up the counter from each server *plus* the message count from JBM_MSG results in
300 (or multiples thereof for more executions).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:47 a.m.
New subject: [JBoss JIRA] Updated: (JBMESSAGING-1822) MessageSucker failures cause the delivery of the failed message to stall
[
https://jira.jboss.org/browse/JBMESSAGING-1822?page=com.atlassian.jira.pl...
]
david.boeren updated JBMESSAGING-1822:
--------------------------------------
Attachment: helloworld.zip
MessageSucker failures cause the delivery of the failed message to
stall
------------------------------------------------------------------------
Key: JBMESSAGING-1822
URL: https://jira.jboss.org/browse/JBMESSAGING-1822
Project: JBoss Messaging
Issue Type: Bug
Components: Messaging Core
Affects Versions: 1.4.6.GA
Reporter: david.boeren
Fix For: Unscheduled
Attachments: helloworld.zip
The MessageSucker is responsible for migrating messages between different members of a
cluster, it is a consumer to the remote queue from which it receives messages destined for
the queue on the local cluster member.
The onMessage routine, at its most basic, does the following
- bookkeeping for the incoming message, including expiry
- acknowledge the incoming message
- attempt to deliver to the local queue
When the delivery fails, the result is the *appearance* of lost messages. Those messages
which are processed during the failure are not redelivered, but they still exist in the
database.
The only way I have found to trigger the redelivery of those messages is to redeploy the
queue containing the messages and/or restart that app server. Obviously neither approach
is acceptable.
In order to trigger the error I created a SOA cluster which *only* shared the JMS
database, and no other. I modified the helloworld quickstart to display a counter of
messages consumed, clustered the *esb* queue, and then used byteman to trigger the faults.
The byteman rule is as follows, the quickstart will be attached.
RULE throw every fifth send
INTERFACE ProducerDelegate
METHOD send
AT ENTRY
IF callerEquals("MessageSucker.onMessage", true) &&
(incrementCounter("throwException") % 5 == 0)
DO THROW new IllegalStateException("Deliberate exception")
ENDRULE
This results in an exception being thrown for every fifth message. Once the delivery has
quiesced, examine the JBM_MSG and JBM_MSG_REF tables to see the messages which have not
been delivered.
The clusters are ports-default and ports-01, the client seeds the gateway by sending 300
messages to the default.
Adding up the counter from each server *plus* the message count from JBM_MSG results in
300 (or multiples thereof for more executions).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:06 p.m.
New subject: [JBoss JIRA] Assigned: (JBMESSAGING-1822) MessageSucker failures cause the delivery of the failed message to stall
[
https://jira.jboss.org/browse/JBMESSAGING-1822?page=com.atlassian.jira.pl...
]
Justin Bertram reassigned JBMESSAGING-1822:
-------------------------------------------
Assignee: Yong Hao Gao
MessageSucker failures cause the delivery of the failed message to
stall
------------------------------------------------------------------------
Key: JBMESSAGING-1822
URL: https://jira.jboss.org/browse/JBMESSAGING-1822
Project: JBoss Messaging
Issue Type: Bug
Components: Messaging Core
Affects Versions: 1.4.6.GA
Reporter: david.boeren
Assignee: Yong Hao Gao
Fix For: Unscheduled
Attachments: helloworld.zip
The MessageSucker is responsible for migrating messages between different members of a
cluster, it is a consumer to the remote queue from which it receives messages destined for
the queue on the local cluster member.
The onMessage routine, at its most basic, does the following
- bookkeeping for the incoming message, including expiry
- acknowledge the incoming message
- attempt to deliver to the local queue
When the delivery fails, the result is the *appearance* of lost messages. Those messages
which are processed during the failure are not redelivered, but they still exist in the
database.
The only way I have found to trigger the redelivery of those messages is to redeploy the
queue containing the messages and/or restart that app server. Obviously neither approach
is acceptable.
In order to trigger the error I created a SOA cluster which *only* shared the JMS
database, and no other. I modified the helloworld quickstart to display a counter of
messages consumed, clustered the *esb* queue, and then used byteman to trigger the faults.
The byteman rule is as follows, the quickstart will be attached.
RULE throw every fifth send
INTERFACE ProducerDelegate
METHOD send
AT ENTRY
IF callerEquals("MessageSucker.onMessage", true) &&
(incrementCounter("throwException") % 5 == 0)
DO THROW new IllegalStateException("Deliberate exception")
ENDRULE
This results in an exception being thrown for every fifth message. Once the delivery has
quiesced, examine the JBM_MSG and JBM_MSG_REF tables to see the messages which have not
been delivered.
The clusters are ports-default and ports-01, the client seeds the gateway by sending 300
messages to the default.
Adding up the counter from each server *plus* the message count from JBM_MSG results in
300 (or multiples thereof for more executions).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:09 p.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1822) MessageSucker failures cause the delivery of the failed message to stall
[
https://jira.jboss.org/browse/JBMESSAGING-1822?page=com.atlassian.jira.pl...
]
Justin Bertram commented on JBMESSAGING-1822:
---------------------------------------------
Howard, I will certainly defer to your implementation knowledge, but it seems to me that
the *safest* thing for the MessageSucker to do is become a normal JMS client and use XA to
consume from the remote destination and send to the local one (much like the JMS bridge
does when using QualityOfServiceMode 2). Of course, there will be a performance penalty
for this, but as I said it seems to be the safest route.
MessageSucker failures cause the delivery of the failed message to
stall
------------------------------------------------------------------------
Key: JBMESSAGING-1822
URL: https://jira.jboss.org/browse/JBMESSAGING-1822
Project: JBoss Messaging
Issue Type: Bug
Components: Messaging Core
Affects Versions: 1.4.6.GA
Reporter: david.boeren
Assignee: Yong Hao Gao
Fix For: Unscheduled
Attachments: helloworld.zip
The MessageSucker is responsible for migrating messages between different members of a
cluster, it is a consumer to the remote queue from which it receives messages destined for
the queue on the local cluster member.
The onMessage routine, at its most basic, does the following
- bookkeeping for the incoming message, including expiry
- acknowledge the incoming message
- attempt to deliver to the local queue
When the delivery fails, the result is the *appearance* of lost messages. Those messages
which are processed during the failure are not redelivered, but they still exist in the
database.
The only way I have found to trigger the redelivery of those messages is to redeploy the
queue containing the messages and/or restart that app server. Obviously neither approach
is acceptable.
In order to trigger the error I created a SOA cluster which *only* shared the JMS
database, and no other. I modified the helloworld quickstart to display a counter of
messages consumed, clustered the *esb* queue, and then used byteman to trigger the faults.
The byteman rule is as follows, the quickstart will be attached.
RULE throw every fifth send
INTERFACE ProducerDelegate
METHOD send
AT ENTRY
IF callerEquals("MessageSucker.onMessage", true) &&
(incrementCounter("throwException") % 5 == 0)
DO THROW new IllegalStateException("Deliberate exception")
ENDRULE
This results in an exception being thrown for every fifth message. Once the delivery has
quiesced, examine the JBM_MSG and JBM_MSG_REF tables to see the messages which have not
been delivered.
The clusters are ports-default and ports-01, the client seeds the gateway by sending 300
messages to the default.
Adding up the counter from each server *plus* the message count from JBM_MSG results in
300 (or multiples thereof for more executions).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:42 a.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1822) MessageSucker failures cause the delivery of the failed message to stall
[
https://jira.jboss.org/browse/JBMESSAGING-1822?page=com.atlassian.jira.pl...
]
Yong Hao Gao commented on JBMESSAGING-1822:
-------------------------------------------
Hi Justin,
I agree with you that using XA will solve this problem. I'll do some investigation and
see if there is some other alternative to this problem that need minimum code change to
existing sucker implementation. Thanks.
Howard
MessageSucker failures cause the delivery of the failed message to
stall
------------------------------------------------------------------------
Key: JBMESSAGING-1822
URL: https://jira.jboss.org/browse/JBMESSAGING-1822
Project: JBoss Messaging
Issue Type: Bug
Components: Messaging Core
Affects Versions: 1.4.6.GA
Reporter: david.boeren
Assignee: Yong Hao Gao
Fix For: Unscheduled
Attachments: helloworld.zip
The MessageSucker is responsible for migrating messages between different members of a
cluster, it is a consumer to the remote queue from which it receives messages destined for
the queue on the local cluster member.
The onMessage routine, at its most basic, does the following
- bookkeeping for the incoming message, including expiry
- acknowledge the incoming message
- attempt to deliver to the local queue
When the delivery fails, the result is the *appearance* of lost messages. Those messages
which are processed during the failure are not redelivered, but they still exist in the
database.
The only way I have found to trigger the redelivery of those messages is to redeploy the
queue containing the messages and/or restart that app server. Obviously neither approach
is acceptable.
In order to trigger the error I created a SOA cluster which *only* shared the JMS
database, and no other. I modified the helloworld quickstart to display a counter of
messages consumed, clustered the *esb* queue, and then used byteman to trigger the faults.
The byteman rule is as follows, the quickstart will be attached.
RULE throw every fifth send
INTERFACE ProducerDelegate
METHOD send
AT ENTRY
IF callerEquals("MessageSucker.onMessage", true) &&
(incrementCounter("throwException") % 5 == 0)
DO THROW new IllegalStateException("Deliberate exception")
ENDRULE
This results in an exception being thrown for every fifth message. Once the delivery has
quiesced, examine the JBM_MSG and JBM_MSG_REF tables to see the messages which have not
been delivered.
The clusters are ports-default and ports-01, the client seeds the gateway by sending 300
messages to the default.
Adding up the counter from each server *plus* the message count from JBM_MSG results in
300 (or multiples thereof for more executions).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:48 a.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1822) MessageSucker failures cause the delivery of the failed message to stall
[
https://jira.jboss.org/browse/JBMESSAGING-1822?page=com.atlassian.jira.pl...
]
Tim Fox commented on JBMESSAGING-1822:
--------------------------------------
Couldn't resist chiming in here.
The first question is, why would the call to send the message from inside the onMessage of
the message sucker fail? Can we have a concrete example of this, that doesn't use
byteman?
If it does fail probably the message sucker just needs to close it's session and
recreate it to force redelivery.
Regarding XA, this is unnecessary, since with JBM all nodes share the same DB, so
there's really only one transactional resource. You'll notice the actual update in
the database from queue1-->queue2 as the message is transferred is done in a single non
XA transaction.
MessageSucker failures cause the delivery of the failed message to
stall
------------------------------------------------------------------------
Key: JBMESSAGING-1822
URL: https://jira.jboss.org/browse/JBMESSAGING-1822
Project: JBoss Messaging
Issue Type: Bug
Components: Messaging Core
Affects Versions: 1.4.6.GA
Reporter: david.boeren
Assignee: Yong Hao Gao
Fix For: Unscheduled
Attachments: helloworld.zip
The MessageSucker is responsible for migrating messages between different members of a
cluster, it is a consumer to the remote queue from which it receives messages destined for
the queue on the local cluster member.
The onMessage routine, at its most basic, does the following
- bookkeeping for the incoming message, including expiry
- acknowledge the incoming message
- attempt to deliver to the local queue
When the delivery fails, the result is the *appearance* of lost messages. Those messages
which are processed during the failure are not redelivered, but they still exist in the
database.
The only way I have found to trigger the redelivery of those messages is to redeploy the
queue containing the messages and/or restart that app server. Obviously neither approach
is acceptable.
In order to trigger the error I created a SOA cluster which *only* shared the JMS
database, and no other. I modified the helloworld quickstart to display a counter of
messages consumed, clustered the *esb* queue, and then used byteman to trigger the faults.
The byteman rule is as follows, the quickstart will be attached.
RULE throw every fifth send
INTERFACE ProducerDelegate
METHOD send
AT ENTRY
IF callerEquals("MessageSucker.onMessage", true) &&
(incrementCounter("throwException") % 5 == 0)
DO THROW new IllegalStateException("Deliberate exception")
ENDRULE
This results in an exception being thrown for every fifth message. Once the delivery has
quiesced, examine the JBM_MSG and JBM_MSG_REF tables to see the messages which have not
been delivered.
The clusters are ports-default and ports-01, the client seeds the gateway by sending 300
messages to the default.
Adding up the counter from each server *plus* the message count from JBM_MSG results in
300 (or multiples thereof for more executions).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:56 a.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1822) MessageSucker failures cause the delivery of the failed message to stall
[
https://jira.jboss.org/browse/JBMESSAGING-1822?page=com.atlassian.jira.pl...
]
Yong Hao Gao commented on JBMESSAGING-1822:
-------------------------------------------
I'm thinking a local transaction may suffice.
MessageSucker failures cause the delivery of the failed message to
stall
------------------------------------------------------------------------
Key: JBMESSAGING-1822
URL: https://jira.jboss.org/browse/JBMESSAGING-1822
Project: JBoss Messaging
Issue Type: Bug
Components: Messaging Core
Affects Versions: 1.4.6.GA
Reporter: david.boeren
Assignee: Yong Hao Gao
Fix For: Unscheduled
Attachments: helloworld.zip
The MessageSucker is responsible for migrating messages between different members of a
cluster, it is a consumer to the remote queue from which it receives messages destined for
the queue on the local cluster member.
The onMessage routine, at its most basic, does the following
- bookkeeping for the incoming message, including expiry
- acknowledge the incoming message
- attempt to deliver to the local queue
When the delivery fails, the result is the *appearance* of lost messages. Those messages
which are processed during the failure are not redelivered, but they still exist in the
database.
The only way I have found to trigger the redelivery of those messages is to redeploy the
queue containing the messages and/or restart that app server. Obviously neither approach
is acceptable.
In order to trigger the error I created a SOA cluster which *only* shared the JMS
database, and no other. I modified the helloworld quickstart to display a counter of
messages consumed, clustered the *esb* queue, and then used byteman to trigger the faults.
The byteman rule is as follows, the quickstart will be attached.
RULE throw every fifth send
INTERFACE ProducerDelegate
METHOD send
AT ENTRY
IF callerEquals("MessageSucker.onMessage", true) &&
(incrementCounter("throwException") % 5 == 0)
DO THROW new IllegalStateException("Deliberate exception")
ENDRULE
This results in an exception being thrown for every fifth message. Once the delivery has
quiesced, examine the JBM_MSG and JBM_MSG_REF tables to see the messages which have not
been delivered.
The clusters are ports-default and ports-01, the client seeds the gateway by sending 300
messages to the default.
Adding up the counter from each server *plus* the message count from JBM_MSG results in
300 (or multiples thereof for more executions).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:56 a.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1822) MessageSucker failures cause the delivery of the failed message to stall
[
https://jira.jboss.org/browse/JBMESSAGING-1822?page=com.atlassian.jira.pl...
]
Yong Hao Gao commented on JBMESSAGING-1822:
-------------------------------------------
Thanks Tim, just saw your comment. :)
MessageSucker failures cause the delivery of the failed message to
stall
------------------------------------------------------------------------
Key: JBMESSAGING-1822
URL: https://jira.jboss.org/browse/JBMESSAGING-1822
Project: JBoss Messaging
Issue Type: Bug
Components: Messaging Core
Affects Versions: 1.4.6.GA
Reporter: david.boeren
Assignee: Yong Hao Gao
Fix For: Unscheduled
Attachments: helloworld.zip
The MessageSucker is responsible for migrating messages between different members of a
cluster, it is a consumer to the remote queue from which it receives messages destined for
the queue on the local cluster member.
The onMessage routine, at its most basic, does the following
- bookkeeping for the incoming message, including expiry
- acknowledge the incoming message
- attempt to deliver to the local queue
When the delivery fails, the result is the *appearance* of lost messages. Those messages
which are processed during the failure are not redelivered, but they still exist in the
database.
The only way I have found to trigger the redelivery of those messages is to redeploy the
queue containing the messages and/or restart that app server. Obviously neither approach
is acceptable.
In order to trigger the error I created a SOA cluster which *only* shared the JMS
database, and no other. I modified the helloworld quickstart to display a counter of
messages consumed, clustered the *esb* queue, and then used byteman to trigger the faults.
The byteman rule is as follows, the quickstart will be attached.
RULE throw every fifth send
INTERFACE ProducerDelegate
METHOD send
AT ENTRY
IF callerEquals("MessageSucker.onMessage", true) &&
(incrementCounter("throwException") % 5 == 0)
DO THROW new IllegalStateException("Deliberate exception")
ENDRULE
This results in an exception being thrown for every fifth message. Once the delivery has
quiesced, examine the JBM_MSG and JBM_MSG_REF tables to see the messages which have not
been delivered.
The clusters are ports-default and ports-01, the client seeds the gateway by sending 300
messages to the default.
Adding up the counter from each server *plus* the message count from JBM_MSG results in
300 (or multiples thereof for more executions).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2 a.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1822) MessageSucker failures cause the delivery of the failed message to stall
[
https://jira.jboss.org/browse/JBMESSAGING-1822?page=com.atlassian.jira.pl...
]
Yong Hao Gao commented on JBMESSAGING-1822:
-------------------------------------------
Tim,
Regarding what this byteman (I don't know what's it yet) simulates, I saw the user
mentioned normally brining down JBM server several times during the test, that may cause
the sucker connection is closed (during shutdown process) when the sucker tries to send a
message to local queue.
Howard
MessageSucker failures cause the delivery of the failed message to
stall
------------------------------------------------------------------------
Key: JBMESSAGING-1822
URL: https://jira.jboss.org/browse/JBMESSAGING-1822
Project: JBoss Messaging
Issue Type: Bug
Components: Messaging Core
Affects Versions: 1.4.6.GA
Reporter: david.boeren
Assignee: Yong Hao Gao
Fix For: Unscheduled
Attachments: helloworld.zip
The MessageSucker is responsible for migrating messages between different members of a
cluster, it is a consumer to the remote queue from which it receives messages destined for
the queue on the local cluster member.
The onMessage routine, at its most basic, does the following
- bookkeeping for the incoming message, including expiry
- acknowledge the incoming message
- attempt to deliver to the local queue
When the delivery fails, the result is the *appearance* of lost messages. Those messages
which are processed during the failure are not redelivered, but they still exist in the
database.
The only way I have found to trigger the redelivery of those messages is to redeploy the
queue containing the messages and/or restart that app server. Obviously neither approach
is acceptable.
In order to trigger the error I created a SOA cluster which *only* shared the JMS
database, and no other. I modified the helloworld quickstart to display a counter of
messages consumed, clustered the *esb* queue, and then used byteman to trigger the faults.
The byteman rule is as follows, the quickstart will be attached.
RULE throw every fifth send
INTERFACE ProducerDelegate
METHOD send
AT ENTRY
IF callerEquals("MessageSucker.onMessage", true) &&
(incrementCounter("throwException") % 5 == 0)
DO THROW new IllegalStateException("Deliberate exception")
ENDRULE
This results in an exception being thrown for every fifth message. Once the delivery has
quiesced, examine the JBM_MSG and JBM_MSG_REF tables to see the messages which have not
been delivered.
The clusters are ports-default and ports-01, the client seeds the gateway by sending 300
messages to the default.
Adding up the counter from each server *plus* the message count from JBM_MSG results in
300 (or multiples thereof for more executions).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2 a.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1822) MessageSucker failures cause the delivery of the failed message to stall
[
https://jira.jboss.org/browse/JBMESSAGING-1822?page=com.atlassian.jira.pl...
]
Tim Fox commented on JBMESSAGING-1822:
--------------------------------------
It should already be using a local transaction. At least, that's how the code *used*
to be last time I looked at it (wrote it), but it seems things have been changed since
then.
Transaction tx = null;
Looking at the code as it is now, the transaction is always null (!). If the transaction
was created properly (like it used to be), then rolling back should solve the issue.
MessageSucker failures cause the delivery of the failed message to
stall
------------------------------------------------------------------------
Key: JBMESSAGING-1822
URL: https://jira.jboss.org/browse/JBMESSAGING-1822
Project: JBoss Messaging
Issue Type: Bug
Components: Messaging Core
Affects Versions: 1.4.6.GA
Reporter: david.boeren
Assignee: Yong Hao Gao
Fix For: Unscheduled
Attachments: helloworld.zip
The MessageSucker is responsible for migrating messages between different members of a
cluster, it is a consumer to the remote queue from which it receives messages destined for
the queue on the local cluster member.
The onMessage routine, at its most basic, does the following
- bookkeeping for the incoming message, including expiry
- acknowledge the incoming message
- attempt to deliver to the local queue
When the delivery fails, the result is the *appearance* of lost messages. Those messages
which are processed during the failure are not redelivered, but they still exist in the
database.
The only way I have found to trigger the redelivery of those messages is to redeploy the
queue containing the messages and/or restart that app server. Obviously neither approach
is acceptable.
In order to trigger the error I created a SOA cluster which *only* shared the JMS
database, and no other. I modified the helloworld quickstart to display a counter of
messages consumed, clustered the *esb* queue, and then used byteman to trigger the faults.
The byteman rule is as follows, the quickstart will be attached.
RULE throw every fifth send
INTERFACE ProducerDelegate
METHOD send
AT ENTRY
IF callerEquals("MessageSucker.onMessage", true) &&
(incrementCounter("throwException") % 5 == 0)
DO THROW new IllegalStateException("Deliberate exception")
ENDRULE
This results in an exception being thrown for every fifth message. Once the delivery has
quiesced, examine the JBM_MSG and JBM_MSG_REF tables to see the messages which have not
been delivered.
The clusters are ports-default and ports-01, the client seeds the gateway by sending 300
messages to the default.
Adding up the counter from each server *plus* the message count from JBM_MSG results in
300 (or multiples thereof for more executions).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:02 a.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1822) MessageSucker failures cause the delivery of the failed message to stall
[
https://jira.jboss.org/browse/JBMESSAGING-1822?page=com.atlassian.jira.pl...
]
Tim Fox commented on JBMESSAGING-1822:
--------------------------------------
Actually, even a local tx should be unnecessary.
If you use a client_acknowledge session and just call sesssion.recover() if the send
fails, then any unacked msgs will be redelivered.
MessageSucker failures cause the delivery of the failed message to
stall
------------------------------------------------------------------------
Key: JBMESSAGING-1822
URL: https://jira.jboss.org/browse/JBMESSAGING-1822
Project: JBoss Messaging
Issue Type: Bug
Components: Messaging Core
Affects Versions: 1.4.6.GA
Reporter: david.boeren
Assignee: Yong Hao Gao
Fix For: Unscheduled
Attachments: helloworld.zip
The MessageSucker is responsible for migrating messages between different members of a
cluster, it is a consumer to the remote queue from which it receives messages destined for
the queue on the local cluster member.
The onMessage routine, at its most basic, does the following
- bookkeeping for the incoming message, including expiry
- acknowledge the incoming message
- attempt to deliver to the local queue
When the delivery fails, the result is the *appearance* of lost messages. Those messages
which are processed during the failure are not redelivered, but they still exist in the
database.
The only way I have found to trigger the redelivery of those messages is to redeploy the
queue containing the messages and/or restart that app server. Obviously neither approach
is acceptable.
In order to trigger the error I created a SOA cluster which *only* shared the JMS
database, and no other. I modified the helloworld quickstart to display a counter of
messages consumed, clustered the *esb* queue, and then used byteman to trigger the faults.
The byteman rule is as follows, the quickstart will be attached.
RULE throw every fifth send
INTERFACE ProducerDelegate
METHOD send
AT ENTRY
IF callerEquals("MessageSucker.onMessage", true) &&
(incrementCounter("throwException") % 5 == 0)
DO THROW new IllegalStateException("Deliberate exception")
ENDRULE
This results in an exception being thrown for every fifth message. Once the delivery has
quiesced, examine the JBM_MSG and JBM_MSG_REF tables to see the messages which have not
been delivered.
The clusters are ports-default and ports-01, the client seeds the gateway by sending 300
messages to the default.
Adding up the counter from each server *plus* the message count from JBM_MSG results in
300 (or multiples thereof for more executions).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:07 a.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1822) MessageSucker failures cause the delivery of the failed message to stall
[
https://jira.jboss.org/browse/JBMESSAGING-1822?page=com.atlassian.jira.pl...
]
Yong Hao Gao commented on JBMESSAGING-1822:
-------------------------------------------
"If you use a client_acknowledge session and just call sesssion.recover() if the send
fails, then any unacked msgs will be redelivered. "
That's great idea. Thanks.
MessageSucker failures cause the delivery of the failed message to
stall
------------------------------------------------------------------------
Key: JBMESSAGING-1822
URL: https://jira.jboss.org/browse/JBMESSAGING-1822
Project: JBoss Messaging
Issue Type: Bug
Components: Messaging Core
Affects Versions: 1.4.6.GA
Reporter: david.boeren
Assignee: Yong Hao Gao
Fix For: Unscheduled
Attachments: helloworld.zip
The MessageSucker is responsible for migrating messages between different members of a
cluster, it is a consumer to the remote queue from which it receives messages destined for
the queue on the local cluster member.
The onMessage routine, at its most basic, does the following
- bookkeeping for the incoming message, including expiry
- acknowledge the incoming message
- attempt to deliver to the local queue
When the delivery fails, the result is the *appearance* of lost messages. Those messages
which are processed during the failure are not redelivered, but they still exist in the
database.
The only way I have found to trigger the redelivery of those messages is to redeploy the
queue containing the messages and/or restart that app server. Obviously neither approach
is acceptable.
In order to trigger the error I created a SOA cluster which *only* shared the JMS
database, and no other. I modified the helloworld quickstart to display a counter of
messages consumed, clustered the *esb* queue, and then used byteman to trigger the faults.
The byteman rule is as follows, the quickstart will be attached.
RULE throw every fifth send
INTERFACE ProducerDelegate
METHOD send
AT ENTRY
IF callerEquals("MessageSucker.onMessage", true) &&
(incrementCounter("throwException") % 5 == 0)
DO THROW new IllegalStateException("Deliberate exception")
ENDRULE
This results in an exception being thrown for every fifth message. Once the delivery has
quiesced, examine the JBM_MSG and JBM_MSG_REF tables to see the messages which have not
been delivered.
The clusters are ports-default and ports-01, the client seeds the gateway by sending 300
messages to the default.
Adding up the counter from each server *plus* the message count from JBM_MSG results in
300 (or multiples thereof for more executions).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:23 a.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1822) MessageSucker failures cause the delivery of the failed message to stall
[
https://jira.jboss.org/browse/JBMESSAGING-1822?page=com.atlassian.jira.pl...
]
Yong Hao Gao commented on JBMESSAGING-1822:
-------------------------------------------
I'm thinking if we use client_ack method, we may need duplicate detection in case the
failure happens between a successful send and the acknowledgement.
Howard
MessageSucker failures cause the delivery of the failed message to
stall
------------------------------------------------------------------------
Key: JBMESSAGING-1822
URL: https://jira.jboss.org/browse/JBMESSAGING-1822
Project: JBoss Messaging
Issue Type: Bug
Components: Messaging Core
Affects Versions: 1.4.6.GA
Reporter: david.boeren
Assignee: Yong Hao Gao
Fix For: Unscheduled
Attachments: helloworld.zip
The MessageSucker is responsible for migrating messages between different members of a
cluster, it is a consumer to the remote queue from which it receives messages destined for
the queue on the local cluster member.
The onMessage routine, at its most basic, does the following
- bookkeeping for the incoming message, including expiry
- acknowledge the incoming message
- attempt to deliver to the local queue
When the delivery fails, the result is the *appearance* of lost messages. Those messages
which are processed during the failure are not redelivered, but they still exist in the
database.
The only way I have found to trigger the redelivery of those messages is to redeploy the
queue containing the messages and/or restart that app server. Obviously neither approach
is acceptable.
In order to trigger the error I created a SOA cluster which *only* shared the JMS
database, and no other. I modified the helloworld quickstart to display a counter of
messages consumed, clustered the *esb* queue, and then used byteman to trigger the faults.
The byteman rule is as follows, the quickstart will be attached.
RULE throw every fifth send
INTERFACE ProducerDelegate
METHOD send
AT ENTRY
IF callerEquals("MessageSucker.onMessage", true) &&
(incrementCounter("throwException") % 5 == 0)
DO THROW new IllegalStateException("Deliberate exception")
ENDRULE
This results in an exception being thrown for every fifth message. Once the delivery has
quiesced, examine the JBM_MSG and JBM_MSG_REF tables to see the messages which have not
been delivered.
The clusters are ports-default and ports-01, the client seeds the gateway by sending 300
messages to the default.
Adding up the counter from each server *plus* the message count from JBM_MSG results in
300 (or multiples thereof for more executions).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:22 a.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1822) MessageSucker failures cause the delivery of the failed message to stall
[
https://jira.jboss.org/browse/JBMESSAGING-1822?page=com.atlassian.jira.pl...
]
Tim Fox commented on JBMESSAGING-1822:
--------------------------------------
You mean if the server crashes between a successful send and an ack?
That doesn't matter since the actual persistent transfer is always done atomically in
the database. So on recovery of that server, the message would either be in the source or
the destination, never both.
MessageSucker failures cause the delivery of the failed message to
stall
------------------------------------------------------------------------
Key: JBMESSAGING-1822
URL: https://jira.jboss.org/browse/JBMESSAGING-1822
Project: JBoss Messaging
Issue Type: Bug
Components: Messaging Core
Affects Versions: 1.4.6.GA
Reporter: david.boeren
Assignee: Yong Hao Gao
Fix For: Unscheduled
Attachments: helloworld.zip
The MessageSucker is responsible for migrating messages between different members of a
cluster, it is a consumer to the remote queue from which it receives messages destined for
the queue on the local cluster member.
The onMessage routine, at its most basic, does the following
- bookkeeping for the incoming message, including expiry
- acknowledge the incoming message
- attempt to deliver to the local queue
When the delivery fails, the result is the *appearance* of lost messages. Those messages
which are processed during the failure are not redelivered, but they still exist in the
database.
The only way I have found to trigger the redelivery of those messages is to redeploy the
queue containing the messages and/or restart that app server. Obviously neither approach
is acceptable.
In order to trigger the error I created a SOA cluster which *only* shared the JMS
database, and no other. I modified the helloworld quickstart to display a counter of
messages consumed, clustered the *esb* queue, and then used byteman to trigger the faults.
The byteman rule is as follows, the quickstart will be attached.
RULE throw every fifth send
INTERFACE ProducerDelegate
METHOD send
AT ENTRY
IF callerEquals("MessageSucker.onMessage", true) &&
(incrementCounter("throwException") % 5 == 0)
DO THROW new IllegalStateException("Deliberate exception")
ENDRULE
This results in an exception being thrown for every fifth message. Once the delivery has
quiesced, examine the JBM_MSG and JBM_MSG_REF tables to see the messages which have not
been delivered.
The clusters are ports-default and ports-01, the client seeds the gateway by sending 300
messages to the default.
Adding up the counter from each server *plus* the message count from JBM_MSG results in
300 (or multiples thereof for more executions).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:58 a.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1822) MessageSucker failures cause the delivery of the failed message to stall
[
https://jira.jboss.org/browse/JBMESSAGING-1822?page=com.atlassian.jira.pl...
]
Yong Hao Gao commented on JBMESSAGING-1822:
-------------------------------------------
Yes the current situation is that we are doing direct update of the sucked message in the
DB, i.e. no deleting of the message when acked and re-insert the message into DB when it
is sent successfully, as would a separate normal receive and normal send do.
I'm thinking the following scenario:
A message M1 is sucked from Node0 to Node1. In Node1 the sucker sends M1 to local queue
but before acknowledging it, Node1 crashed, resulting in M1 being successfully updated for
local queue. Node0 will then receive connection lost failure, M1 is canceled back to local
queue in Node0, and redelivered (to whatever consumer available).
It seems using XA is an easy way to do it.
MessageSucker failures cause the delivery of the failed message to
stall
------------------------------------------------------------------------
Key: JBMESSAGING-1822
URL: https://jira.jboss.org/browse/JBMESSAGING-1822
Project: JBoss Messaging
Issue Type: Bug
Components: Messaging Core
Affects Versions: 1.4.6.GA
Reporter: david.boeren
Assignee: Yong Hao Gao
Fix For: Unscheduled
Attachments: helloworld.zip
The MessageSucker is responsible for migrating messages between different members of a
cluster, it is a consumer to the remote queue from which it receives messages destined for
the queue on the local cluster member.
The onMessage routine, at its most basic, does the following
- bookkeeping for the incoming message, including expiry
- acknowledge the incoming message
- attempt to deliver to the local queue
When the delivery fails, the result is the *appearance* of lost messages. Those messages
which are processed during the failure are not redelivered, but they still exist in the
database.
The only way I have found to trigger the redelivery of those messages is to redeploy the
queue containing the messages and/or restart that app server. Obviously neither approach
is acceptable.
In order to trigger the error I created a SOA cluster which *only* shared the JMS
database, and no other. I modified the helloworld quickstart to display a counter of
messages consumed, clustered the *esb* queue, and then used byteman to trigger the faults.
The byteman rule is as follows, the quickstart will be attached.
RULE throw every fifth send
INTERFACE ProducerDelegate
METHOD send
AT ENTRY
IF callerEquals("MessageSucker.onMessage", true) &&
(incrementCounter("throwException") % 5 == 0)
DO THROW new IllegalStateException("Deliberate exception")
ENDRULE
This results in an exception being thrown for every fifth message. Once the delivery has
quiesced, examine the JBM_MSG and JBM_MSG_REF tables to see the messages which have not
been delivered.
The clusters are ports-default and ports-01, the client seeds the gateway by sending 300
messages to the default.
Adding up the counter from each server *plus* the message count from JBM_MSG results in
300 (or multiples thereof for more executions).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:48 a.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1822) MessageSucker failures cause the delivery of the failed message to stall
[
https://jira.jboss.org/browse/JBMESSAGING-1822?page=com.atlassian.jira.pl...
]
Tim Fox commented on JBMESSAGING-1822:
--------------------------------------
XA will kill performance
MessageSucker failures cause the delivery of the failed message to
stall
------------------------------------------------------------------------
Key: JBMESSAGING-1822
URL: https://jira.jboss.org/browse/JBMESSAGING-1822
Project: JBoss Messaging
Issue Type: Bug
Components: Messaging Core
Affects Versions: 1.4.6.GA
Reporter: david.boeren
Assignee: Yong Hao Gao
Fix For: Unscheduled
Attachments: helloworld.zip
The MessageSucker is responsible for migrating messages between different members of a
cluster, it is a consumer to the remote queue from which it receives messages destined for
the queue on the local cluster member.
The onMessage routine, at its most basic, does the following
- bookkeeping for the incoming message, including expiry
- acknowledge the incoming message
- attempt to deliver to the local queue
When the delivery fails, the result is the *appearance* of lost messages. Those messages
which are processed during the failure are not redelivered, but they still exist in the
database.
The only way I have found to trigger the redelivery of those messages is to redeploy the
queue containing the messages and/or restart that app server. Obviously neither approach
is acceptable.
In order to trigger the error I created a SOA cluster which *only* shared the JMS
database, and no other. I modified the helloworld quickstart to display a counter of
messages consumed, clustered the *esb* queue, and then used byteman to trigger the faults.
The byteman rule is as follows, the quickstart will be attached.
RULE throw every fifth send
INTERFACE ProducerDelegate
METHOD send
AT ENTRY
IF callerEquals("MessageSucker.onMessage", true) &&
(incrementCounter("throwException") % 5 == 0)
DO THROW new IllegalStateException("Deliberate exception")
ENDRULE
This results in an exception being thrown for every fifth message. Once the delivery has
quiesced, examine the JBM_MSG and JBM_MSG_REF tables to see the messages which have not
been delivered.
The clusters are ports-default and ports-01, the client seeds the gateway by sending 300
messages to the default.
Adding up the counter from each server *plus* the message count from JBM_MSG results in
300 (or multiples thereof for more executions).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:50 a.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1822) MessageSucker failures cause the delivery of the failed message to stall
[
https://jira.jboss.org/browse/JBMESSAGING-1822?page=com.atlassian.jira.pl...
]
Tim Fox commented on JBMESSAGING-1822:
--------------------------------------
Also, in the case you describe, if the message got redelivered on node 0, when that
message was acked you'd get an exception since node 0 wouldn't own that message
any more (node 1 owns it in the database), i.e. the ack wouldn't work.
MessageSucker failures cause the delivery of the failed message to
stall
------------------------------------------------------------------------
Key: JBMESSAGING-1822
URL: https://jira.jboss.org/browse/JBMESSAGING-1822
Project: JBoss Messaging
Issue Type: Bug
Components: Messaging Core
Affects Versions: 1.4.6.GA
Reporter: david.boeren
Assignee: Yong Hao Gao
Fix For: Unscheduled
Attachments: helloworld.zip
The MessageSucker is responsible for migrating messages between different members of a
cluster, it is a consumer to the remote queue from which it receives messages destined for
the queue on the local cluster member.
The onMessage routine, at its most basic, does the following
- bookkeeping for the incoming message, including expiry
- acknowledge the incoming message
- attempt to deliver to the local queue
When the delivery fails, the result is the *appearance* of lost messages. Those messages
which are processed during the failure are not redelivered, but they still exist in the
database.
The only way I have found to trigger the redelivery of those messages is to redeploy the
queue containing the messages and/or restart that app server. Obviously neither approach
is acceptable.
In order to trigger the error I created a SOA cluster which *only* shared the JMS
database, and no other. I modified the helloworld quickstart to display a counter of
messages consumed, clustered the *esb* queue, and then used byteman to trigger the faults.
The byteman rule is as follows, the quickstart will be attached.
RULE throw every fifth send
INTERFACE ProducerDelegate
METHOD send
AT ENTRY
IF callerEquals("MessageSucker.onMessage", true) &&
(incrementCounter("throwException") % 5 == 0)
DO THROW new IllegalStateException("Deliberate exception")
ENDRULE
This results in an exception being thrown for every fifth message. Once the delivery has
quiesced, examine the JBM_MSG and JBM_MSG_REF tables to see the messages which have not
been delivered.
The clusters are ports-default and ports-01, the client seeds the gateway by sending 300
messages to the default.
Adding up the counter from each server *plus* the message count from JBM_MSG results in
300 (or multiples thereof for more executions).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:05 p.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1822) MessageSucker failures cause the delivery of the failed message to stall
[
https://jira.jboss.org/browse/JBMESSAGING-1822?page=com.atlassian.jira.pl...
]
Yong Hao Gao commented on JBMESSAGING-1822:
-------------------------------------------
"when that message was acked you'd get an exception since node 0 wouldn't own
that message any more (node 1 owns it in the database), i.e. the ack wouldn't work.
"
That's true for a normal ack. Currently if an ack comes from a sucker this ack
won't delete it from DB as it knows that for a sucked message we already did an
"Update" for it.
MessageSucker failures cause the delivery of the failed message to
stall
------------------------------------------------------------------------
Key: JBMESSAGING-1822
URL: https://jira.jboss.org/browse/JBMESSAGING-1822
Project: JBoss Messaging
Issue Type: Bug
Components: Messaging Core
Affects Versions: 1.4.6.GA
Reporter: david.boeren
Assignee: Yong Hao Gao
Fix For: Unscheduled
Attachments: helloworld.zip
The MessageSucker is responsible for migrating messages between different members of a
cluster, it is a consumer to the remote queue from which it receives messages destined for
the queue on the local cluster member.
The onMessage routine, at its most basic, does the following
- bookkeeping for the incoming message, including expiry
- acknowledge the incoming message
- attempt to deliver to the local queue
When the delivery fails, the result is the *appearance* of lost messages. Those messages
which are processed during the failure are not redelivered, but they still exist in the
database.
The only way I have found to trigger the redelivery of those messages is to redeploy the
queue containing the messages and/or restart that app server. Obviously neither approach
is acceptable.
In order to trigger the error I created a SOA cluster which *only* shared the JMS
database, and no other. I modified the helloworld quickstart to display a counter of
messages consumed, clustered the *esb* queue, and then used byteman to trigger the faults.
The byteman rule is as follows, the quickstart will be attached.
RULE throw every fifth send
INTERFACE ProducerDelegate
METHOD send
AT ENTRY
IF callerEquals("MessageSucker.onMessage", true) &&
(incrementCounter("throwException") % 5 == 0)
DO THROW new IllegalStateException("Deliberate exception")
ENDRULE
This results in an exception being thrown for every fifth message. Once the delivery has
quiesced, examine the JBM_MSG and JBM_MSG_REF tables to see the messages which have not
been delivered.
The clusters are ports-default and ports-01, the client seeds the gateway by sending 300
messages to the default.
Adding up the counter from each server *plus* the message count from JBM_MSG results in
300 (or multiples thereof for more executions).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:33 a.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1822) MessageSucker failures cause the delivery of the failed message to stall
[
https://jira.jboss.org/browse/JBMESSAGING-1822?page=com.atlassian.jira.pl...
]
Yong Hao Gao commented on JBMESSAGING-1822:
-------------------------------------------
As XA is not a preferred way to this issue, I come up with an idea that can hopefully
solve the issue. I give it here for discussion.
1. Introduce a new state in JBM_MSG_REF table's STATE column. The new state
'S' marks that the message is in a special "To be sucked" state.
2. Change the sucking process as the following:
a) When a Message M is ready to be sucked, it is put to the remote consumer
(ServerConsumerEndpoint) for delivery. This remote consumer on accepting M, update the
M's state to 'S'. Then it goes on to actually deliver M to the Message Sucker
on another node.
b) When the message sucker receives M, it first acknowledges it and then sends it to the
local queue.
c) When the acknowledgement arrives at the source node, the session simply forget the
message (without any DB operations).
Failure handling
The sucking process will have to handle the following failures
1. Source node crash. That will leave M in either 'C' (normal state) or
'S' state. If M is in normal state. When the node is restarted, it will be
delivered as normal. If M is already in 'S' state, this message won't be
delivered as normal messages when the node is restarted, no matter whether this M has been
delivered to the Sucker or not. At the target node, if the sucker has received the message
and updated it successfully, M will be in the local queue already. If the sucker has
received the message but failed to update M, leaving M in 'S' state.
When source node comes up again, the sucker will reconnected to the source node and
registers a remote consumer. On starting up of the remoting consumer, it will first check
in DB is there is a message that has state of 'S', if so it picks up this message
and deliver it to the sucker right away.
2. Target node crash. That will leave M in several situations:
1st - M being put to the remote consumer be hasn't been updated to 'S' state.
Then M will eventually be cancelled to queue for re-delivery.
2nd - M being put to the remote consumer and updated to 'S' state, but not yet
delivered. When remote consumer be closed eventually, it will acknowlege it to the Session
to forget it. That prevent the session from redeliver M when it is in 'S' state.
When target node comes up and a new remote consumer registered, the message will be picked
up and delivered (the first thing such a remote consumer does when started).
3rd - M being put to the remote consumer and updated to 'S' state and delivered to
the sucker in target node, but failed to acknowledge it. This is equivalent to 2nd case,
i.e. when target node is back the message M will be redelivered.
4th - M being delivered to target node and acked, but failed to update its state (still in
'S'). Same thing as 3rd, i.e. the M will be picked up when targe node is back and
a remote consumer is registered.
3. Both nodes crash. We just monitor the state of M. When either node starts up again, it
ignores messages marked as 'S' states. Once a sucker is created and registers a
remote consumer, it's first task is to look up the message in 'S' state and
deliver it if any.
MessageSucker failures cause the delivery of the failed message to
stall
------------------------------------------------------------------------
Key: JBMESSAGING-1822
URL: https://jira.jboss.org/browse/JBMESSAGING-1822
Project: JBoss Messaging
Issue Type: Bug
Components: Messaging Core
Affects Versions: 1.4.6.GA
Reporter: david.boeren
Assignee: Yong Hao Gao
Fix For: Unscheduled
Attachments: helloworld.zip
The MessageSucker is responsible for migrating messages between different members of a
cluster, it is a consumer to the remote queue from which it receives messages destined for
the queue on the local cluster member.
The onMessage routine, at its most basic, does the following
- bookkeeping for the incoming message, including expiry
- acknowledge the incoming message
- attempt to deliver to the local queue
When the delivery fails, the result is the *appearance* of lost messages. Those messages
which are processed during the failure are not redelivered, but they still exist in the
database.
The only way I have found to trigger the redelivery of those messages is to redeploy the
queue containing the messages and/or restart that app server. Obviously neither approach
is acceptable.
In order to trigger the error I created a SOA cluster which *only* shared the JMS
database, and no other. I modified the helloworld quickstart to display a counter of
messages consumed, clustered the *esb* queue, and then used byteman to trigger the faults.
The byteman rule is as follows, the quickstart will be attached.
RULE throw every fifth send
INTERFACE ProducerDelegate
METHOD send
AT ENTRY
IF callerEquals("MessageSucker.onMessage", true) &&
(incrementCounter("throwException") % 5 == 0)
DO THROW new IllegalStateException("Deliberate exception")
ENDRULE
This results in an exception being thrown for every fifth message. Once the delivery has
quiesced, examine the JBM_MSG and JBM_MSG_REF tables to see the messages which have not
been delivered.
The clusters are ports-default and ports-01, the client seeds the gateway by sending 300
messages to the default.
Adding up the counter from each server *plus* the message count from JBM_MSG results in
300 (or multiples thereof for more executions).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:33 a.m.
New subject: [JBoss JIRA] Issue Comment Edited: (JBMESSAGING-1822) MessageSucker failures cause the delivery of the failed message to stall
[
https://jira.jboss.org/browse/JBMESSAGING-1822?page=com.atlassian.jira.pl...
]
Yong Hao Gao edited comment on JBMESSAGING-1822 at 10/25/10 11:33 AM:
----------------------------------------------------------------------
As XA is not a preferred way to this issue, I come up with an idea that can hopefully
solve the issue. I give it here for discussion.
1. Introduce a new state in JBM_MSG_REF table's STATE column. The new state
'S' marks that the message is in a special "To be sucked" state.
2. Change the sucking process as the following:
a) When a Message M is ready to be sucked, it is put to the remote consumer
(ServerConsumerEndpoint) for delivery. This remote consumer on accepting M, update the
M's state to 'S'. Then it goes on to actually deliver M to the Message Sucker
on another node.
b) When the message sucker receives M, it first acknowledges it and then sends it to the
local queue, after that the message state changes back to 'C' (normal).
c) When the acknowledgement arrives at the source node, the session simply forget the
message (without any DB operations).
Failure handling
The sucking process will have to handle the following failures
1. Source node crash. That will leave M in either 'C' (normal state) or
'S' state. If M is in normal state. When the node is restarted, it will be
delivered as normal. If M is already in 'S' state, this message won't be
delivered as normal messages when the node is restarted, no matter whether this M has been
delivered to the Sucker or not. At the target node, if the sucker has received the message
and updated it successfully, M will be in the local queue already. If the sucker has
received the message but failed to update M, leaving M in 'S' state.
When source node comes up again, the sucker will reconnected to the source node and
registers a remote consumer. On starting up of the remoting consumer, it will first check
in DB is there is a message that has state of 'S', if so it picks up this message
and deliver it to the sucker right away.
2. Target node crash. That will leave M in several situations:
1st - M being put to the remote consumer be hasn't been updated to 'S' state.
Then M will eventually be cancelled to queue for re-delivery.
2nd - M being put to the remote consumer and updated to 'S' state, but not yet
delivered. When remote consumer be closed eventually, it will acknowlege it to the Session
to forget it. That prevent the session from redeliver M when it is in 'S' state.
When target node comes up and a new remote consumer registered, the message will be picked
up and delivered (the first thing such a remote consumer does when started).
3rd - M being put to the remote consumer and updated to 'S' state and delivered to
the sucker in target node, but failed to acknowledge it. This is equivalent to 2nd case,
i.e. when target node is back the message M will be redelivered.
4th - M being delivered to target node and acked, but failed to update its state (still in
'S'). Same thing as 3rd, i.e. the M will be picked up when targe node is back and
a remote consumer is registered.
3. Both nodes crash. We just monitor the state of M. When either node starts up again, it
ignores messages marked as 'S' states. Once a sucker is created and registers a
remote consumer, it's first task is to look up the message in 'S' state and
deliver it if any.
was (Author: gaohoward):
As XA is not a preferred way to this issue, I come up with an idea that can hopefully
solve the issue. I give it here for discussion.
1. Introduce a new state in JBM_MSG_REF table's STATE column. The new state
'S' marks that the message is in a special "To be sucked" state.
2. Change the sucking process as the following:
a) When a Message M is ready to be sucked, it is put to the remote consumer
(ServerConsumerEndpoint) for delivery. This remote consumer on accepting M, update the
M's state to 'S'. Then it goes on to actually deliver M to the Message Sucker
on another node.
b) When the message sucker receives M, it first acknowledges it and then sends it to the
local queue.
c) When the acknowledgement arrives at the source node, the session simply forget the
message (without any DB operations).
Failure handling
The sucking process will have to handle the following failures
1. Source node crash. That will leave M in either 'C' (normal state) or
'S' state. If M is in normal state. When the node is restarted, it will be
delivered as normal. If M is already in 'S' state, this message won't be
delivered as normal messages when the node is restarted, no matter whether this M has been
delivered to the Sucker or not. At the target node, if the sucker has received the message
and updated it successfully, M will be in the local queue already. If the sucker has
received the message but failed to update M, leaving M in 'S' state.
When source node comes up again, the sucker will reconnected to the source node and
registers a remote consumer. On starting up of the remoting consumer, it will first check
in DB is there is a message that has state of 'S', if so it picks up this message
and deliver it to the sucker right away.
2. Target node crash. That will leave M in several situations:
1st - M being put to the remote consumer be hasn't been updated to 'S' state.
Then M will eventually be cancelled to queue for re-delivery.
2nd - M being put to the remote consumer and updated to 'S' state, but not yet
delivered. When remote consumer be closed eventually, it will acknowlege it to the Session
to forget it. That prevent the session from redeliver M when it is in 'S' state.
When target node comes up and a new remote consumer registered, the message will be picked
up and delivered (the first thing such a remote consumer does when started).
3rd - M being put to the remote consumer and updated to 'S' state and delivered to
the sucker in target node, but failed to acknowledge it. This is equivalent to 2nd case,
i.e. when target node is back the message M will be redelivered.
4th - M being delivered to target node and acked, but failed to update its state (still in
'S'). Same thing as 3rd, i.e. the M will be picked up when targe node is back and
a remote consumer is registered.
3. Both nodes crash. We just monitor the state of M. When either node starts up again, it
ignores messages marked as 'S' states. Once a sucker is created and registers a
remote consumer, it's first task is to look up the message in 'S' state and
deliver it if any.
MessageSucker failures cause the delivery of the failed message to
stall
------------------------------------------------------------------------
Key: JBMESSAGING-1822
URL: https://jira.jboss.org/browse/JBMESSAGING-1822
Project: JBoss Messaging
Issue Type: Bug
Components: Messaging Core
Affects Versions: 1.4.6.GA
Reporter: david.boeren
Assignee: Yong Hao Gao
Fix For: Unscheduled
Attachments: helloworld.zip
The MessageSucker is responsible for migrating messages between different members of a
cluster, it is a consumer to the remote queue from which it receives messages destined for
the queue on the local cluster member.
The onMessage routine, at its most basic, does the following
- bookkeeping for the incoming message, including expiry
- acknowledge the incoming message
- attempt to deliver to the local queue
When the delivery fails, the result is the *appearance* of lost messages. Those messages
which are processed during the failure are not redelivered, but they still exist in the
database.
The only way I have found to trigger the redelivery of those messages is to redeploy the
queue containing the messages and/or restart that app server. Obviously neither approach
is acceptable.
In order to trigger the error I created a SOA cluster which *only* shared the JMS
database, and no other. I modified the helloworld quickstart to display a counter of
messages consumed, clustered the *esb* queue, and then used byteman to trigger the faults.
The byteman rule is as follows, the quickstart will be attached.
RULE throw every fifth send
INTERFACE ProducerDelegate
METHOD send
AT ENTRY
IF callerEquals("MessageSucker.onMessage", true) &&
(incrementCounter("throwException") % 5 == 0)
DO THROW new IllegalStateException("Deliberate exception")
ENDRULE
This results in an exception being thrown for every fifth message. Once the delivery has
quiesced, examine the JBM_MSG and JBM_MSG_REF tables to see the messages which have not
been delivered.
The clusters are ports-default and ports-01, the client seeds the gateway by sending 300
messages to the default.
Adding up the counter from each server *plus* the message count from JBM_MSG results in
300 (or multiples thereof for more executions).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:45 a.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1822) MessageSucker failures cause the delivery of the failed message to stall
[
https://jira.jboss.org/browse/JBMESSAGING-1822?page=com.atlassian.jira.pl...
]
Clebert Suconic commented on JBMESSAGING-1822:
----------------------------------------------
Don't we already have optimistic lock in place here? The update also looking at the
previous values on the field?
In case of a failure during the update, the sucker could just ignore the message.
Having the sucker doing an extra update would create a big performance hit IMO, as you
would need to commit the row as soon as you set the "S" state.
MessageSucker failures cause the delivery of the failed message to
stall
------------------------------------------------------------------------
Key: JBMESSAGING-1822
URL: https://jira.jboss.org/browse/JBMESSAGING-1822
Project: JBoss Messaging
Issue Type: Bug
Components: Messaging Core
Affects Versions: 1.4.6.GA
Reporter: david.boeren
Assignee: Yong Hao Gao
Fix For: Unscheduled
Attachments: helloworld.zip
The MessageSucker is responsible for migrating messages between different members of a
cluster, it is a consumer to the remote queue from which it receives messages destined for
the queue on the local cluster member.
The onMessage routine, at its most basic, does the following
- bookkeeping for the incoming message, including expiry
- acknowledge the incoming message
- attempt to deliver to the local queue
When the delivery fails, the result is the *appearance* of lost messages. Those messages
which are processed during the failure are not redelivered, but they still exist in the
database.
The only way I have found to trigger the redelivery of those messages is to redeploy the
queue containing the messages and/or restart that app server. Obviously neither approach
is acceptable.
In order to trigger the error I created a SOA cluster which *only* shared the JMS
database, and no other. I modified the helloworld quickstart to display a counter of
messages consumed, clustered the *esb* queue, and then used byteman to trigger the faults.
The byteman rule is as follows, the quickstart will be attached.
RULE throw every fifth send
INTERFACE ProducerDelegate
METHOD send
AT ENTRY
IF callerEquals("MessageSucker.onMessage", true) &&
(incrementCounter("throwException") % 5 == 0)
DO THROW new IllegalStateException("Deliberate exception")
ENDRULE
This results in an exception being thrown for every fifth message. Once the delivery has
quiesced, examine the JBM_MSG and JBM_MSG_REF tables to see the messages which have not
been delivered.
The clusters are ports-default and ports-01, the client seeds the gateway by sending 300
messages to the default.
Adding up the counter from each server *plus* the message count from JBM_MSG results in
300 (or multiples thereof for more executions).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:40 p.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1822) MessageSucker failures cause the delivery of the failed message to stall
[
https://jira.jboss.org/browse/JBMESSAGING-1822?page=com.atlassian.jira.pl...
]
Yong Hao Gao commented on JBMESSAGING-1822:
-------------------------------------------
Hi Clebert,
Don't we already have optimistic lock in place here? The update also looking at the
previous values on the field?
-- We do optimization by reducing a normal delivery/sent into one DB update of the
message, from source channel to target channel. But that cannot solve the issues we are
facing, as the JIRA has described.
In case of a failure during the update, the sucker could just ignore the message.
-- When this issue happens, the message is ignored by the sucker but also forget by the
source node. Unless a re-start of the source is performed, this message appears
'stalled' in the queue.
Having the sucker doing an extra update would create a big performance hit IMO, as you
would need to commit the row as soon as you set the "S" state.
-- Agreed. But comparing with XA, this should be less expensive.
MessageSucker failures cause the delivery of the failed message to
stall
------------------------------------------------------------------------
Key: JBMESSAGING-1822
URL: https://jira.jboss.org/browse/JBMESSAGING-1822
Project: JBoss Messaging
Issue Type: Bug
Components: Messaging Core
Affects Versions: 1.4.6.GA
Reporter: david.boeren
Assignee: Yong Hao Gao
Fix For: Unscheduled
Attachments: helloworld.zip
The MessageSucker is responsible for migrating messages between different members of a
cluster, it is a consumer to the remote queue from which it receives messages destined for
the queue on the local cluster member.
The onMessage routine, at its most basic, does the following
- bookkeeping for the incoming message, including expiry
- acknowledge the incoming message
- attempt to deliver to the local queue
When the delivery fails, the result is the *appearance* of lost messages. Those messages
which are processed during the failure are not redelivered, but they still exist in the
database.
The only way I have found to trigger the redelivery of those messages is to redeploy the
queue containing the messages and/or restart that app server. Obviously neither approach
is acceptable.
In order to trigger the error I created a SOA cluster which *only* shared the JMS
database, and no other. I modified the helloworld quickstart to display a counter of
messages consumed, clustered the *esb* queue, and then used byteman to trigger the faults.
The byteman rule is as follows, the quickstart will be attached.
RULE throw every fifth send
INTERFACE ProducerDelegate
METHOD send
AT ENTRY
IF callerEquals("MessageSucker.onMessage", true) &&
(incrementCounter("throwException") % 5 == 0)
DO THROW new IllegalStateException("Deliberate exception")
ENDRULE
This results in an exception being thrown for every fifth message. Once the delivery has
quiesced, examine the JBM_MSG and JBM_MSG_REF tables to see the messages which have not
been delivered.
The clusters are ports-default and ports-01, the client seeds the gateway by sending 300
messages to the default.
Adding up the counter from each server *plus* the message count from JBM_MSG results in
300 (or multiples thereof for more executions).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:48 a.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1822) MessageSucker failures cause the delivery of the failed message to stall
[
https://jira.jboss.org/browse/JBMESSAGING-1822?page=com.atlassian.jira.pl...
]
Yong Hao Gao commented on JBMESSAGING-1822:
-------------------------------------------
I have finished the implementation. During the coding process, i've made various
changes to my previous proposal. I'll go on with some tests to verify it. Below is
what I actually did:
Changes to the code:
1. Introduce a new state in JBM_MSG_REF table's STATE column. The new state
'S' marks that the message is in a special "To be sucked" state.
2. Change the sucking process as the following:
a) When a Message M is ready to be sucked, it is put to the remote consumer
(ServerConsumerEndpoint) for delivery. This remote consumer on accepting M, update the
M's state to 'S'. Then it goes on to actually deliver M to the Message Sucker
on target node.
b) When the message sucker receives M, now we acknowledge it and then sends it to the
local queue, M's state changed to 'C' (normal).
c) When the acknowledgement arrives at the source node, the session simply forget the
message (without any DB operations).
d) Failover -- when mergeIn the failed channel, it will try to claim 'S'
messages.
Failure handling
The sucking process will have to handle the following failures. Suppose we have a message
M that is just in the process of being sucked. 'Source node' refers to the node
where the source channel (from which M is sucked) resides. 'Target node' refers to
the node where the target channel (to which M is sucked) resides.
I. Source node crash. Different ways of handling are given based on where M is at the
moment of crash.
I.a. M is being processed at the source node.
Normal processing steps: M's state is changed from 'C' to 'S' and then
sent for delivery to target node (sucker).
Case-I.a.1 M's state is 'C'. That means M is still a normal message. The
sucking process has no effect on it.
Case-I.a.2 M's state has been changed to 'S'. M will be merged to backup node
on server failover. It then will be loaded as normal message for delivery.
I.b. M is being processed at the target node
Normal processing steps: M is first acked and then sent to local queue (M updated to the
target channel and its state changed from 'S' to 'C').
I.b.1 M's state is 'S' and hasn't been acked. So the ack will fail. M
remains in 'S' with the source channel. Target node will not get M sucked
eventually. M will be merged to other channel for re-delivery.
I.b.2 M's state is 'S' and has been acked successfully. M will go on to be
sent to local queue. However there is a contention between the sending and failover. The
failover will try to merge any M that is in 'S' state of the source channel, while
the sending will try to update M to the target channel. We guarantee only one of the two
actions will succeed, i.e. M will either be merged to another channel or be sucked to the
target channel.
II. Target node crash. Still different ways of handling are given based on where M is at
the moment of crash.
II-a. M is being processed at the source node.
M will be processed as normal, only the delivery will fail as the target node is down.
However connection failure will come in and close the sessions, in turn sessions will
cancel all existing deliveries, including M. When canceling M, its state will also be
changed back to 'C' if it is already in 'S' state.
II.b. M is being processed at the target node.
II.b.1 M hasn't been acked yet. It will be eventually canceled at the source node for
re-delivery.
II.b.2 M has been acked successfully but haven't been sent to local queue. When
source node eventually get the connection failure notification, it will close the related
session where M will be reclaimed from the DB and put to the channel for redelivery.
III. Cases that involve crashes of both source and target nodes
Depending on the order and timing of the crashes of both nodes, there are cases that may
be same as those in the above one-crash cases. We only consider the case where M is in a
specific state bit "C" or "S" but none of the above handlings are
successful.
If M is in "C" state after both has crashed, that means M either is with source
channel or is with the target channel. In either case M is a normal message. We don't
need to worry about it.
If M is in "S" state after both has crashed, that means M is in sucking process
and we cannot know how far the process has gone before crashing. It is the responsibility
of source node to reclaim it to its source channel when startup, and then redeliver it as
a normal message. If the crashed source server is failed over, it is the failover server
that reclaims M.
IV. Cluster Connection failures
Sometimes both servers are alive but the network that links them becomes problematic. When
ever that happened, either node behaves as if the other has failed. We guarantee that only
one node got M and processed it successfully. For example, when source node session is
closing and canceling M's delivery, target node may send in the ack of M. We
synchronized this two actions so that M either be canceled back to source channel
(resulting in ack failure) or be acked before canceling.
Conclusion
So, for issues specific to this JIRA report, when send is broken, the message will be
reclaimed by the source node (case II.b.2) for redelivery.
MessageSucker failures cause the delivery of the failed message to
stall
------------------------------------------------------------------------
Key: JBMESSAGING-1822
URL: https://jira.jboss.org/browse/JBMESSAGING-1822
Project: JBoss Messaging
Issue Type: Bug
Components: Messaging Core
Affects Versions: 1.4.6.GA
Reporter: david.boeren
Assignee: Yong Hao Gao
Fix For: Unscheduled
Attachments: helloworld.zip
The MessageSucker is responsible for migrating messages between different members of a
cluster, it is a consumer to the remote queue from which it receives messages destined for
the queue on the local cluster member.
The onMessage routine, at its most basic, does the following
- bookkeeping for the incoming message, including expiry
- acknowledge the incoming message
- attempt to deliver to the local queue
When the delivery fails, the result is the *appearance* of lost messages. Those messages
which are processed during the failure are not redelivered, but they still exist in the
database.
The only way I have found to trigger the redelivery of those messages is to redeploy the
queue containing the messages and/or restart that app server. Obviously neither approach
is acceptable.
In order to trigger the error I created a SOA cluster which *only* shared the JMS
database, and no other. I modified the helloworld quickstart to display a counter of
messages consumed, clustered the *esb* queue, and then used byteman to trigger the faults.
The byteman rule is as follows, the quickstart will be attached.
RULE throw every fifth send
INTERFACE ProducerDelegate
METHOD send
AT ENTRY
IF callerEquals("MessageSucker.onMessage", true) &&
(incrementCounter("throwException") % 5 == 0)
DO THROW new IllegalStateException("Deliberate exception")
ENDRULE
This results in an exception being thrown for every fifth message. Once the delivery has
quiesced, examine the JBM_MSG and JBM_MSG_REF tables to see the messages which have not
been delivered.
The clusters are ports-default and ports-01, the client seeds the gateway by sending 300
messages to the default.
Adding up the counter from each server *plus* the message count from JBM_MSG results in
300 (or multiples thereof for more executions).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:29 a.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1822) MessageSucker failures cause the delivery of the failed message to stall
[
https://jira.jboss.org/browse/JBMESSAGING-1822?page=com.atlassian.jira.pl...
]
Yong Hao Gao commented on JBMESSAGING-1822:
-------------------------------------------
Note: we are dealing with persistent messages. non-persistent messages are not guaranteed
to survive such failure conditions.
MessageSucker failures cause the delivery of the failed message to
stall
------------------------------------------------------------------------
Key: JBMESSAGING-1822
URL: https://jira.jboss.org/browse/JBMESSAGING-1822
Project: JBoss Messaging
Issue Type: Bug
Components: Messaging Core
Affects Versions: 1.4.6.GA
Reporter: david.boeren
Assignee: Yong Hao Gao
Fix For: Unscheduled
Attachments: helloworld.zip
The MessageSucker is responsible for migrating messages between different members of a
cluster, it is a consumer to the remote queue from which it receives messages destined for
the queue on the local cluster member.
The onMessage routine, at its most basic, does the following
- bookkeeping for the incoming message, including expiry
- acknowledge the incoming message
- attempt to deliver to the local queue
When the delivery fails, the result is the *appearance* of lost messages. Those messages
which are processed during the failure are not redelivered, but they still exist in the
database.
The only way I have found to trigger the redelivery of those messages is to redeploy the
queue containing the messages and/or restart that app server. Obviously neither approach
is acceptable.
In order to trigger the error I created a SOA cluster which *only* shared the JMS
database, and no other. I modified the helloworld quickstart to display a counter of
messages consumed, clustered the *esb* queue, and then used byteman to trigger the faults.
The byteman rule is as follows, the quickstart will be attached.
RULE throw every fifth send
INTERFACE ProducerDelegate
METHOD send
AT ENTRY
IF callerEquals("MessageSucker.onMessage", true) &&
(incrementCounter("throwException") % 5 == 0)
DO THROW new IllegalStateException("Deliberate exception")
ENDRULE
This results in an exception being thrown for every fifth message. Once the delivery has
quiesced, examine the JBM_MSG and JBM_MSG_REF tables to see the messages which have not
been delivered.
The clusters are ports-default and ports-01, the client seeds the gateway by sending 300
messages to the default.
Adding up the counter from each server *plus* the message count from JBM_MSG results in
300 (or multiples thereof for more executions).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:47 a.m.
New subject: [JBoss JIRA] Updated: (JBMESSAGING-1822) MessageSucker failures cause the delivery of the failed message to stall
[
https://jira.jboss.org/browse/JBMESSAGING-1822?page=com.atlassian.jira.pl...
]
Yong Hao Gao updated JBMESSAGING-1822:
--------------------------------------
Fix Version/s: 1.4.0.SP3.CP11
1.4.8.GA
(was: Unscheduled)
Affects Version/s: 1.4.7.GA
1.4.6.GA.SP1
1.4.0.SP3.CP10
MessageSucker failures cause the delivery of the failed message to
stall
------------------------------------------------------------------------
Key: JBMESSAGING-1822
URL: https://jira.jboss.org/browse/JBMESSAGING-1822
Project: JBoss Messaging
Issue Type: Bug
Components: Messaging Core
Affects Versions: 1.4.0.SP3.CP10, 1.4.6.GA, 1.4.6.GA.SP1, 1.4.7.GA
Reporter: david.boeren
Assignee: Yong Hao Gao
Fix For: 1.4.0.SP3.CP11, 1.4.8.GA
Attachments: helloworld.zip
The MessageSucker is responsible for migrating messages between different members of a
cluster, it is a consumer to the remote queue from which it receives messages destined for
the queue on the local cluster member.
The onMessage routine, at its most basic, does the following
- bookkeeping for the incoming message, including expiry
- acknowledge the incoming message
- attempt to deliver to the local queue
When the delivery fails, the result is the *appearance* of lost messages. Those messages
which are processed during the failure are not redelivered, but they still exist in the
database.
The only way I have found to trigger the redelivery of those messages is to redeploy the
queue containing the messages and/or restart that app server. Obviously neither approach
is acceptable.
In order to trigger the error I created a SOA cluster which *only* shared the JMS
database, and no other. I modified the helloworld quickstart to display a counter of
messages consumed, clustered the *esb* queue, and then used byteman to trigger the faults.
The byteman rule is as follows, the quickstart will be attached.
RULE throw every fifth send
INTERFACE ProducerDelegate
METHOD send
AT ENTRY
IF callerEquals("MessageSucker.onMessage", true) &&
(incrementCounter("throwException") % 5 == 0)
DO THROW new IllegalStateException("Deliberate exception")
ENDRULE
This results in an exception being thrown for every fifth message. Once the delivery has
quiesced, examine the JBM_MSG and JBM_MSG_REF tables to see the messages which have not
been delivered.
The clusters are ports-default and ports-01, the client seeds the gateway by sending 300
messages to the default.
Adding up the counter from each server *plus* the message count from JBM_MSG results in
300 (or multiples thereof for more executions).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:47 a.m.
New subject: [JBoss JIRA] Resolved: (JBMESSAGING-1822) MessageSucker failures cause the delivery of the failed message to stall
[
https://jira.jboss.org/browse/JBMESSAGING-1822?page=com.atlassian.jira.pl...
]
Yong Hao Gao resolved JBMESSAGING-1822.
---------------------------------------
Resolution: Done
MessageSucker failures cause the delivery of the failed message to
stall
------------------------------------------------------------------------
Key: JBMESSAGING-1822
URL: https://jira.jboss.org/browse/JBMESSAGING-1822
Project: JBoss Messaging
Issue Type: Bug
Components: Messaging Core
Affects Versions: 1.4.0.SP3.CP10, 1.4.6.GA, 1.4.6.GA.SP1, 1.4.7.GA
Reporter: david.boeren
Assignee: Yong Hao Gao
Fix For: 1.4.0.SP3.CP11, 1.4.8.GA
Attachments: helloworld.zip
The MessageSucker is responsible for migrating messages between different members of a
cluster, it is a consumer to the remote queue from which it receives messages destined for
the queue on the local cluster member.
The onMessage routine, at its most basic, does the following
- bookkeeping for the incoming message, including expiry
- acknowledge the incoming message
- attempt to deliver to the local queue
When the delivery fails, the result is the *appearance* of lost messages. Those messages
which are processed during the failure are not redelivered, but they still exist in the
database.
The only way I have found to trigger the redelivery of those messages is to redeploy the
queue containing the messages and/or restart that app server. Obviously neither approach
is acceptable.
In order to trigger the error I created a SOA cluster which *only* shared the JMS
database, and no other. I modified the helloworld quickstart to display a counter of
messages consumed, clustered the *esb* queue, and then used byteman to trigger the faults.
The byteman rule is as follows, the quickstart will be attached.
RULE throw every fifth send
INTERFACE ProducerDelegate
METHOD send
AT ENTRY
IF callerEquals("MessageSucker.onMessage", true) &&
(incrementCounter("throwException") % 5 == 0)
DO THROW new IllegalStateException("Deliberate exception")
ENDRULE
This results in an exception being thrown for every fifth message. Once the delivery has
quiesced, examine the JBM_MSG and JBM_MSG_REF tables to see the messages which have not
been delivered.
The clusters are ports-default and ports-01, the client seeds the gateway by sending 300
messages to the default.
Adding up the counter from each server *plus* the message count from JBM_MSG results in
300 (or multiples thereof for more executions).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:49 a.m.
New subject: [JBoss JIRA] Closed: (JBMESSAGING-1822) MessageSucker failures cause the delivery of the failed message to stall
[
https://jira.jboss.org/browse/JBMESSAGING-1822?page=com.atlassian.jira.pl...
]
Yong Hao Gao closed JBMESSAGING-1822.
-------------------------------------
MessageSucker failures cause the delivery of the failed message to
stall
------------------------------------------------------------------------
Key: JBMESSAGING-1822
URL: https://jira.jboss.org/browse/JBMESSAGING-1822
Project: JBoss Messaging
Issue Type: Bug
Components: Messaging Core
Affects Versions: 1.4.0.SP3.CP10, 1.4.6.GA, 1.4.6.GA.SP1, 1.4.7.GA
Reporter: david.boeren
Assignee: Yong Hao Gao
Fix For: 1.4.0.SP3.CP11, 1.4.8.GA
Attachments: helloworld.zip
The MessageSucker is responsible for migrating messages between different members of a
cluster, it is a consumer to the remote queue from which it receives messages destined for
the queue on the local cluster member.
The onMessage routine, at its most basic, does the following
- bookkeeping for the incoming message, including expiry
- acknowledge the incoming message
- attempt to deliver to the local queue
When the delivery fails, the result is the *appearance* of lost messages. Those messages
which are processed during the failure are not redelivered, but they still exist in the
database.
The only way I have found to trigger the redelivery of those messages is to redeploy the
queue containing the messages and/or restart that app server. Obviously neither approach
is acceptable.
In order to trigger the error I created a SOA cluster which *only* shared the JMS
database, and no other. I modified the helloworld quickstart to display a counter of
messages consumed, clustered the *esb* queue, and then used byteman to trigger the faults.
The byteman rule is as follows, the quickstart will be attached.
RULE throw every fifth send
INTERFACE ProducerDelegate
METHOD send
AT ENTRY
IF callerEquals("MessageSucker.onMessage", true) &&
(incrementCounter("throwException") % 5 == 0)
DO THROW new IllegalStateException("Deliberate exception")
ENDRULE
This results in an exception being thrown for every fifth message. Once the delivery has
quiesced, examine the JBM_MSG and JBM_MSG_REF tables to see the messages which have not
been delivered.
The clusters are ports-default and ports-01, the client seeds the gateway by sending 300
messages to the default.
Adding up the counter from each server *plus* the message count from JBM_MSG results in
300 (or multiples thereof for more executions).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:52 p.m.
New subject: [JBoss JIRA] Reopened: (JBMESSAGING-1822) MessageSucker failures cause the delivery of the failed message to stall
[
https://jira.jboss.org/browse/JBMESSAGING-1822?page=com.atlassian.jira.pl...
]
Yong Hao Gao reopened JBMESSAGING-1822:
---------------------------------------
reopen it. Message may get stuck when source node crashes and doesn't restart. In that
case failover node should take care of this.
MessageSucker failures cause the delivery of the failed message to
stall
------------------------------------------------------------------------
Key: JBMESSAGING-1822
URL: https://jira.jboss.org/browse/JBMESSAGING-1822
Project: JBoss Messaging
Issue Type: Bug
Components: Messaging Core
Affects Versions: 1.4.0.SP3.CP10, 1.4.6.GA, 1.4.6.GA.SP1, 1.4.7.GA
Reporter: david.boeren
Assignee: Yong Hao Gao
Fix For: 1.4.0.SP3.CP11, 1.4.8.GA
Attachments: helloworld.zip
The MessageSucker is responsible for migrating messages between different members of a
cluster, it is a consumer to the remote queue from which it receives messages destined for
the queue on the local cluster member.
The onMessage routine, at its most basic, does the following
- bookkeeping for the incoming message, including expiry
- acknowledge the incoming message
- attempt to deliver to the local queue
When the delivery fails, the result is the *appearance* of lost messages. Those messages
which are processed during the failure are not redelivered, but they still exist in the
database.
The only way I have found to trigger the redelivery of those messages is to redeploy the
queue containing the messages and/or restart that app server. Obviously neither approach
is acceptable.
In order to trigger the error I created a SOA cluster which *only* shared the JMS
database, and no other. I modified the helloworld quickstart to display a counter of
messages consumed, clustered the *esb* queue, and then used byteman to trigger the faults.
The byteman rule is as follows, the quickstart will be attached.
RULE throw every fifth send
INTERFACE ProducerDelegate
METHOD send
AT ENTRY
IF callerEquals("MessageSucker.onMessage", true) &&
(incrementCounter("throwException") % 5 == 0)
DO THROW new IllegalStateException("Deliberate exception")
ENDRULE
This results in an exception being thrown for every fifth message. Once the delivery has
quiesced, examine the JBM_MSG and JBM_MSG_REF tables to see the messages which have not
been delivered.
The clusters are ports-default and ports-01, the client seeds the gateway by sending 300
messages to the default.
Adding up the counter from each server *plus* the message count from JBM_MSG results in
300 (or multiples thereof for more executions).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:38 a.m.
New subject: [JBoss JIRA] Updated: (JBMESSAGING-1822) MessageSucker failures cause the delivery of the failed message to stall
[
https://jira.jboss.org/browse/JBMESSAGING-1822?page=com.atlassian.jira.pl...
]
Yong Hao Gao updated JBMESSAGING-1822:
--------------------------------------
Fix Version/s: 1.4.0.SP3.CP12
(was: 1.4.0.SP3.CP11)
MessageSucker failures cause the delivery of the failed message to
stall
------------------------------------------------------------------------
Key: JBMESSAGING-1822
URL: https://jira.jboss.org/browse/JBMESSAGING-1822
Project: JBoss Messaging
Issue Type: Bug
Components: Messaging Core
Affects Versions: 1.4.0.SP3.CP10, 1.4.6.GA, 1.4.6.GA.SP1, 1.4.7.GA
Reporter: david.boeren
Assignee: Yong Hao Gao
Fix For: 1.4.0.SP3.CP12, 1.4.8.GA
Attachments: helloworld.zip
The MessageSucker is responsible for migrating messages between different members of a
cluster, it is a consumer to the remote queue from which it receives messages destined for
the queue on the local cluster member.
The onMessage routine, at its most basic, does the following
- bookkeeping for the incoming message, including expiry
- acknowledge the incoming message
- attempt to deliver to the local queue
When the delivery fails, the result is the *appearance* of lost messages. Those messages
which are processed during the failure are not redelivered, but they still exist in the
database.
The only way I have found to trigger the redelivery of those messages is to redeploy the
queue containing the messages and/or restart that app server. Obviously neither approach
is acceptable.
In order to trigger the error I created a SOA cluster which *only* shared the JMS
database, and no other. I modified the helloworld quickstart to display a counter of
messages consumed, clustered the *esb* queue, and then used byteman to trigger the faults.
The byteman rule is as follows, the quickstart will be attached.
RULE throw every fifth send
INTERFACE ProducerDelegate
METHOD send
AT ENTRY
IF callerEquals("MessageSucker.onMessage", true) &&
(incrementCounter("throwException") % 5 == 0)
DO THROW new IllegalStateException("Deliberate exception")
ENDRULE
This results in an exception being thrown for every fifth message. Once the delivery has
quiesced, examine the JBM_MSG and JBM_MSG_REF tables to see the messages which have not
been delivered.
The clusters are ports-default and ports-01, the client seeds the gateway by sending 300
messages to the default.
Adding up the counter from each server *plus* the message count from JBM_MSG results in
300 (or multiples thereof for more executions).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
12:51 p.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1822) MessageSucker failures cause the delivery of the failed message to stall
[
https://jira.jboss.org/browse/JBMESSAGING-1822?page=com.atlassian.jira.pl...
]
Justin Bertram commented on JBMESSAGING-1822:
---------------------------------------------
Please note, revision 8125 should be a part of this as well. It contains a small fix for
org.jboss.messaging.core.impl.PagingChannelSupport.
MessageSucker failures cause the delivery of the failed message to
stall
------------------------------------------------------------------------
Key: JBMESSAGING-1822
URL: https://jira.jboss.org/browse/JBMESSAGING-1822
Project: JBoss Messaging
Issue Type: Bug
Components: Messaging Core
Affects Versions: 1.4.0.SP3.CP10, 1.4.6.GA, 1.4.6.GA.SP1, 1.4.7.GA
Reporter: david.boeren
Assignee: Yong Hao Gao
Fix For: 1.4.0.SP3.CP12, 1.4.8.GA
Attachments: helloworld.zip
The MessageSucker is responsible for migrating messages between different members of a
cluster, it is a consumer to the remote queue from which it receives messages destined for
the queue on the local cluster member.
The onMessage routine, at its most basic, does the following
- bookkeeping for the incoming message, including expiry
- acknowledge the incoming message
- attempt to deliver to the local queue
When the delivery fails, the result is the *appearance* of lost messages. Those messages
which are processed during the failure are not redelivered, but they still exist in the
database.
The only way I have found to trigger the redelivery of those messages is to redeploy the
queue containing the messages and/or restart that app server. Obviously neither approach
is acceptable.
In order to trigger the error I created a SOA cluster which *only* shared the JMS
database, and no other. I modified the helloworld quickstart to display a counter of
messages consumed, clustered the *esb* queue, and then used byteman to trigger the faults.
The byteman rule is as follows, the quickstart will be attached.
RULE throw every fifth send
INTERFACE ProducerDelegate
METHOD send
AT ENTRY
IF callerEquals("MessageSucker.onMessage", true) &&
(incrementCounter("throwException") % 5 == 0)
DO THROW new IllegalStateException("Deliberate exception")
ENDRULE
This results in an exception being thrown for every fifth message. Once the delivery has
quiesced, examine the JBM_MSG and JBM_MSG_REF tables to see the messages which have not
been delivered.
The clusters are ports-default and ports-01, the client seeds the gateway by sending 300
messages to the default.
Adding up the counter from each server *plus* the message count from JBM_MSG results in
300 (or multiples thereof for more executions).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
12:01 a.m.
New subject: [JBoss JIRA] Resolved: (JBMESSAGING-1822) MessageSucker failures cause the delivery of the failed message to stall
[
https://jira.jboss.org/browse/JBMESSAGING-1822?page=com.atlassian.jira.pl...
]
Yong Hao Gao resolved JBMESSAGING-1822.
---------------------------------------
Resolution: Done
MessageSucker failures cause the delivery of the failed message to
stall
------------------------------------------------------------------------
Key: JBMESSAGING-1822
URL: https://jira.jboss.org/browse/JBMESSAGING-1822
Project: JBoss Messaging
Issue Type: Bug
Components: Messaging Core
Affects Versions: 1.4.0.SP3.CP10, 1.4.6.GA, 1.4.6.GA.SP1, 1.4.7.GA
Reporter: david.boeren
Assignee: Yong Hao Gao
Fix For: 1.4.0.SP3.CP12, 1.4.8.GA
Attachments: helloworld.zip
The MessageSucker is responsible for migrating messages between different members of a
cluster, it is a consumer to the remote queue from which it receives messages destined for
the queue on the local cluster member.
The onMessage routine, at its most basic, does the following
- bookkeeping for the incoming message, including expiry
- acknowledge the incoming message
- attempt to deliver to the local queue
When the delivery fails, the result is the *appearance* of lost messages. Those messages
which are processed during the failure are not redelivered, but they still exist in the
database.
The only way I have found to trigger the redelivery of those messages is to redeploy the
queue containing the messages and/or restart that app server. Obviously neither approach
is acceptable.
In order to trigger the error I created a SOA cluster which *only* shared the JMS
database, and no other. I modified the helloworld quickstart to display a counter of
messages consumed, clustered the *esb* queue, and then used byteman to trigger the faults.
The byteman rule is as follows, the quickstart will be attached.
RULE throw every fifth send
INTERFACE ProducerDelegate
METHOD send
AT ENTRY
IF callerEquals("MessageSucker.onMessage", true) &&
(incrementCounter("throwException") % 5 == 0)
DO THROW new IllegalStateException("Deliberate exception")
ENDRULE
This results in an exception being thrown for every fifth message. Once the delivery has
quiesced, examine the JBM_MSG and JBM_MSG_REF tables to see the messages which have not
been delivered.
The clusters are ports-default and ports-01, the client seeds the gateway by sending 300
messages to the default.
Adding up the counter from each server *plus* the message count from JBM_MSG results in
300 (or multiples thereof for more executions).
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
12:02 a.m.
New subject: [JBoss JIRA] Closed: (JBMESSAGING-1822) MessageSucker failures cause the delivery of the failed message to stall
[
https://jira.jboss.org/browse/JBMESSAGING-1822?page=com.atlassian.jira.pl...
]
Yong Hao Gao closed JBMESSAGING-1822.
-------------------------------------
MessageSucker failures cause the delivery of the failed message to
stall
------------------------------------------------------------------------
Key: JBMESSAGING-1822
URL: https://jira.jboss.org/browse/JBMESSAGING-1822
Project: JBoss Messaging
Issue Type: Bug
Components: Messaging Core
Affects Versions: 1.4.0.SP3.CP10, 1.4.6.GA, 1.4.6.GA.SP1, 1.4.7.GA
Reporter: david.boeren
Assignee: Yong Hao Gao
Fix For: 1.4.0.SP3.CP12, 1.4.8.GA
Attachments: helloworld.zip
The MessageSucker is responsible for migrating messages between different members of a
cluster, it is a consumer to the remote queue from which it receives messages destined for
the queue on the local cluster member.
The onMessage routine, at its most basic, does the following
- bookkeeping for the incoming message, including expiry
- acknowledge the incoming message
- attempt to deliver to the local queue
When the delivery fails, the result is the *appearance* of lost messages. Those messages
which are processed during the failure are not redelivered, but they still exist in the
database.
The only way I have found to trigger the redelivery of those messages is to redeploy the
queue containing the messages and/or restart that app server. Obviously neither approach
is acceptable.
In order to trigger the error I created a SOA cluster which *only* shared the JMS
database, and no other. I modified the helloworld quickstart to display a counter of
messages consumed, clustered the *esb* queue, and then used byteman to trigger the faults.
The byteman rule is as follows, the quickstart will be attached.
RULE throw every fifth send
INTERFACE ProducerDelegate
METHOD send
AT ENTRY
IF callerEquals("MessageSucker.onMessage", true) &&
(incrementCounter("throwException") % 5 == 0)
DO THROW new IllegalStateException("Deliberate exception")
ENDRULE
This results in an exception being thrown for every fifth message. Once the delivery has
quiesced, examine the JBM_MSG and JBM_MSG_REF tables to see the messages which have not
been delivered.
The clusters are ports-default and ports-01, the client seeds the gateway by sending 300
messages to the default.
Adding up the counter from each server *plus* the message count from JBM_MSG results in
300 (or multiples thereof for more executions).
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:51 p.m.
New subject: [JBoss JIRA] Reopened: (JBMESSAGING-1822) MessageSucker failures cause the delivery of the failed message to stall
[
https://issues.jboss.org/browse/JBMESSAGING-1822?page=com.atlassian.jira....
]
Yong Hao Gao reopened JBMESSAGING-1822:
---------------------------------------
some coding flaw found
MessageSucker failures cause the delivery of the failed message to
stall
------------------------------------------------------------------------
Key: JBMESSAGING-1822
URL: https://issues.jboss.org/browse/JBMESSAGING-1822
Project: JBoss Messaging
Issue Type: Bug
Components: Messaging Core
Affects Versions: 1.4.0.SP3.CP10, 1.4.6.GA, 1.4.6.GA.SP1, 1.4.7.GA
Reporter: david.boeren
Assignee: Yong Hao Gao
Fix For: 1.4.0.SP3.CP12, 1.4.8.GA
Attachments: helloworld.zip
The MessageSucker is responsible for migrating messages between different members of a
cluster, it is a consumer to the remote queue from which it receives messages destined for
the queue on the local cluster member.
The onMessage routine, at its most basic, does the following
- bookkeeping for the incoming message, including expiry
- acknowledge the incoming message
- attempt to deliver to the local queue
When the delivery fails, the result is the *appearance* of lost messages. Those messages
which are processed during the failure are not redelivered, but they still exist in the
database.
The only way I have found to trigger the redelivery of those messages is to redeploy the
queue containing the messages and/or restart that app server. Obviously neither approach
is acceptable.
In order to trigger the error I created a SOA cluster which *only* shared the JMS
database, and no other. I modified the helloworld quickstart to display a counter of
messages consumed, clustered the *esb* queue, and then used byteman to trigger the faults.
The byteman rule is as follows, the quickstart will be attached.
RULE throw every fifth send
INTERFACE ProducerDelegate
METHOD send
AT ENTRY
IF callerEquals("MessageSucker.onMessage", true) &&
(incrementCounter("throwException") % 5 == 0)
DO THROW new IllegalStateException("Deliberate exception")
ENDRULE
This results in an exception being thrown for every fifth message. Once the delivery has
quiesced, examine the JBM_MSG and JBM_MSG_REF tables to see the messages which have not
been delivered.
The clusters are ports-default and ports-01, the client seeds the gateway by sending 300
messages to the default.
Adding up the counter from each server *plus* the message count from JBM_MSG results in
300 (or multiples thereof for more executions).
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:18 a.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1822) MessageSucker failures cause the delivery of the failed message to stall
[
https://issues.jboss.org/browse/JBMESSAGING-1822?page=com.atlassian.jira....
]
Yong Hao Gao commented on JBMESSAGING-1822:
-------------------------------------------
1. JDBCPersistenceManager.claimMessagesInSuck(final long channelID) uses two sql separate
statements in a transaction, the first is a sql query and the second is an update. As
JBM's DB transaction isolation level is READ_COMMITTED,
the two sql executions can result in inconsistent state.
Fix: use the result from the first query to do the update.
2. In ServerSessionEndpoint.localClose(boolean isFromFailure), when the messages has been
claimed back to the channel, the channel is not prompted for immediate delivery, this
could result in those messages stuck in the queue if there is no other events that
triggers the delivery (like more messages being added, consumer changes etc).
Fix: make the channel do an immediate delivery after such messages has been claimed.
MessageSucker failures cause the delivery of the failed message to
stall
------------------------------------------------------------------------
Key: JBMESSAGING-1822
URL: https://issues.jboss.org/browse/JBMESSAGING-1822
Project: JBoss Messaging
Issue Type: Bug
Components: Messaging Core
Affects Versions: 1.4.0.SP3.CP10, 1.4.6.GA, 1.4.6.GA.SP1, 1.4.7.GA
Reporter: david.boeren
Assignee: Yong Hao Gao
Fix For: 1.4.0.SP3.CP12, 1.4.8.GA
Attachments: helloworld.zip
The MessageSucker is responsible for migrating messages between different members of a
cluster, it is a consumer to the remote queue from which it receives messages destined for
the queue on the local cluster member.
The onMessage routine, at its most basic, does the following
- bookkeeping for the incoming message, including expiry
- acknowledge the incoming message
- attempt to deliver to the local queue
When the delivery fails, the result is the *appearance* of lost messages. Those messages
which are processed during the failure are not redelivered, but they still exist in the
database.
The only way I have found to trigger the redelivery of those messages is to redeploy the
queue containing the messages and/or restart that app server. Obviously neither approach
is acceptable.
In order to trigger the error I created a SOA cluster which *only* shared the JMS
database, and no other. I modified the helloworld quickstart to display a counter of
messages consumed, clustered the *esb* queue, and then used byteman to trigger the faults.
The byteman rule is as follows, the quickstart will be attached.
RULE throw every fifth send
INTERFACE ProducerDelegate
METHOD send
AT ENTRY
IF callerEquals("MessageSucker.onMessage", true) &&
(incrementCounter("throwException") % 5 == 0)
DO THROW new IllegalStateException("Deliberate exception")
ENDRULE
This results in an exception being thrown for every fifth message. Once the delivery has
quiesced, examine the JBM_MSG and JBM_MSG_REF tables to see the messages which have not
been delivered.
The clusters are ports-default and ports-01, the client seeds the gateway by sending 300
messages to the default.
Adding up the counter from each server *plus* the message count from JBM_MSG results in
300 (or multiples thereof for more executions).
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:42 p.m.
New subject: [JBoss JIRA] Closed: (JBMESSAGING-1822) MessageSucker failures cause the delivery of the failed message to stall
[
https://issues.jboss.org/browse/JBMESSAGING-1822?page=com.atlassian.jira....
]
Yong Hao Gao closed JBMESSAGING-1822.
-------------------------------------
MessageSucker failures cause the delivery of the failed message to
stall
------------------------------------------------------------------------
Key: JBMESSAGING-1822
URL: https://issues.jboss.org/browse/JBMESSAGING-1822
Project: JBoss Messaging
Issue Type: Bug
Components: Messaging Core
Affects Versions: 1.4.0.SP3.CP10, 1.4.6.GA, 1.4.6.GA.SP1, 1.4.7.GA
Reporter: david.boeren
Assignee: Yong Hao Gao
Fix For: 1.4.0.SP3.CP12, 1.4.8.GA
Attachments: helloworld.zip
The MessageSucker is responsible for migrating messages between different members of a
cluster, it is a consumer to the remote queue from which it receives messages destined for
the queue on the local cluster member.
The onMessage routine, at its most basic, does the following
- bookkeeping for the incoming message, including expiry
- acknowledge the incoming message
- attempt to deliver to the local queue
When the delivery fails, the result is the *appearance* of lost messages. Those messages
which are processed during the failure are not redelivered, but they still exist in the
database.
The only way I have found to trigger the redelivery of those messages is to redeploy the
queue containing the messages and/or restart that app server. Obviously neither approach
is acceptable.
In order to trigger the error I created a SOA cluster which *only* shared the JMS
database, and no other. I modified the helloworld quickstart to display a counter of
messages consumed, clustered the *esb* queue, and then used byteman to trigger the faults.
The byteman rule is as follows, the quickstart will be attached.
RULE throw every fifth send
INTERFACE ProducerDelegate
METHOD send
AT ENTRY
IF callerEquals("MessageSucker.onMessage", true) &&
(incrementCounter("throwException") % 5 == 0)
DO THROW new IllegalStateException("Deliberate exception")
ENDRULE
This results in an exception being thrown for every fifth message. Once the delivery has
quiesced, examine the JBM_MSG and JBM_MSG_REF tables to see the messages which have not
been delivered.
The clusters are ports-default and ports-01, the client seeds the gateway by sending 300
messages to the default.
Adding up the counter from each server *plus* the message count from JBM_MSG results in
300 (or multiples thereof for more executions).
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:42 p.m.
New subject: [JBoss JIRA] Resolved: (JBMESSAGING-1822) MessageSucker failures cause the delivery of the failed message to stall
[
https://issues.jboss.org/browse/JBMESSAGING-1822?page=com.atlassian.jira....
]
Yong Hao Gao resolved JBMESSAGING-1822.
---------------------------------------
Resolution: Done
MessageSucker failures cause the delivery of the failed message to
stall
------------------------------------------------------------------------
Key: JBMESSAGING-1822
URL: https://issues.jboss.org/browse/JBMESSAGING-1822
Project: JBoss Messaging
Issue Type: Bug
Components: Messaging Core
Affects Versions: 1.4.0.SP3.CP10, 1.4.6.GA, 1.4.6.GA.SP1, 1.4.7.GA
Reporter: david.boeren
Assignee: Yong Hao Gao
Fix For: 1.4.0.SP3.CP12, 1.4.8.GA
Attachments: helloworld.zip
The MessageSucker is responsible for migrating messages between different members of a
cluster, it is a consumer to the remote queue from which it receives messages destined for
the queue on the local cluster member.
The onMessage routine, at its most basic, does the following
- bookkeeping for the incoming message, including expiry
- acknowledge the incoming message
- attempt to deliver to the local queue
When the delivery fails, the result is the *appearance* of lost messages. Those messages
which are processed during the failure are not redelivered, but they still exist in the
database.
The only way I have found to trigger the redelivery of those messages is to redeploy the
queue containing the messages and/or restart that app server. Obviously neither approach
is acceptable.
In order to trigger the error I created a SOA cluster which *only* shared the JMS
database, and no other. I modified the helloworld quickstart to display a counter of
messages consumed, clustered the *esb* queue, and then used byteman to trigger the faults.
The byteman rule is as follows, the quickstart will be attached.
RULE throw every fifth send
INTERFACE ProducerDelegate
METHOD send
AT ENTRY
IF callerEquals("MessageSucker.onMessage", true) &&
(incrementCounter("throwException") % 5 == 0)
DO THROW new IllegalStateException("Deliberate exception")
ENDRULE
This results in an exception being thrown for every fifth message. Once the delivery has
quiesced, examine the JBM_MSG and JBM_MSG_REF tables to see the messages which have not
been delivered.
The clusters are ports-default and ports-01, the client seeds the gateway by sending 300
messages to the default.
Adding up the counter from each server *plus* the message count from JBM_MSG results in
300 (or multiples thereof for more executions).
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
4833
days inactive
4936
days old
35 comments
5 participants
participants (5)
-
Clebert Suconic (JIRA) -
david.boeren (JIRA)
-
Justin Bertram (JIRA)
-
Tim Fox (JIRA)
-
Yong Hao Gao (JIRA)