[ https://jira.jboss.org/jira/browse/JBMESSAGING-1558?page=com.atlassian.ji... ] Clebert Suconic updated JBMESSAGING-1558: ----------------------------------------- Fix Version/s: 2.0.0 Beta -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1558?page=com.atlassian.ji... ] Clebert Suconic updated JBMESSAGING-1558: ----------------------------------------- Attachment: backup-6694.patch -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1558?page=com.atlassian.ji... ] Clebert Suconic updated JBMESSAGING-1558: ----------------------------------------- Summary: Order of Delivery is not being respected on backup node (was: Order of ACKs is not being respected on backup node) When a message is delivered, the replication on the backup is happening out of order. I'm still investigating, and I've replicated it in a test where paging is disabled. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1558?page=com.atlassian.ji... ] Clebert Suconic commented on JBMESSAGING-1558: ---------------------------------------------- The delivery is being replicated in a different order thatn what happened on the live node, what is causing issues during the repplication. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1558?page=com.atlassian.ji... ] Clebert Suconic updated JBMESSAGING-1558: ----------------------------------------- Description: The order of delivery is not being respected on backup. When a delivery is replicated it may arrive in a different order than what happened on live what will cause issues on handleReplicatedDelivery ServerConsumerImpl::deliverReplicated. (That includes paging.. but not only paging). You will see exceptions like: throw new IllegalStateException("Cannot find ref " + messageID + " in queue " + messageQueue.getName()); As this opertaion will fail: public void deliverReplicated(final long messageID) throws Exception { MessageReference ref = messageQueue.removeFirstReference(messageID); was: The order of delivery is not being respected on backup. When a delivery is replicated it may arrive in a different order than what happened on live what will cause issues with deliveryReplicate. (That includes paging.. but not only paging). -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1558?page=com.atlassian.ji... ] Clebert Suconic updated JBMESSAGING-1558: ----------------------------------------- Summary: Order of Delivery is not being respected on backup node after ServerConsumerImpl.close is called with pending deliveries (was: Order of Delivery is not being respected on backup node) Description: The order of queues is not being respected between backup and livenode after ServerConsumerImpl.close is called. ServerConsumerImpl::close will call QueueImpl::cancel for every delivering reference, and the order of the Queues will be different between backup and live node that operation. Next time deliveries are replicated they will be in different orders between the two nodes and several issues will happen after that. You will see exceptions like: throw new IllegalStateException("Cannot find ref " + messageID + " in queue " + messageQueue.getName()); As this opertaion will fail: public void deliverReplicated(final long messageID) throws Exception { MessageReference ref = messageQueue.removeFirstReference(messageID); was: The order of delivery is not being respected on backup. When a delivery is replicated it may arrive in a different order than what happened on live what will cause issues on ServerConsumerImpl::deliverReplicated. (That includes paging.. but not only paging). You will see exceptions like: throw new IllegalStateException("Cannot find ref " + messageID + " in queue " + messageQueue.getName()); As this opertaion will fail: public void deliverReplicated(final long messageID) throws Exception { MessageReference ref = messageQueue.removeFirstReference(messageID); -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1558?page=com.atlassian.ji... ] Tim Fox commented on JBMESSAGING-1558: -------------------------------------- What is causing the consumer to close? Are you explicitly closing the consumer, or is it from session close, or connection failure? -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1558?page=com.atlassian.ji... ] Tim Fox commented on JBMESSAGING-1558: -------------------------------------- "possible fix is to scan for the ID if not found at the head. I was still having some other issues with IDs not being found on Paging, but I have some idea of what could be. " We need the core issue to be fixed not a workaround. Replication should not allow deliveries to occur in a different order on live compared to backup. If that is occurring you need to find out the core reason why it's occurring. Once you find that and fix it, there will be no need for hacky workarounds. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1558?page=com.atlassian.ji... ] Clebert Suconic updated JBMESSAGING-1558: ----------------------------------------- Attachment: diff-6724.patch This patch includes my backup including tests.. and lots of traces that are not intended to commit. It's just a backup ATM -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1558?page=com.atlassian.ji... ] Clebert Suconic updated JBMESSAGING-1558: ----------------------------------------- Attachment: diagram2.png #www.websequencediagrams.com URL for regeneration: http://www.websequencediagrams.com/?lz=cGFydGljaXBhbnQgQ29uc3VtZXIoQSkKAA... Syntax: ____________________________________________________________ participant Consumer(A) participant Consumer(B) #Second diagram Consumer(A)->ServerConsumer(A): close activate ServerConsumer(A) ServerConsumer(A)->ServerConsumerBackup(A): close (Reference (A)) ServerConsumerBackup(A)->ServerConsumer(A): done deactivate ServerConsumer(A) participant ServerConsumer(B) participant ServerConsumerBackup(B) participant QueueImpl activate QueueImpl QueueImpl->QueueImpl: promptDelivery Reference(A) note over QueueImpl Something will activate promptDelivery end note Consumer(B)->ServerConsumer(B): close (Reference (B)) QueueImpl->ServerConsumer(A): deliverReference(A) note over QueueImpl At this point top of the list on Live is ReferenceA end note deactivate QueueImpl ServerConsumer(B)->ServerConsumerBackup(B): close (Reference(B)) ServerConsumer(A)->ServerConsumerBackup(A): replicateDelivery (Reference(A)) note over ServerConsumerBackup(A) Exception! Top of the list is reference(B) and not reference(A) end note -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1558?page=com.atlassian.ji... ] Clebert Suconic updated JBMESSAGING-1558: ----------------------------------------- Attachment: diagram1.png www.websequencediagrams.com: #generated on www.websequencediagrams.com with this syntax: participant Consumer(A) participant Consumer(B) participant ServerSession participant Queue ServerSession->Queue: deliverAsync (from promptDelivery) activate Queue Queue->ServerConsumer(A): deliver Ref[A] ServerConsumer(A)->ServerConsumerBackup(A): replicate Ref[A] ServerConsumerBackup(A)->ServerConsumer(A): done ServerConsumer(A)->Consumer(A): deliver Ref[A] Queue->ServerConsumer(B): deliver Ref[B] ServerConsumer(B)->ServerConsumerBackup(B): replicate Ref[B] ServerConsumerBackup(B)->ServerConsumer(B): done deactivate Queue ServerConsumer(B)->Consumer(B): deliver Ref[B] -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1558?page=com.atlassian.ji... ] Clebert Suconic updated JBMESSAGING-1558: ----------------------------------------- Attachment: diagram3.png This is a more detailed diagram: http://www.websequencediagrams.com/?lz=cGFydGljaXBhbnQgQ29uc3VtZXIoQSkKAA... ______________________ participant Consumer(A) participant Consumer(B) participant ServerConsumer(A) participant ServerConsumerBackup(A) participant Queue participant QueueBackup participant ServerConsumer(B) participant ServerConsumerBackup(B) #Second diagram Consumer(A)->ServerConsumer(A): close activate ServerConsumer(A) ServerConsumer(A)->ServerConsumerBackup(A): close (Reference (A)) ServerConsumerBackup(A)->QueueBackup: addReferenceOnTop (Reference(A)) ServerConsumerBackup(A)->ServerConsumer(A): doneClose ServerConsumer(A)->Queue: addReferenceOnTop (Reference(A)) deactivate ServerConsumer(A) activate Queue Queue->Queue: deliverAsync note over Queue Something will activate deliverAsync end note Consumer(B)->ServerConsumer(B): close (Reference (B)) Queue->ServerConsumer(A): deliverReference(A) ServerConsumer(B)->ServerConsumerBackup(B): close (Reference(B)) ServerConsumerBackup(B)->QueueBackup: addReferenceOnTop (Reference(B)) ServerConsumerBackup(B)->ServerConsumer(B): doneClose ServerConsumer(B)->Queue: addReferenceOnTop(Reference(B)) ServerConsumer(A)->ServerConsumerBackup(A): replicateDelivery (Reference(A)) ServerConsumer(A)->QueueBackup: getTopReference (Reference(A)) note over ServerConsumerBackup(A) Exception! Top of the list is reference(B) and not reference(A) end note -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1558?page=com.atlassian.ji... ] Clebert Suconic updated JBMESSAGING-1558: ----------------------------------------- Attachment: (was: diagram3.png) -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1558?page=com.atlassian.ji... ] Clebert Suconic closed JBMESSAGING-1558. ---------------------------------------- Resolution: Done -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1558?page=com.atlassian.ji... ] Clebert Suconic closed JBMESSAGING-1558. ---------------------------------------- Resolution: Done -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira