[ https://issues.jboss.org/browse/JBMESSAGING-1842?page=com.atlassian.jira.... ] Justin Bertram updated JBMESSAGING-1842: ---------------------------------------- Description: When a live node is kicked out of the cluster (for whatever reason) its JBoss Messaging ServerPeer remains active which means the node is still available to send messages to clients. However, when the node is kicked out of the cluster another node in the cluster performs fail-over for that node and takes ownership of that node's messages in the database. The "dead" node knows nothing about this and believes it still owns those messages and therefore will deliver those messages to clients. After delivery it tries to remove the message from the database and can't (because it doesn't actually own that message anymore). When this happens the "dead" node issues a WARN like this: WARN [JDBCPersistenceManager] Failed to remove row for: Reference[23318958991900672]:RELIABLE Of course, the node which performed the fail-over and actually owns the message now may also deliver the message to client. was: When a live node is kicked out of the cluster (for whatever reason) its JBoss Messaging ServerPeer remains active which means the node is still available to receive messages and, if necessary, persist them to the database. However, the CHANNEL_ID values used for the persisted messages will be invalid since the fail-over procedure would have removed the "dead" node's channels from the JBM_POSTOFFICE and new CHANNEL_ID values will be used when the "dead" node is finally restarted. This essentially orphans any persistent messages sent to the "dead" node after it has been kicked out of the cluster. Recovering the orphaned messages would require manual intervention with the database, but even then it might not be possible to match the old CHANNEL_ID values with the new ones. The simplest way to solve this problem is to add a constraint to the database that forces the inserted message's CHANNEL_ID to correspond to a CHANNEL_ID from JBM_POSTOFFICE. Of course, such a constraint will hurt performance a bit, but it could be optional - customers could trade robustness for speed. The node would still need to be restarted in order to rejoin the cluster appropriately, but the constraint would avoid orphaned messages. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JBMESSAGING-1842?page=com.atlassian.jira.... ] Yong Hao Gao commented on JBMESSAGING-1842: ------------------------------------------- Solution proposed by Justin: -Create a new table (e.g. CREATE TABLE JBM_CLUSTER (NODE_ID INTEGER, PING_TIMESTAMP BIGINT, PRIMARY KEY(NODE_ID))) that would be used to keep track of which nodes in the cluster were still alive. -Each node in the cluster would update the table every 30 seconds to indicate it was still alive. -When a node is notified by JGroups that its "buddy" failed it will check this table to see if the node is actually dead or not. -If the data from the table indicated that the node was, in fact, dead then fail-over would continue as normal. During fail-over the "dead" node's row would be removed from the table. -If the data from the table did not indicate that the node was dead then fail-over would not occur. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JBMESSAGING-1842?page=com.atlassian.jira.... ] Yong Hao Gao commented on JBMESSAGING-1842: ------------------------------------------- The general picture could be described as this: We have a cluster of n nodes, at a time m nodes (m < n) of the cluster leaves the cluster but they are still alive. We should guarantee: 1. Each of the m nodes can continue work as a standalone server. 2. The rest of the (n-m) nodes work as a cluster. 3. No duplicate message delivery, no stuck messages in such a situation. 4. any of the m nodes should be able to re-join the cluster. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JBMESSAGING-1842?page=com.atlassian.jira.... ] Yong Hao Gao commented on JBMESSAGING-1842: ------------------------------------------- Also make this configurable to keep it backward compatible. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JBMESSAGING-1842?page=com.atlassian.jira.... ] Yong Hao Gao commented on JBMESSAGING-1842: ------------------------------------------- --- The Test Plan --- I - Verify the old failover works ( keepOldFailoverModel = true ) All existing tests should pass. All example works. JBM_CLUSTER table can be created, but no actions against it (timestamp updates and state changes). II - New failover model (keepOldFailoverModel = false) 0. verify All examples 1. Timestamp update • timestamp refresh interval: timestamp should be updated after each interval. 2. Normal Failover • Node shutdown, crash should cause failover to happen (if failoverOnNodeLeave is true) • Multiple nodes shutdown and crash 3. Node leaving cluster alive (Using JGroups util to simulate) • one node leaves the cluster: failover should not happen. the node serves as standalone server. the cluster quarantined it as expected. • if the node later shutdown/crash: failover should happen (if failoverOnNodeLeave is true) • if the node later re-joins the cluster: its quarantined state will be back to clustered. Repeat the above 3 steps with multiple node leaving, shutdown/crashing and rejoining. All message producers and consumers in the above test should work properly. The following should be checked: • if a cluster has some nodes (>1) in quarantined state, will a client side failover be limited to those normal nodes? or the client may failover to /from a quarantined node? What happens if the quarantined node rejoins the cluster? or what if it crashed/shutdown? • if client side failover works consistently in a dynamic situation where nodes consistently leave, rejoin, shutdown and crash? -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JBMESSAGING-1842?page=com.atlassian.jira.... ] Yong Hao Gao resolved JBMESSAGING-1842. --------------------------------------- Resolution: Done Finally! -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JBMESSAGING-1842?page=com.atlassian.jira.... ] Yong Hao Gao updated JBMESSAGING-1842: -------------------------------------- Affects Version/s: 1.4.0.SP3.CP11 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira