[infinispan-issues] [JBoss JIRA] (ISPN-3791) Silence "Received invalid rebalance confirmation from NodeX" exceptions

[Dan Berindei (JIRA)]

Dan Berindei created ISPN-3791:
----------------------------------

             Summary: Silence "Received invalid rebalance confirmation from NodeX" exceptions
                 Key: ISPN-3791
                 URL: https://issues.jboss.org/browse/ISPN-3791
             Project: Infinispan
          Issue Type: Bug
          Components: State transfer
    Affects Versions: 6.0.0.Final
            Reporter: Dan Berindei
            Assignee: Dan Berindei
            Priority: Minor


When the coordinator shuts down, it tries to shut down each of its caches first. This triggers a rebalance for the rest of the members, but the rebalance usually finishes only after the coordinator's channel also shuts down. 

The nodes who finish their state transfer will then send a REBALANCE_CONFIRM command to the new coordinator, but the new coordinator doesn't know about that rebalance (it will start the rebalance process from scratch). This results in exceptions like this in the new coordinator's log:

{noformat}
12:36:04,977 WARN  [org.infinispan.topology.CacheTopologyControlCommand] (remote-thread-2,ISPN-Node-1) ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=MyCoolCache, type=REBALANCE_CONFIRM, sender=ISPN-Node-3-54019, joinInfo=null, topologyId=8, currentCH=null, pendingCH=null, throwable=null, viewId=4}: org.infinispan.commons.CacheException: Received invalid rebalance confirmation from ISPN-Node-3-54019 for cache MyCoolCache, we don't have a rebalance in progress
	at org.infinispan.topology.ClusterTopologyManagerImpl.handleRebalanceCompleted(ClusterTopologyManagerImpl.java:190) [infinispan-core-6.0.0.Final.jar:6.0.0.Final]
	at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:147) [infinispan-core-6.0.0.Final.jar:6.0.0.Final]
	at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:124) [infinispan-core-6.0.0.Final.jar:6.0.0.Final]
	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$4.run(CommandAwareRpcDispatcher.java:270) [infinispan-core-6.0.0.Final.jar:6.0.0.Final]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_45]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_45]
	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_45]
{noformat}

A simple way to avoid these warnings would be to keep track of the coordinator that initiated a particular rebalance on each node, and only send the confirmation message to that coordinator. The same warnings  seem to appear on the old coordinator, when it receives a confirmation after its ClusterTopologyManager started shutting down, so we may need another check there.

A more ambitious approach would be to keep the old rebalance when the new coordinator takes over, and have another round in the cluster state recovery asking if any members have already sent REBALANCE_CONFIRMATION commands (after the new coordinator is ready to process those commands). This should eliminate the duplicate state transfer that happens now.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira