I'm commenting on the wiki page. On 27 Sep 2011, at 17:22, Dan Berindei wrote: -- Manik Surtani manik(a)jboss.org twitter.com/maniksurtani Lead, Infinispan http://www.infinispan.org

On Wed, Sep 28, 2011 at 12:59 PM, Bela Ban <bban(a)redhat.com&gt; wrote: Indeed, one member could disable view delivery for the entire cluster. The main scenario I had in mind for this was stopping the entire cluster. A good rebalancing policy might handle such cases automatically, so I'm not 100% sure we need this. I'm not sure what benefits you would get from joining the cache view but not receiving state compared to not sending the join request at all. Non-members are allowed to send/receive commands, so in the future we could even have separate "server" nodes (that join the cache view) and "client" nodes (joining the JGroups cluster to send commands, but not the cache view and so not holding any data except L1). My idea was that the cache view was a representation of the caches that are able to service requests, so it doesn't make sense to include in the view caches that don't hold data. As a practical note, if we add nodes to the consistent hash as they come along, a key will have one set of owners if written to before the join and another set of owner if written to after the join. How should another node determine the current location of the key, especially considering that there can be many view changes before rebalancing is triggerred? Nope, this doesn't have anything to do with transactional caches, instead it is all about computing the owner that will push the key during the rebalance operation. In order to do it deterministically we need to have a common "last good consistent hash" for the last rebalance that finished successfully, and each node must determine if it should push a key or not based on that last good CH. A rebalance operation can also fail for various reasons (e.g. the coordinator died). If that happens the new owners won't have all the state, so they should not receive requests for the state that they would have had in the pending CH. If view installation wasn't tied to state transfer then we'd have to keep yet the last rebalanced view somewhere else. We would hold the "last pending view" (pending rebalance, that is) in the CacheViewsManager and the "last rebalanced view" in another component, and that component would have it's own mechanism for synchronizing the "last rebalanced view" among cache members. So I think the 2PC approach in CacheViewsManager actually simplifies things. Cheers Dan

Yes, I think a merge needs to be treated as a special case, in which we might have to compute the owner(s) for *every* key in a cache. Or we take a deterministic decision, as you mentioned, and only do this for the non-primary partition. On 9/28/11 3:01 PM, Dan Berindei wrote: -- Bela Ban Lead JGroups (http://www.jgroups.org) JBoss / Red Hat

On 9/28/11 1:48 PM, Dan Berindei wrote: I had the scenario in mind where you join 100 members and only *then* do a state transfer (rebalancing). OK. So with periodic rebalancing, you'd hold (virtual) views and state *until* the trigger fires, which then installs the new virtual views and rebalances the state ? In this case, tying view delivery and rebalancing together makes sense... OK. I just hope this makes sense for large clusters, as it is a 2PC, which doesn't scale to a larger number of nodes. I mean, we don't use FLUSH in large clusters for the same reason. Hmm, on the upside, you don't run this algorithm a lot though, so maybe running it only a few times amortizes the cost of it. With this algorithm, I assume you won't need the transitory view anymore (UnionConsistentHashFunction or whatever it was called), which includes both current and new owners of a key ? OK, fair enough OK, agreed. I would not like this if it was run on every view installation, but since we're running it after a cooldown period, or after having received N JOIN requests or M LEAVE requests, I guess it should be fine. +1 on simplification -- Bela Ban Lead JGroups (http://www.jgroups.org) JBoss / Red Hat

Nice stuff. Some q: - When a node CM that has c1 and c2 caches running on it goes down, you'll end up having REQUEST_LEAVE(c1) and REQUEST_LEAVE(c2) at the same time. Is this handled in serial or in parallel? - The view updates can be disabled dynamically (DISABLE_VIEW_UPDATES). Shouldn't we also be able to re-enabled them manually after that? The view accept handler updates the pending view acquires the view change lock in exclusive mode, then releases it immediately Shouldn't this be: - acquire exclusive lock - updates the pending view - release lock ? On 27 Sep 2011, at 17:22, Dan Berindei wrote:

On 9/28/11 4:24 PM, Mircea Markus wrote: I don't think any requests should be sent in this case: the current coordinator should add the LEAVE(s) into its queue, as if it indeed did receive 2 LEAVE requests. This may or may not lead to rebalance, depending on the rebalance policy (e.g. JOINS are queued, but LEAVES and MERGES trigger a rebalance immediately). -- Bela Ban Lead JGroups (http://www.jgroups.org) JBoss / Red Hat

Yes, the idea was to have a bunch of conditions ORed together, so if one of them was true a rebalance would occur. E.g.: - Try to see if a rebalance is needed every 5 seconds. When there's a view change, reset the timer, so it'll fire 5 secs from now. This would prevent rebalancing to happen in a high churn period, e.g. when 10 nodes join - When N JOINs have been received - When M LEAVES have been received - On a MERGE On 9/29/11 6:14 PM, Mircea Markus wrote: -- Bela Ban Lead JGroups (http://www.jgroups.org) JBoss / Red Hat