[infinispan-dev] Asymmetric caches and manual rehashing design

[Dan Berindei]

On Wed, Sep 28, 2011 at 12:59 PM, Bela Ban <bban at redhat.com> wrote:
> My 5 cents:
>
> Virtual Cache Views:
> ================
>
>
> - The REQUEST_VIEW_UPDATE is a good idea to manually trigger flushing of
> pending views. However, I'm not sure what the goal of
> DISABLE_VIEW_UPDATES is. Since I assume this is for the entire cluster,
> can 1 member disable view delivery for the entire cluster ? What
> scenario is this for ?
>

Indeed, one member could disable view delivery for the entire cluster.
The main scenario I had in mind for this was stopping the entire
cluster. A good rebalancing policy might handle such cases
automatically, so I'm not 100% sure we need this.

> - Are you clubbing (virtual) view updates and rebalancing together ? And
> if so (I should probably read on first...), can't you have view
> installations *without* rebalancing ?
>

I'm not sure what benefits you would get from joining the cache view
but not receiving state compared to not sending the join request at
all. Non-members are allowed to send/receive commands, so in the
future we could even have separate "server" nodes (that join the cache
view) and "client" nodes (joining the JGroups cluster to send
commands, but not the cache view and so not holding any data except
L1).

My idea was that the cache view was a representation of the caches
that are able to service requests, so it doesn't make sense to include
in the view caches that don't hold data.

As a practical note, if we add nodes to the consistent hash as they
come along, a key will have one set of owners if written to before the
join and another set of owner if written to after the join. How should
another node determine the current location of the key, especially
considering that there can be many view changes before rebalancing is
triggerred?

> - Do we need the complex PREPARE_VIEW / ROLLBACK_VIEW / COMMIT_VIEW 2PC
> handling ? This adds a lot of complexity. Is it only used when we have a
> transactional cache ?
>

Nope, this doesn't have anything to do with transactional caches,
instead it is all about computing the owner that will push the key
during the rebalance operation.

In order to do it deterministically we need to have a common "last
good consistent hash" for the last rebalance that finished
successfully, and each node must determine if it should push a key or
not based on that last good CH.

A rebalance operation can also fail for various reasons (e.g. the
coordinator died). If that happens the new owners won't have all the
state, so they should not receive requests for the state that they
would have had in the pending CH.

> - State is to be transferred *within* this 2PC time frame. Hmm, again,
> this ties rebalancing and view installation together (see my argument
> above)...
>

If view installation wasn't tied to state transfer then we'd have to
keep yet the last rebalanced view somewhere else. We would hold the
"last pending view" (pending rebalance, that is) in the
CacheViewsManager and the "last rebalanced view" in another component,
and that component would have it's own mechanism for synchronizing the
"last rebalanced view" among cache members. So I think the 2PC
approach in CacheViewsManager actually simplifies things.

>
> On 9/27/11 6:22 PM, Dan Berindei wrote:
>> Following the discussions last week I've written up a wiki page
>> describing the strategies for cache view management and state transfer
>> that will enable asymmetric caches and manual rehashing:
>> http://community.jboss.org/wiki/AsymmetricCachesAndManualRehashingDesign
>>
>> The state transfer part is not very detailed, as you'll see we want to
>> go with a non-blocking approach but I'm not sure we can finish that
>> for 5.1 so we're keeping a blocking fallback option.
>>
>> Your comments are welcome, either on the list or on the wiki page.
>
>
> --
> Bela Ban
> Lead JGroups (http://www.jgroups.org)
> JBoss / Red Hat
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>

Cheers
Dan