numOwners==2 is and will very likely remain the most common case,
particularly for small clusters.
But if we have two sites, it makes sense to configure 2 owners per
site. If only one node goes down, the surviving owner will supply
state to the new owner. If both nodes go down, the new owners will
fetch the data from the other site. So while 2 nodes going down will
be quite costly, it should be infrequent enough that it's worth
optimizing for the more frequent "1 node goes down and than comes
back
up" case.
Agreed; this mixed batching (leaves with joins) makes sense for non-site clusters
as well.
> For total shutdown, I guess we can use other means that rehash,
> e.g. a specific command that would disable it and start flushing
> to the store.
>
I think just stopping the cache is enough to get it to flush data to
the store with passivation enabled.
ATM, wouldn't the shutdown of a cluster of
servers trigger a rehash storm?
But for now any data saved to a
private store in distributed mode is useless after restart, because
we
have no safe way to push data that we don't own to other nodes (and
by
safe I mean avoiding overwriting newer data or resurrecting deleted
data).
I think that should work with a clustered cache store though.