Re: [infinispan-dev] Distribution, take 2
Manik Surtani wrote:
On 20 Jul 2009, at 14:51, Bela Ban wrote:
> General comments:
> - It seems that this is a mechanism where you replicate to the
> repl_count instances 'next' to you, ie. similar to Buddy Replication ?
Yes. So if you consider the hash space on a clockwise-incrementing
wheel [1], nodes {A, B, C, D} each have fixed positions on this wheel
(perhaps by making use of the hash codes on their Addresses). So this
becomes deterministic. Any key (K1 ~ K4 in the diagram in [1]) can be
located by ascertaining their place on the hash wheel, moving
clockwise, and considering the first "repl_count" nodes encountered.
OK
> - Are you tying your rebalancing mechanism to the consistent hash
> implementation ? This would IMO be bad because it doesn't allow for
> plugging in of a different CH algorithm !
Unfortunately the steps I have outlined below does imply some
knowledge of the hash algorithm, to determine which nodes are affected
by a LEAVE event, to minimise rebalancing. Perhaps this can be
provided by the ConsistentHash interface, so that any new
implementation would need to provide a list of affected nodes when a
give node leaves.
If this is the case, then I'd make sure that both the CH implementation
*and* the rebalancing policy are pluggable, so you could write an
instance of both and use it.
For example, I can think of CH implementations that have 'geographical'
knowledge, e.g. hosts on which nodes are running and try to store keys
on nodes which are as 'far' apart from each other as possible, similar
to what you did with Buddy Replication where you make sure the buddies
are on different hosts, or not on blades within the same rack.
> What does 'installing a consisten hash instance' mean ?
Literally, it means swapping out the CH instance used on a given node
with an alternate one. A CH instance encapsulates a "node list" and
the logic to locate a key within the node list, using hash functions
to determine which nodes would hold a given key.
Ah, so I assume you're sending a new node list, but don't swap out the
CH *logic* which is always the same, right ?
See this interface here [2], and a default (if currently imperfect)
implementation [3]. CHs are immutable so when a new view is
available, a new CH instance with a new node list should be created.
In addition, aggregate CHs are allowed, to consider the union of two
different views. This is represented by a delegating CH impl, and is
used in step 5.2 when a JoinTask is kicked off on a new joiner.
>> 4.5. GetConsistentHashCommand - an RPC command that "asks" a
>> node to serialize and transmit across its current CH impl.
>
> Serialize what ? The state (I assume) ? Or the consistent hash ? How
> can you serialize a CH in the latter case ?
This refers to the CH impl. The CH impl that needs to be serialized
would typically be an instance of [3]. Since the CH impl just
contains a List of Addresses, this should be possible. This is
necessary so that the joiner knows the state of the cluster before it
joined, and is able to ask specific members for state.
Another option to this is to assume that the previous state is the
same as the new state, minus the new joiner. Saves on RPC calls.
OK, so why don't you simply use the View ? Or do you actually, on a view
change, call setCaches() with the list of addresses you got from the View ?
--
Bela Ban
Lead JGroups / Clustering Team
JBoss