Re: [infinispan-dev] Distribution, take 2

Manik Surtani wrote: > > On 20 Jul 2009, at 14:51, Bela Ban wrote: > >> General comments: >> - It seems that this is a mechanism where you replicate to the >> repl_count instances 'next' to you, ie. similar to Buddy >> Replication ? > > Yes. So if you consider the hash space on a clockwise-incrementing > wheel [1], nodes {A, B, C, D} each have fixed positions on this > wheel (perhaps by making use of the hash codes on their > Addresses). So this becomes deterministic. Any key (K1 ~ K4 in > the diagram in [1]) can be located by ascertaining their place on > the hash wheel, moving clockwise, and considering the first > "repl_count" nodes encountered. OK > >> - Are you tying your rebalancing mechanism to the consistent hash >> implementation ? This would IMO be bad because it doesn't allow >> for plugging in of a different CH algorithm ! > > Unfortunately the steps I have outlined below does imply some > knowledge of the hash algorithm, to determine which nodes are > affected by a LEAVE event, to minimise rebalancing. Perhaps this > can be provided by the ConsistentHash interface, so that any new > implementation would need to provide a list of affected nodes when > a give node leaves. If this is the case, then I'd make sure that both the CH implementation *and* the rebalancing policy are pluggable, so you could write an instance of both and use it. For example, I can think of CH implementations that have 'geographical' knowledge, e.g. hosts on which nodes are running and try to store keys on nodes which are as 'far' apart from each other as possible, similar to what you did with Buddy Replication where you make sure the buddies are on different hosts, or not on blades within the same rack.

>> What does 'installing a consisten hash instance' mean ? > > Literally, it means swapping out the CH instance used on a given > node with an alternate one. A CH instance encapsulates a "node > list" and the logic to locate a key within the node list, using > hash functions to determine which nodes would hold a given key. Ah, so I assume you're sending a new node list, but don't swap out the CH *logic* which is always the same, right ?

See this interface here [2], and a default (if currently imperfect) implementation [3]. CHs are immutable so when a new view is available, a new CH instance with a new node list should be created. > > In addition, aggregate CHs are allowed, to consider the union of > two different views. This is represented by a delegating CH impl, > and is used in step 5.2 when a JoinTask is kicked off on a new > joiner. > >>> 4.5. GetConsistentHashCommand - an RPC command that "asks" a >>> node to serialize and transmit across its current CH impl. >> >> Serialize what ? The state (I assume) ? Or the consistent hash ? >> How can you serialize a CH in the latter case ? > > This refers to the CH impl. The CH impl that needs to be > serialized would typically be an instance of [3]. Since the CH > impl just contains a List of Addresses, this should be possible. > This is necessary so that the joiner knows the state of the cluster > before it joined, and is able to ask specific members for state. > > Another option to this is to assume that the previous state is the > same as the new state, minus the new joiner. Saves on RPC calls. OK, so why don't you simply use the View ? Or do you actually, on a view change, call setCaches() with the list of addresses you got from the View ?