[infinispan-dev] Non-blocking state transfer (ISPN-1424)

[Bela Ban]

Wow !

Does this need to be so complex ? I've spent a hour trying to understand 
it, and am still overwhelmed... :-)

My understanding (based on my changed in 4.2) is that state transfer 
moves/deletes keys based on the diff between 2 subsequent views:
- Each node checks all of the affected keys
- If a key should be stored in additional nodes, the key is pushed there
- If a key shouldn't be stored locally anymore, it is removed

IMO, there's no need to handle a merge differently from a regular view, 
and we might end up with inconsistent state, but that's unavoidable 
until we have eventual consistency. Fine...

Also, why do we need to transfer ownership information ? Can't ownership 
be calculated purely on local information ?

I'm afraid that the complexity will increase the state space (hard to 
test all possible state transitions), lead to unnecessary messages being 
sent and most importantly, might lead to blocks.

The section on locking outright scares me :-) Perhaps reducing the level 
of details here - as Galder suggested - might help to understand the 
basic design.

Sorry for being a bit negative, but I think state transfer is one of the 
most critical and important pieces of code in DIST mode, and this needs 
to work against large (say a couple of hundreds) clusters and nodes 
joining, leaving or crashing all the times...

I'm going to re-read the design again, maybe what I said above is just 
BS ... :-)


On 3/8/12 11:55 AM, Dan Berindei wrote:
> Hi guys
>
> It's been a long time coming, but I finally published the non-blocking
> state transfer draft on the wiki:
> https://community.jboss.org/wiki/Non-blockingStateTransfer
>
> Unlike my previous state transfer design document, I think I've
> fleshed out most of the implications. Still, there are some things I
> don't have a clear solution for yet. As you would expect it's mostly
> around merging and delayed state transfer.
>
> I'm looking forward to hearing your comments/advice!
>
> Cheers
> Dan
>
> PS: Let's discuss this over the mailing list only.
>
-- 
Bela Ban, JGroups lead (http://www.jgroups.org)