[infinispan-dev] Partial state transfer in Infinispan

[Bela Ban]

On 6/1/11 4:21 PM, Sanne Grinovero wrote:
> Hi Bela,
>
> 2011/6/1 Bela Ban<bban at redhat.com>:
>> We currently use JGroups' partial state transfer to transfer individual
>> caches from one Infinispan instance to another.
>>
>> Since I got rid of partial state transfer in JGroups 3.0, and don't like
>> to add it back, I'd like to know whether this is still needed.
>>
>> I thought that we currently require the same set of caches to be
>> available in all Infinispan instances, and the reason (IIRC) was that
>> distribution wouldn't work if we have caches 1 and 2 available on
>> instances A and B, but not on C, because consistent hashing distributes
>> the data based on views, and we didn't want to have to keep track of
>> individual caches...
>
> Well I really don't like this limitation in Infinispan and was
> actually hoping that we could remove it at some point.
> Imagine the scenario in which you have a running cluster, and at some
> point the new release of your application needs an additional cache:
> there's no way to start a new node having this new cache.


Yes, I fully agree. Another example is HTTP web sessions: currently 1 
webapp == 1 cache, so we currently require the same webapps to be 
deployed in all JBoss instances *if* we use replication (distribution is 
different) !


> Also right now when an application starts, it's possible with the
> proper timing that it joins the cluster before having defined and
> started all caches (starting the cachemanager and the caches is not an
> atomic operation), basically failing to start because of this
> limitation.


Yep


> Maybe it's still possible to build such a thing on top of non-partial
> state transfer? As it doesn't exist, we didn't design it.


Yes. Well, Infinispan already uses its own state transfer for 
distribution, I wonder why this isn't the case for replication.


>> Why are we actually using JGroups' state transfer with replication, but
>> use our own state transfer with distribution ?
>
> I don't know, but guess it's because each node has a different set of
> keys so no node has the same state as another ?

You could still use JGroups state transfer; getState() would list the 
state provider as target node.

In general, partial state transfer involves transferring (1) the partial 
state and (2) the digest, which is a vector of highest seqnos seen for 
every member. When we get a partial state, we always overwrite our own 
digest with the one received, and update our state accordingly. However, 
when this is done a couple of times, for different partial states, I'm 
not sure that we won't receive a few messages multiple times, due to the 
digest overwriting...

I think the cleanest solution would be for you guys to reuse the state 
transfer you already use for state transfer in distribution mode.

-- 
Bela Ban
Lead JGroups / Clustering Team
JBoss