[infinispan-dev] State transfer improvements for 5.2?

[Paolo Romano]

On 1/26/12 5:43 PM, Dan Berindei wrote:
> On Thu, Jan 26, 2012 at 7:17 PM, Manik Surtani<manik at jboss.org>  wrote:
>> On 26 Jan 2012, at 16:05, Sanne Grinovero wrote:
>>
>>> On 26 January 2012 10:36, Manik Surtani<manik at jboss.org>  wrote:
>>>> Dan,
>>>>
>>>> Looking through what we currently have, given the blocking nature of state transfer, I wonder if we can improve on it by pausing the state transfer from time to time to flush waiting transactions?  I.e., if state transfer is in progress and a write command or transaction is waiting for it to finish, and since state transfer is chunked anyway, perhaps to let a few transactions through in between transferring chunks?
>>> Wondering about my proposal about transfering values separately from
>>> locks not being easier to implement? Or likely same effort, but
>>> better..
>> Well, thats what I want to try and understand.  If periodic flushing is easier/quicker/less risky to implement, I think we could do it for 5.2.0.  :)
>>
> I don't think it would be any easier/less risky - the moment we allow
> transactions to go through our data container iterator is no longer
> valid and we could e.g. push data that's been already deleted on the
> old owner.
>
> The only improvement to the current blocking state transfer that I
> think would be easy enough is
> https://issues.jboss.org/browse/ISPN-1731 - when we acquire a key lock
> we'd have to try first with 0 timeout and then when we wait with a
> non-zero timeout we'd have to register our thread so that state
> transfer can interrupt it.
>
> In fact, since non-blocking state transfer also requires threads to
> line up at a barrier at some point, this should help non-blocking
> state transfer as well.
>
> Actually there is something else, but I'd consider it more of a bug.
> With optimistic transactions we retry the prepare after it failed to
> acquire the state transfer lock, but with pessimistic transactions I'm
> not sure the retry works properly.
>
>> Sanne, have you documented your design on separate transfer of state vs locks on the wiki somewhere?  I'd be curious to see what Paolo and the others at INESC and CINI think about it as well.  :)
>>
> I wrote a paragraph about it here:
> https://community.jboss.org/wiki/AsymmetricCachesAndManualRehashingDesign
> I'm not sure if Sanne validated it...
>
> Cheers
> Dan
Thanks Manik, Dan.

In fact this is something very interesting for us (especially 
Sebastiano, who's reading in cc).

Although I cannot promise that we'll be able to be very quick in sending 
a feedback as we're swamped with several deadlines in this moment.

Cheers,

     Paolo