[infinispan-dev] IRC meeting

Tue May 1 13:13:04 EDT 2012

Hi Dan,

the easiest way to me seems to treat the state transfer as a special 
transaction that is TO-broadcast using the sequencer, as you have also 
been suggesting in your email.

I guess that this way you may even get rid of the ST lock, as 
transactions that request a commit after a ST is started will be 
TO-delivered after the "ST transaction", which will:
     a) start transfering state only after having waited for the 
completion of the txs TO-delivered before the ST transaction, and
     b) prevent the thread in charge of managing TO-delivered 
transactions from processing transactions that are TO-delivered after 
the ST transaction, until the ST transaction is finished on that node.

Let me try to clarify this by outlining a possible protocol:
A) the coordinator TO-broadcasts the PREPARE_VIEW
B) when this message is TO-delivered at a node n, the thread (say thread 
t) that is responsible of managing incoming TO messages, enqueues itself 
on the count-down latches associated with the transactions that are 
being appied by the TO thread pool. This ensures that every node starts 
transferring state only after having applied all the updates of the 
previously TO delivered xacts.
C) The state transfer is activated at node n by the same thread t 
responsible of processing incoming TO messages. This guarantees that no 
updates will be performed while the state is being transferred
D) An ACK_VIEW is sent to the coordinator.

Notice that:
- in parallel application level threads may request to commit new 
transactions, and TO-broadcast their requests, which will be however 
enqueued by JGroups, as the thread "t" will not process them until it 
has finalized ST.
- if you wanted to squeeze in an extra-optimization, after sending the 
ACK_VIEW you may already let thread t conclude the ST transaction (after 
having released any count-down latch that it may have acquired in phase 
B above), without waiting for a COMMIT_VIEW. The rationale is that node 
has finished transferring its portion of state, and may already start 
processing the transactions that have been totally ordered after the ST 
request. I admit however not to have carefully thought about the case of 
view changes being cancelled, as I don't know exactly if/why/how you are 
doing this.

Cheers,

     Paolo

On 5/1/12 3:39 PM, Dan Berindei wrote:
> Hi Pedro
>
> We've talked about sending PREPARE_VIEW commands through SEQUENCER as
> well, but it would only work if we could make it really synchronous
> (i.e. we can get the response from all the targets). The state
> transfer fails automatically when we notice another node leaving - so
> having the state transfer succeed on the originator does not guarantee
> success on the other nodes as well.
>
> It would also be more inconvenient to use different protocols for each
> cache, since the component that sends PREPARE_VIEW commands is global,
> but that wouldn't be a show-stopper.
>
> A representative failing test is
> TotalOrderStateTransfer1PcTest.testSTWithWritingNonTxTest (it's
> actually using implicit transactions, but it doesn't matter). There
> are two ways I've seen it fail:
> 1. It appears that SEQUENCER doesn't start delivering TO messages
> right away after a node joins. It's not very clear to me why this
> happens, but I see in the logs (http://pastie.org/3843863) that both
> nodes receive the prepare message, yet only the SEQUENCER on the
> originator logs that it delivered the message.
> 2. We're releasing the state transfer lock while waiting for the total
> order thread to execute the prepare. This means there is a window
> between the ST lock release in the user thread and the ST lock acquire
> in the total order thread, where a node could join and start receiving
> state - but not this prepare command. I suspect this could be fixed by
> sending PREPARE_VIEW through SEQUENCER, but it is rather hard to
> reproduce, I only saw it when I added lots of logging (above the
> regular TRACE logging) to debug the first issue.
>
> Cheers
> Dan
>
>
> On Mon, Apr 30, 2012 at 8:41 PM, Pedro Ruivo<pruivo at gsd.inesc-id.pt>  wrote:
>> Hi all,
>>
>> I've looked at the IRC meeting and I've noticed that the State Transfer
>> is not working in the Total Order based protocol. I am wondering, if it
>> is not enough to send the prepare view command in total order. In this
>> way, all the nodes will deliver the command in the same order, and then,
>> all the nodes are in the same state when the command is delivered.
>>
>> What do you think?
>>
>> If you have questions, I'm available to help.
>>
>> Cheers,
>> Pedro Ruivo
>>
>> On 4/30/12 3:48 PM, Galder Zamarreño wrote:
>>> Minutes of our last IRC meeting can be found here:
>>> http://transcripts.jboss.org/meeting/irc.freenode.org/infinispan/2012/infinispan.2012-04-30-14.20.html
>>>
>>> Cheers,
>>> --
>>> Galder Zamarreño
>>> Sr. Software Engineer
>>> Infinispan, JBoss Cache
>>>
>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev