Hi Dan,
the easiest way to me seems to treat the state transfer as a special
transaction that is TO-broadcast using the sequencer, as you have also
been suggesting in your email.
I guess that this way you may even get rid of the ST lock, as
transactions that request a commit after a ST is started will be
TO-delivered after the "ST transaction", which will:
a) start transfering state only after having waited for the
completion of the txs TO-delivered before the ST transaction, and
b) prevent the thread in charge of managing TO-delivered
transactions from processing transactions that are TO-delivered after
the ST transaction, until the ST transaction is finished on that node.
Let me try to clarify this by outlining a possible protocol:
A) the coordinator TO-broadcasts the PREPARE_VIEW
B) when this message is TO-delivered at a node n, the thread (say thread
t) that is responsible of managing incoming TO messages, enqueues itself
on the count-down latches associated with the transactions that are
being appied by the TO thread pool. This ensures that every node starts
transferring state only after having applied all the updates of the
previously TO delivered xacts.
C) The state transfer is activated at node n by the same thread t
responsible of processing incoming TO messages. This guarantees that no
updates will be performed while the state is being transferred
D) An ACK_VIEW is sent to the coordinator.
Notice that:
- in parallel application level threads may request to commit new
transactions, and TO-broadcast their requests, which will be however
enqueued by JGroups, as the thread "t" will not process them until it
has finalized ST.
- if you wanted to squeeze in an extra-optimization, after sending the
ACK_VIEW you may already let thread t conclude the ST transaction (after
having released any count-down latch that it may have acquired in phase
B above), without waiting for a COMMIT_VIEW. The rationale is that node
has finished transferring its portion of state, and may already start
processing the transactions that have been totally ordered after the ST
request. I admit however not to have carefully thought about the case of
view changes being cancelled, as I don't know exactly if/why/how you are
doing this.
Cheers,
Paolo
On 5/1/12 3:39 PM, Dan Berindei wrote:
Hi Pedro
We've talked about sending PREPARE_VIEW commands through SEQUENCER as
well, but it would only work if we could make it really synchronous
(i.e. we can get the response from all the targets). The state
transfer fails automatically when we notice another node leaving - so
having the state transfer succeed on the originator does not guarantee
success on the other nodes as well.
It would also be more inconvenient to use different protocols for each
cache, since the component that sends PREPARE_VIEW commands is global,
but that wouldn't be a show-stopper.
A representative failing test is
TotalOrderStateTransfer1PcTest.testSTWithWritingNonTxTest (it's
actually using implicit transactions, but it doesn't matter). There
are two ways I've seen it fail:
1. It appears that SEQUENCER doesn't start delivering TO messages
right away after a node joins. It's not very clear to me why this
happens, but I see in the logs (
http://pastie.org/3843863) that both
nodes receive the prepare message, yet only the SEQUENCER on the
originator logs that it delivered the message.
2. We're releasing the state transfer lock while waiting for the total
order thread to execute the prepare. This means there is a window
between the ST lock release in the user thread and the ST lock acquire
in the total order thread, where a node could join and start receiving
state - but not this prepare command. I suspect this could be fixed by
sending PREPARE_VIEW through SEQUENCER, but it is rather hard to
reproduce, I only saw it when I added lots of logging (above the
regular TRACE logging) to debug the first issue.
Cheers
Dan
On Mon, Apr 30, 2012 at 8:41 PM, Pedro Ruivo<pruivo(a)gsd.inesc-id.pt> wrote:
> Hi all,
>
> I've looked at the IRC meeting and I've noticed that the State Transfer
> is not working in the Total Order based protocol. I am wondering, if it
> is not enough to send the prepare view command in total order. In this
> way, all the nodes will deliver the command in the same order, and then,
> all the nodes are in the same state when the command is delivered.
>
> What do you think?
>
> If you have questions, I'm available to help.
>
> Cheers,
> Pedro Ruivo
>
> On 4/30/12 3:48 PM, Galder ZamarreƱo wrote:
>> Minutes of our last IRC meeting can be found here:
>>
http://transcripts.jboss.org/meeting/irc.freenode.org/infinispan/2012/inf...
>>
>> Cheers,
>> --
>> Galder ZamarreƱo
>> Sr. Software Engineer
>> Infinispan, JBoss Cache
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>>
https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
>
https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev