[infinispan-dev] IRC meeting

[Dan Berindei]

Hi guys

We're getting closer...

On Wed, May 2, 2012 at 10:53 AM, Pedro Ruivo <pruivo at gsd.inesc-id.pt> wrote:
> Hi Dan,
>
> comment inline :)
>
> Cheers,
> Pedro
>
> On 5/2/12 8:36 AM, Dan Berindei wrote:
>> Hi Paolo
>>
>> On Tue, May 1, 2012 at 8:13 PM, Paolo Romano<romano at inesc-id.pt>  wrote:
>>> Hi Dan,
>>>
>>> the easiest way to me seems to treat the state transfer as a special
>>> transaction that is TO-broadcast using the sequencer, as you have also
>>> been suggesting in your email.
>>>
>>> I guess that this way you may even get rid of the ST lock, as
>>> transactions that request a commit after a ST is started will be
>>> TO-delivered after the "ST transaction", which will:
>>>      a) start transfering state only after having waited for the
>>> completion of the txs TO-delivered before the ST transaction, and
>>>      b) prevent the thread in charge of managing TO-delivered
>>> transactions from processing transactions that are TO-delivered after
>>> the ST transaction, until the ST transaction is finished on that node.
>>>
>>> Let me try to clarify this by outlining a possible protocol:
>>> A) the coordinator TO-broadcasts the PREPARE_VIEW
>>> B) when this message is TO-delivered at a node n, the thread (say thread
>>> t) that is responsible of managing incoming TO messages, enqueues itself
>>> on the count-down latches associated with the transactions that are
>>> being appied by the TO thread pool. This ensures that every node starts
>>> transferring state only after having applied all the updates of the
>>> previously TO delivered xacts.
>>> C) The state transfer is activated at node n by the same thread t
>>> responsible of processing incoming TO messages. This guarantees that no
>>> updates will be performed while the state is being transferred
>> This part isn't very clear to me - aren't the tx latches necessary
>> exactly because there may be more than one thread processing incoming
>> TO messages?
> The tx latches is used when write skew is enabled and ispn is executing
> distributed transaction (via XA Resource). But, only one thread is
> delivering the message, and only in TotalOrderInterceptor, it puts the
> transaction in a thread pool.
>

Yeah, obviously you're both right... Mircea explained this to me some
time ago but I completely forgot about it.

> For example:
>
> thread T: delivers PrepareCommand with Tx invoking handle() in the
> CommandAwareRPCDispatcher.
> T: when Tx arrives to TotalOrderIntercerptor, it uses the latches to
> ensure the deliver order, and then puts Tx in the thread pool
> another thread T': start processing tx
> T: returns from handle() and picks the next transaction
>
> What Paolo is suggesting is:
> 1) sending PrepareView through Sequencer, in synchronous mode
>
> thread T: delivers PrepareView invoking handle()
> T: PrepareView, asks for the latches of previous transactions
> T: PrepareView waits in all of them //this will block thread T, so no
> transactions are deliver in this phase
> T: the same thread, executes the state transfer, pushing the state //the
> Apply State messages should be sent with OOB flag if possible
> T: when finish, it returns from handle() //jgroups will pick the return
> value and send it back to coordinator as a response. Paolo named ACK_VIEW

Ok, I thought you were using the in-VM communication between the user
thread and the TO thread because of some limitation of SEQUENCER, but
I realize now it was just an optimization to avoid receiving responses
from all the nodes. I think I can change JGroupsTransport to not mark
sync messages as OOB when totalOrder is also set to true (without
breaking anything else), so this should indeed work without any
changes to state transfer itself.

> T: pick the next transaction
>
> When JGroups receives all the responses, it will unblock the synchronous
> remote invocation and then the coordinator sends the Commit/Rollback View
>
> In my opinion, this should work.
>

I think this will properly block prepare commands, but not necessarily
commit/rollback commands for txs that have already been prepared -
since they could be OOB.

Am I missing something again?


Cheers
Dan