[infinispan-dev] Prepare and Rollback messages order

[Mircea Markus]

On 31 May 2012, at 10:21, Sebastiano Peluso wrote:

> Hi all,
> 
> I have a question about the messages order guaranteed in the execution 
> of the 2PC for committing a transaction in Infinispan (all versions).
> 
> In particular I have noticed that both in replication and distribution 
> mode the RollbackCommand command is multicast by using a priority queue. 
> This means that:
>     1) In replication mode, the visitRollbackCommand method of the 
> ReplicationInterceptor class invokes the broadcastRpcCommand method on 
> the RpcManager by using a true value for the usePriorityQueue parameter;
>     2) In distribution mode, the visitRollbackCommand method of the 
> DistributionInterceptor class invokes the invokeRemotely method on the 
> RpcManager by using a true value for the usePriorityQueue parameter.
> 
> Therefore the JGroups' packets containing the RollbackCommand commands 
> will be marked with the org.jgroups.Message.OOB and will be sent and 
> received by means of the Jgroups' out-of-band (OOB) thread pool.
> 
> If I'm not wrong until now, I deduce that since OBB threads don't 
> guarantee ordering, then it can happen that a message M2 multicast by 
> node N1 after it has multicast message M1, can be delivered to a node N2 
> before message M1. So the sending order is M1 -> M2 and the delivery 
> order is M2 -> M1 [1].
yes, you're right.
> 
> If this is true as well, I can have runs in which the PreparedCommand 
> command of a transaction T involved in the execution of a 2PC, can be 
> received by a 2PC's participant node after a RollbackCommand command for 
> T, even if the JGroups' messages containing the PrepareCommand commands 
> are actually managed by means of the JGroups's regular thread pool.
The rollbackCommand is only sent once the prepare execution is acknowledged. So rollback should always arrive *after* prepare: and this ordering is not enforced at jgroups level, but by the infinispan logic.
> 
> For instance, we can consider a scenario in which a transaction T 
> requests a commit on node N0 and N0 multicasts the T's prepare message 
> on a large set of nodes (e.g. 100 nodes) S={N1,...,N100}. If the lock 
> acquisition fails on at least one of the 100 nodes, N0 can multicasts a 
> rollback message without waiting for the reply of the prepare from all 
> the nodes in S.
That's only possible if lockAcquisitionTimeout(default to 10secs) is higher than the replicationTimeout(15 secs by default).

> This means that the last rollback message can be 
> delivered to a node in S, say N100, before the T's prepare message and 
> that the locks successfully acquired by N100 for T will not be released 
> anymore.
> 
> Am I missing anything?
> 
> Thanks.
> 
> Best Regards,
> 
>     Sebastiano Peluso
> 
> 
> [1] http://www.jgroups.org/manual/html/user-advanced.html
> 
> 
> -- 
> Sebastiano Peluso, PhD student
> 
> Department of Computer, Control and Management Engineering
> Sapienza University of Rome
> Via Ariosto 25, 00185, Rome, Italy
> Tel. +39 06 77274108
> Webpage http://www.dis.uniroma1.it/~peluso
> 
> Distributed Systems Group, INESC-ID
> Instituto Superior Técnico
> Rua Alves Redol 9, 1000-029, Lisbon, Portugal
> Webpage http://www.gsd.inesc-id.pt/~peluso
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev