[infinispan-dev] put if absent not atomic: https://jira.jboss.org/jira/browse/ISPN-236

[Mircea Markus]

Hi Manik,

While the fix I made for EntityWrapper fixed the cache in local mode,  
the replicated mode[1] (pretty sure dist is the same) is far from  
working.
The issue is the following:
node1.put(k,v2) :: WL(k1) -> releaseLock(k1) -> replicate(k1) ->  
incoming_thread_performs put (k, *v2*)
node2.put(k,v2) :: WL(k2) -> releaseLock(k2) -> replicate(k2) ->  
incoming_thread_performs put (k, *v1*)

now in this scenario both operation succeed! This is caused by the  
fact that lock interceptor is *after* repl interceptor in the chain,  
so when repl is triggered the key is no longer locked.
In other words, if the puts happen at the same time, on the same key  
then both might succeed, leaving the cluster in an inconsistent state.
IMO, the correct approach would be to keep the lock while doing the  
replication. This would reduce the concurrency though, as locks would  
be held for longer amounts of time. But the data consistency would  
increase. (this would be an easy fix for Repl, but dist relies on  
interceptor's order).

I've also tried doing the call within a tx, but there's a deadlock -  
I'll investigate this on Mon.

[1] I know disyt is what we'll use but problem exists for repl as well

Cheers,
Mircea