[jbosscache-dev] ClusteredCacheLoader deadlocks and JBCCACHE-1103

[Manik Surtani]

Yes, it would timeout, but that is better than the deadlock that  
currently occurs.  IMO I think the timeout is a valid response to  
such a call.  If the CCL cannot complete a remote call because of a  
remote lock, it should timeout.  And when it does, it releases the  
lock on Fqn-A and the update originating remotely can proceed.

The problem with making the CCL call without a lock is that this  
exposes concurrent loading and overwriting for all cache loaders (the  
CCL is treated as a simple cache loader impl by the  
CacheLoaderInterceptor).  This also has implications with race  
conditions on eviction (Without locks in the cache loader  
interceptor, the following could happen: Thread-1 does a get(), goes  
through the cache loader interceptor, sees the node requested in  
memory and does not load.  Eviction-thread gets a WL on the same node  
and evicts it.  Thread-1 now gets to the PessimisticLockInterceptor,  
cannot find the node, and since this is a get() call, doesn't create  
the node but returns a null)


On 15 Jun 2007, at 08:09, Bela Ban wrote:

> I looked at 1103 in a bit more detail and concluded that the change  
> in JGroups (http://jira.jboss.com/jira/browse/JGRP-533) would not  
> help. The underlying issue is that (a) 2.4.1 has a single incoming  
> request queue and (b) the ClusteredCacheLoader holds a lock while  
> making a cluster-wide call. Let's look at an example:
>
>   1. The CCL acquires a lock on Fqn-A and makes a cluster-wide call
>   2. We get a replication message for A (RM-A), so someone made an
>      update to A and is now trying to commit the change. RM-A tries to
>      acquire the lock on A, but is blocked because the CCL holds it.
>   3. Now a result for the CCL call arrives. It is not processed  
> (single
>      queue) until RM-A gets processed. However, that's not the case
>      until the CCL call completes. In this case, the only way for the
>      CCL call to complete is via a timeout, as it will never get its
>      results.
>
> So even if I implemented 533, it wouldn't help, as the interleaving  
> between CCL calls and RM messages for the same FQNs would lead to  
> timeouts.
>
> Now, a possible solution to 1103 is that we make the CCL call  
> *without* holding a lock. When we get the result(s), only *then* do  
> we acquire a lock and update the FQN. We also need to check whether  
> FQN-A was updated in the mean time and then decide which value to  
> return (the value set by the RM or the one gotten from the CCL call).
>
> WDYT ?
>
> -- 
> Bela Ban
> Lead JGroups / JBoss Clustering team
> JBoss - a division of Red Hat

--
Manik Surtani

Lead, JBoss Cache
JBoss, a division of Red Hat