[infinispan-dev] DIST-SYNC, put(), a problem and a solution

[Bela Ban]

Hey Dan,

On 06/08/14 16:13, Dan Berindei wrote:
> I could create the issue in JIRA, but I wouldn't make it high priority
> because I think it have lots of corner cases with NBST and cause
> headaches for the maintainers of state transfer ;)

I do believe the put-while-holding-the-lock issue *is* a critical issue; 
anyone banging a cluster of Infinispan nodes with more than 1 thread 
will run into lock timeouts, with or without transactions. The only 
workaround for now is to use total order, but at the cost of reduced 
performance. However, once a system starts hitting the lock timeout 
issues, performance drops to a crawl, way slower than TO, and work 
starts to pile up, which compounds the problem.

I believe doing a sync RPC while holding the lock on a key is asking for 
trouble and is (IMO) an anti-pattern.

Sorry if this has a negative impact on NBST, but should we not fix this 
because we don't want to risk a change to NBST ?

> Besides, I'm still not sure I understood your proposals properly, e.g.
> whether they are meant only for non-tx caches or you want to change
> something for tx caches as well...

I think this can be used for both cases; however, I think either Sanne's 
solution of using seqnos *per key* and updating in the order of seqnos 
or using Pedro's total order impl are probably better solutions.

I'm not pretending these solutions are final (e.g. Sanne's solution 
needs more thought when multiple keys are involved), but we should at 
least acknowledge the issue exists, create a JIRA to prioritize it and 
then start discussing solutions.


> Cheers
> Dan
>
>
>
>
> On Wed, Aug 6, 2014 at 1:02 PM, Bela Ban <bban at redhat.com
> <mailto:bban at redhat.com>> wrote:
>
>     Seems like this discussion has died with the general agreement that this
>     is broken and with a few proposals on how to fix it, but without any
>     follow-up action items.
>
>     I think we (= someone from the ISPN team) need to create a JIRA,
>     preferably blocking.
>
>     WDYT ?
>
>     If not, here's what our options are:
>
>     #1 I'll create a JIRA
>
>     #2 We'll hold the team meeting in Krasnojarsk, Russia
>
>     #3 There will be only vodka, no beers in #2
>
>     #4 Bela will join the ISPN team
>
>     Thoughts ?


-- 
Bela Ban, JGroups lead (http://www.jgroups.org)