[infinispan-dev] State Transfer & Invalidation Caches

[William Burns]

On Wed, Jun 17, 2015 at 3:08 PM Sanne Grinovero <sanne at infinispan.org>
wrote:

> Hi Will,
> thanks for asking! It's a big decision indeed, and I might have some
> objections :) although I can be easily persuaded otherwise as I'm not
> really sure how it's currently implemented.
>

Yeah that is part of it is I am not sure the original intent for
invalidation mode.  We all know it gets left ignored in regards to our
attention, unless something goes wrong :P


>
> My doubts would be around cache state consistency across nodes. I
> agree with you that state transfer is unexpected in the scope of
> invalidation, but the consistent hash distribution is also used to
> identify the "primary owner" for each entry - which I think still is
> an essential role for example to implement atomic operations and
> locking.
>

That is the part that is odd, sorry I didn't go into more details.

Atomic operations are only guaranteed for the local node data in nontx or
tx mode.  For example if you have 2 nodes and only node1 has a value for
key1.  If a replace(key1, newvalue) is ran on node2 it will fail since only
node1 has a value for key1.

Locks are the thing that are really weird:
- No remote locks are acquired for a nontx cache.

- Remote locks are only acquired for tx caches.  Since data modification is
already protected by just a local lock, this only serves to prevent 2
invalidation messages to be sent from different nodes at the same time.

The usefulness of remote locks in tx mode seems dubious as it will slow
down uncontended writes since it will require replicating the lock
information to all owners (includes backups) and having to include this
owner in the tx commit handling to clean up said lock. We also have to
handle a topology change to move the lock information between nodes.

Maybe there is some use case here I am not aware of with invalidation and
tx mode locking on remote nodes?


>
> One might say that such things are not needed for some use cases, but
> I think the contract of Invalidation with a client is around the lines
> of "it's ok to return null if you don't have a local copy" but it's
> not relaxing the consistency and transactional guarantees which we
> normally provide with other cache modes.
> We could of course argue if it could be useful to provide a relaxed
> consistency mode, but I'd really not drop the stronger model.
>
> -- Sanne
>
>
> On 17 June 2015 at 19:32, William Burns <mudokonman at gmail.com> wrote:
> > Recently [1] was found.  The underlying cause is that now initial state
> > transfer is enabled by default including invalidation caches.
> >
> > When thinking about this I didn't even realize that we utilize a
> Replicated
> > consistent hash for invalidation mode.  This all strikes me as just
> wrong.
> > To me an invalidation cache is just a local cache that sends broadcasted
> > invalidation messages to other members.  It shouldn't have any need for a
> > consistent hash.
> >
> > Currently though we have an invalidation cache that will try to acquire
> > remote locks [2], reports ownership of keys by the consistent hash
> (instead
> > of each node owning every key as it would be in a local cache [3]), which
> > will participate in initial state transfer (REPL based) [4] among other
> > things.  We even have invalidation commands that are tied to given
> > topologies [5].
> >
> > I was originally planning on cutting the cord between invalidation caches
> > and state transfer, but I am finding so many places that assume we have a
> > topology that it requires more code that I would have thought.  So I
> wanted
> > to check if anyone had any objections.  If there are objections I can
> end up
> > taking the quick and easy route of just forcing disabling of initial
> state
> > transfer and leaving everything as it is otherwise.  I personally feel we
> > should completely separate these concepts to guarantee what I would
> think as
> > proper behavior.
> >
> > Thanks,
> >
> >  - Will
> >
> > [1] https://issues.jboss.org/browse/ISPN-5560
> > [2]
> >
> https://github.com/infinispan/infinispan/blob/master/core/src/test/java/org/infinispan/lock/InvalidationModePessimisticLockReleaseTest.java
> > [3]
> >
> https://github.com/infinispan/infinispan/blob/master/core/src/main/java/org/infinispan/interceptors/locking/ClusteringDependentLogic.java#L300
> > [4]
> >
> https://github.com/infinispan/infinispan/blob/master/core/src/test/java/org/infinispan/statetransfer/NonTxStateTransferInvalidationTest.java#L56
> > [5] https://issues.jboss.org/browse/ISPN-4016
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20150617/38e10931/attachment.html