[infinispan-dev] PutForExternalRead consistency

[Dan Berindei]

That doesn't sound right, we don't keep any lock for the duration of the
replication. In non-tx mode, we have to do a RPC to the primary owner
before we acquire any key. So there's nothing stopping the PFER from
writing its value after a regular (sync) put when the put was initiated
after the PFER.


On Fri, Nov 22, 2013 at 2:49 PM, William Burns <mudokonman at gmail.com> wrote:

> I wonder if we are over analyzing this.  It seems the main issue is
> that the replication is done asynchronously.  Infinispan has many ways
> to be make something asynchronous.  In my opinion we just chose the
> wrong way.  Wouldn't it be easier to just change the PFER to instead
> of passing along the FORCE_ASYNCHRONOUS flag we instead just have the
> operation performed asynchronous using putIfAbsentAsync ?  This way
> the lock is held during the duration of the replication and should be
> consistent with other operations.  Also the user can regain control
> back faster as it doesn't even have to process the local interceptor
> chain.  We could also change the putForExternalRead method declaration
> to also return a NotifiableFuture<Void> or something so they know when
> the operation is completed (if they want).
>
>  - Will
>
> On Thu, Nov 21, 2013 at 9:54 AM, Dan Berindei <dan.berindei at gmail.com>
> wrote:
> >
> >
> >
> > On Thu, Nov 21, 2013 at 12:35 PM, Galder Zamarreño <galder at redhat.com>
> > wrote:
> >>
> >>
> >> On Nov 18, 2013, at 12:42 PM, Dan Berindei <dan.berindei at gmail.com>
> wrote:
> >>
> >> >
> >> >
> >> >
> >> > On Mon, Nov 18, 2013 at 9:43 AM, Galder Zamarreño <galder at redhat.com>
> >> > wrote:
> >> >
> >> > On Nov 14, 2013, at 1:20 PM, Pedro Ruivo <pedro at infinispan.org>
> wrote:
> >> >
> >> > > Hi,
> >> > >
> >> > > Simple question: shouldn't PFER ensure some consistency?
> >> > >
> >> > > I know that PFER is asynchronous but (IMO) it can create
> >> > > inconsistencies
> >> > > in the data. the primary owner replicates the PFER follow by a PUT
> >> > > (PFER
> >> > > is sent async log the lock is released immediately) for the same
> key,
> >> > > we
> >> > > have no way to be sure if the PFER is delivered after or before in
> all
> >> > > the backup owners.
> >> > >
> >> > > comments?
> >> >
> >> > Assuming that PFER and PUT happen in the same thread, we're normally
> >> > relying on the JGroups sequence of events to send the first, wait no
> >> > response, and then send the second put. That should guarantee order
> in which
> >> > puts are received in the other nodes, but after that yeah, there's a
> risk
> >> > that it could happen. PFER and PUT for a given key normally happen in
> the
> >> > same thread in cache heavy use cases such as Hibernate 2LC, but
> there's no
> >> > guarantee.
> >> >
> >> > I don't think that's correct. If the cache is synchronous, the PUT
> will
> >> > be sent as an OOB message, and as such it can be delivered on the
> target
> >> > before the previous PFER command. That's regardless of whether the
> PFER
> >> > command was sent as a regular or as an OOB message.
> >>
> >> ^ Hmmmm, that's definitely risky. I think we should make PFER local
> only.
> >>
> >> The fact that PFER is asynchronous is nice to have. IOW, if you read a
> >> value from a database and you want to store it in the cache for later
> read,
> >> the fact that it's replicated asynchronously is just so that other
> nodes can
> >> take advantage of the value being in the cache. Since it's asynchronous
> some
> >> nodes could fail to apply, but that's fine since you can go to the
> database
> >> and re-retrieve it from there. So, making PFER local only would be the
> >> degenerate case, where all nodes fail to apply except the local node,
> which
> >> is fine. This is better than having the reordering above.
> >>
> >> In a chat I had with Dan, he pointed out that having PFER local only
> would
> >> be problematic for DIST mode w/ L1 enabled, since the local write would
> not
> >> invalidate other nodes, but this is fine because PFER only really makes
> >> sense for situations where the Infinispan is used as a cache. So, if the
> >> data is in the DB, you might as well go there (1 network trip), as
> opposed
> >> to askign the other nodes for data and the database in the worst case (2
> >> network trips).
> >>
> >> PFER is really designed for replication or invalidation use cases, which
> >> are precisely the ones configured for Hibernate 2LC.
> >>
> >> Thoughts?
> >>
> >
> > +1 to make PFER local-only in replicated caches, but I now think we
> should
> > go all the way and disallow PFER completely in dist caches.
> >
> > I still think having L1 enabled would be a problem, because a regular
> put()
> > won't invalidate the entry on all the nodes that did a PFER for that key
> > (there are no requestors, and even if we assume that we do a remote get
> > before the PFER we'd still have race conditions).
> >
> > With L1 disabled, we have the problem that you mentioned: we're trying to
> > read the value from the proper owners, but we never write it to the
> proper
> > owners, so the hit ratio will be pretty bad. Using the SKIP_REMOTE_LOOKUP
> > flag on reads, we'll avoid the extra RPC in Infinispan, but that will
> make
> > the hit ratio even worse. E.g. in a 4-nodes cluster with numOwners=2, the
> > hit ratio will never go above 50%.
> >
> > I don't think anyone would use a cache knowing that its hit ratio can
> never
> > get above 50%, so we should just save ourselves some effort and stop
> > supporting PFER in DIST mode.
> >
> > Cheers
> > Dan
> >
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20131122/8fa3d8ae/attachment.html