[infinispan-dev] PutForExternalRead consistency

[Dan Berindei]

On Thu, Nov 21, 2013 at 12:35 PM, Galder Zamarreño <galder at redhat.com>wrote:

>
> On Nov 18, 2013, at 12:42 PM, Dan Berindei <dan.berindei at gmail.com> wrote:
>
> >
> >
> >
> > On Mon, Nov 18, 2013 at 9:43 AM, Galder Zamarreño <galder at redhat.com>
> wrote:
> >
> > On Nov 14, 2013, at 1:20 PM, Pedro Ruivo <pedro at infinispan.org> wrote:
> >
> > > Hi,
> > >
> > > Simple question: shouldn't PFER ensure some consistency?
> > >
> > > I know that PFER is asynchronous but (IMO) it can create
> inconsistencies
> > > in the data. the primary owner replicates the PFER follow by a PUT
> (PFER
> > > is sent async log the lock is released immediately) for the same key,
> we
> > > have no way to be sure if the PFER is delivered after or before in all
> > > the backup owners.
> > >
> > > comments?
> >
> > Assuming that PFER and PUT happen in the same thread, we're normally
> relying on the JGroups sequence of events to send the first, wait no
> response, and then send the second put. That should guarantee order in
> which puts are received in the other nodes, but after that yeah, there's a
> risk that it could happen. PFER and PUT for a given key normally happen in
> the same thread in cache heavy use cases such as Hibernate 2LC, but there's
> no guarantee.
> >
> > I don't think that's correct. If the cache is synchronous, the PUT will
> be sent as an OOB message, and as such it can be delivered on the target
> before the previous PFER command. That's regardless of whether the PFER
> command was sent as a regular or as an OOB message.
>
> ^ Hmmmm, that's definitely risky. I think we should make PFER local only.
>
> The fact that PFER is asynchronous is nice to have. IOW, if you read a
> value from a database and you want to store it in the cache for later read,
> the fact that it's replicated asynchronously is just so that other nodes
> can take advantage of the value being in the cache. Since it's asynchronous
> some nodes could fail to apply, but that's fine since you can go to the
> database and re-retrieve it from there. So, making PFER local only would be
> the degenerate case, where all nodes fail to apply except the local node,
> which is fine. This is better than having the reordering above.
>
> In a chat I had with Dan, he pointed out that having PFER local only would
> be problematic for DIST mode w/ L1 enabled, since the local write would not
> invalidate other nodes, but this is fine because PFER only really makes
> sense for situations where the Infinispan is used as a cache. So, if the
> data is in the DB, you might as well go there (1 network trip), as opposed
> to askign the other nodes for data and the database in the worst case (2
> network trips).
>
> PFER is really designed for replication or invalidation use cases, which
> are precisely the ones configured for Hibernate 2LC.
>
> Thoughts?
>
>
+1 to make PFER local-only in replicated caches, but I now think we should
go all the way and disallow PFER completely in dist caches.

I still think having L1 enabled would be a problem, because a regular put()
won't invalidate the entry on all the nodes that did a PFER for that key
(there are no requestors, and even if we assume that we do a remote get
before the PFER we'd still have race conditions).

With L1 disabled, we have the problem that you mentioned: we're trying to
read the value from the proper owners, but we never write it to the proper
owners, so the hit ratio will be pretty bad. Using the SKIP_REMOTE_LOOKUP
flag on reads, we'll avoid the extra RPC in Infinispan, but that will make
the hit ratio even worse. E.g. in a 4-nodes cluster with numOwners=2, the
hit ratio will never go above 50%.

I don't think anyone would use a cache knowing that its hit ratio can never
get above 50%, so we should just save ourselves some effort and stop
supporting PFER in DIST mode.

Cheers
Dan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20131121/1323a119/attachment-0001.html