[infinispan-dev] PutForExternalRead consistency

[Pedro Ruivo]

I think Will has a point here. if we make the PFER sync but performed by 
other thread we make it available for REPL and DIST and will not destroy 
the consistency...

On 11/22/2013 12:49 PM, William Burns wrote:
> I wonder if we are over analyzing this.  It seems the main issue is
> that the replication is done asynchronously.  Infinispan has many ways
> to be make something asynchronous.  In my opinion we just chose the
> wrong way.  Wouldn't it be easier to just change the PFER to instead
> of passing along the FORCE_ASYNCHRONOUS flag we instead just have the
> operation performed asynchronous using putIfAbsentAsync ?  This way
> the lock is held during the duration of the replication and should be
> consistent with other operations.  Also the user can regain control
> back faster as it doesn't even have to process the local interceptor
> chain.  We could also change the putForExternalRead method declaration
> to also return a NotifiableFuture<Void> or something so they know when
> the operation is completed (if they want).
>
>   - Will
>
> On Thu, Nov 21, 2013 at 9:54 AM, Dan Berindei <dan.berindei at gmail.com> wrote:
>>
>>
>>
>> On Thu, Nov 21, 2013 at 12:35 PM, Galder Zamarreño <galder at redhat.com>
>> wrote:
>>>
>>>
>>> On Nov 18, 2013, at 12:42 PM, Dan Berindei <dan.berindei at gmail.com> wrote:
>>>
>>>>
>>>>
>>>>
>>>> On Mon, Nov 18, 2013 at 9:43 AM, Galder Zamarreño <galder at redhat.com>
>>>> wrote:
>>>>
>>>> On Nov 14, 2013, at 1:20 PM, Pedro Ruivo <pedro at infinispan.org> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Simple question: shouldn't PFER ensure some consistency?
>>>>>
>>>>> I know that PFER is asynchronous but (IMO) it can create
>>>>> inconsistencies
>>>>> in the data. the primary owner replicates the PFER follow by a PUT
>>>>> (PFER
>>>>> is sent async log the lock is released immediately) for the same key,
>>>>> we
>>>>> have no way to be sure if the PFER is delivered after or before in all
>>>>> the backup owners.
>>>>>
>>>>> comments?
>>>>
>>>> Assuming that PFER and PUT happen in the same thread, we're normally
>>>> relying on the JGroups sequence of events to send the first, wait no
>>>> response, and then send the second put. That should guarantee order in which
>>>> puts are received in the other nodes, but after that yeah, there's a risk
>>>> that it could happen. PFER and PUT for a given key normally happen in the
>>>> same thread in cache heavy use cases such as Hibernate 2LC, but there's no
>>>> guarantee.
>>>>
>>>> I don't think that's correct. If the cache is synchronous, the PUT will
>>>> be sent as an OOB message, and as such it can be delivered on the target
>>>> before the previous PFER command. That's regardless of whether the PFER
>>>> command was sent as a regular or as an OOB message.
>>>
>>> ^ Hmmmm, that's definitely risky. I think we should make PFER local only.
>>>
>>> The fact that PFER is asynchronous is nice to have. IOW, if you read a
>>> value from a database and you want to store it in the cache for later read,
>>> the fact that it's replicated asynchronously is just so that other nodes can
>>> take advantage of the value being in the cache. Since it's asynchronous some
>>> nodes could fail to apply, but that's fine since you can go to the database
>>> and re-retrieve it from there. So, making PFER local only would be the
>>> degenerate case, where all nodes fail to apply except the local node, which
>>> is fine. This is better than having the reordering above.
>>>
>>> In a chat I had with Dan, he pointed out that having PFER local only would
>>> be problematic for DIST mode w/ L1 enabled, since the local write would not
>>> invalidate other nodes, but this is fine because PFER only really makes
>>> sense for situations where the Infinispan is used as a cache. So, if the
>>> data is in the DB, you might as well go there (1 network trip), as opposed
>>> to askign the other nodes for data and the database in the worst case (2
>>> network trips).
>>>
>>> PFER is really designed for replication or invalidation use cases, which
>>> are precisely the ones configured for Hibernate 2LC.
>>>
>>> Thoughts?
>>>
>>
>> +1 to make PFER local-only in replicated caches, but I now think we should
>> go all the way and disallow PFER completely in dist caches.
>>
>> I still think having L1 enabled would be a problem, because a regular put()
>> won't invalidate the entry on all the nodes that did a PFER for that key
>> (there are no requestors, and even if we assume that we do a remote get
>> before the PFER we'd still have race conditions).
>>
>> With L1 disabled, we have the problem that you mentioned: we're trying to
>> read the value from the proper owners, but we never write it to the proper
>> owners, so the hit ratio will be pretty bad. Using the SKIP_REMOTE_LOOKUP
>> flag on reads, we'll avoid the extra RPC in Infinispan, but that will make
>> the hit ratio even worse. E.g. in a 4-nodes cluster with numOwners=2, the
>> hit ratio will never go above 50%.
>>
>> I don't think anyone would use a cache knowing that its hit ratio can never
>> get above 50%, so we should just save ourselves some effort and stop
>> supporting PFER in DIST mode.
>>
>> Cheers
>> Dan
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>