<div dir="ltr"><div>That doesn't sound right, we don't keep any lock for the duration of the replication. In non-tx mode, we have to do a RPC to the primary owner before we acquire any key. So there's nothing stopping the PFER from writing its value after a regular (sync) put when the put was initiated after the PFER.</div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Nov 22, 2013 at 2:49 PM, William Burns <span dir="ltr"><<a href="mailto:mudokonman@gmail.com" target="_blank">mudokonman@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I wonder if we are over analyzing this. It seems the main issue is<br>
that the replication is done asynchronously. Infinispan has many ways<br>
to be make something asynchronous. In my opinion we just chose the<br>
wrong way. Wouldn't it be easier to just change the PFER to instead<br>
of passing along the FORCE_ASYNCHRONOUS flag we instead just have the<br>
operation performed asynchronous using putIfAbsentAsync ? This way<br>
the lock is held during the duration of the replication and should be<br>
consistent with other operations. Also the user can regain control<br>
back faster as it doesn't even have to process the local interceptor<br>
chain. We could also change the putForExternalRead method declaration<br>
to also return a NotifiableFuture<Void> or something so they know when<br>
the operation is completed (if they want).<br>
<br>
- Will<br>
<div class="HOEnZb"><div class="h5"><br>
On Thu, Nov 21, 2013 at 9:54 AM, Dan Berindei <<a href="mailto:dan.berindei@gmail.com">dan.berindei@gmail.com</a>> wrote:<br>
><br>
><br>
><br>
> On Thu, Nov 21, 2013 at 12:35 PM, Galder Zamarreño <<a href="mailto:galder@redhat.com">galder@redhat.com</a>><br>
> wrote:<br>
>><br>
>><br>
>> On Nov 18, 2013, at 12:42 PM, Dan Berindei <<a href="mailto:dan.berindei@gmail.com">dan.berindei@gmail.com</a>> wrote:<br>
>><br>
>> ><br>
>> ><br>
>> ><br>
>> > On Mon, Nov 18, 2013 at 9:43 AM, Galder Zamarreño <<a href="mailto:galder@redhat.com">galder@redhat.com</a>><br>
>> > wrote:<br>
>> ><br>
>> > On Nov 14, 2013, at 1:20 PM, Pedro Ruivo <<a href="mailto:pedro@infinispan.org">pedro@infinispan.org</a>> wrote:<br>
>> ><br>
>> > > Hi,<br>
>> > ><br>
>> > > Simple question: shouldn't PFER ensure some consistency?<br>
>> > ><br>
>> > > I know that PFER is asynchronous but (IMO) it can create<br>
>> > > inconsistencies<br>
>> > > in the data. the primary owner replicates the PFER follow by a PUT<br>
>> > > (PFER<br>
>> > > is sent async log the lock is released immediately) for the same key,<br>
>> > > we<br>
>> > > have no way to be sure if the PFER is delivered after or before in all<br>
>> > > the backup owners.<br>
>> > ><br>
>> > > comments?<br>
>> ><br>
>> > Assuming that PFER and PUT happen in the same thread, we're normally<br>
>> > relying on the JGroups sequence of events to send the first, wait no<br>
>> > response, and then send the second put. That should guarantee order in which<br>
>> > puts are received in the other nodes, but after that yeah, there's a risk<br>
>> > that it could happen. PFER and PUT for a given key normally happen in the<br>
>> > same thread in cache heavy use cases such as Hibernate 2LC, but there's no<br>
>> > guarantee.<br>
>> ><br>
>> > I don't think that's correct. If the cache is synchronous, the PUT will<br>
>> > be sent as an OOB message, and as such it can be delivered on the target<br>
>> > before the previous PFER command. That's regardless of whether the PFER<br>
>> > command was sent as a regular or as an OOB message.<br>
>><br>
>> ^ Hmmmm, that's definitely risky. I think we should make PFER local only.<br>
>><br>
>> The fact that PFER is asynchronous is nice to have. IOW, if you read a<br>
>> value from a database and you want to store it in the cache for later read,<br>
>> the fact that it's replicated asynchronously is just so that other nodes can<br>
>> take advantage of the value being in the cache. Since it's asynchronous some<br>
>> nodes could fail to apply, but that's fine since you can go to the database<br>
>> and re-retrieve it from there. So, making PFER local only would be the<br>
>> degenerate case, where all nodes fail to apply except the local node, which<br>
>> is fine. This is better than having the reordering above.<br>
>><br>
>> In a chat I had with Dan, he pointed out that having PFER local only would<br>
>> be problematic for DIST mode w/ L1 enabled, since the local write would not<br>
>> invalidate other nodes, but this is fine because PFER only really makes<br>
>> sense for situations where the Infinispan is used as a cache. So, if the<br>
>> data is in the DB, you might as well go there (1 network trip), as opposed<br>
>> to askign the other nodes for data and the database in the worst case (2<br>
>> network trips).<br>
>><br>
>> PFER is really designed for replication or invalidation use cases, which<br>
>> are precisely the ones configured for Hibernate 2LC.<br>
>><br>
>> Thoughts?<br>
>><br>
><br>
> +1 to make PFER local-only in replicated caches, but I now think we should<br>
> go all the way and disallow PFER completely in dist caches.<br>
><br>
> I still think having L1 enabled would be a problem, because a regular put()<br>
> won't invalidate the entry on all the nodes that did a PFER for that key<br>
> (there are no requestors, and even if we assume that we do a remote get<br>
> before the PFER we'd still have race conditions).<br>
><br>
> With L1 disabled, we have the problem that you mentioned: we're trying to<br>
> read the value from the proper owners, but we never write it to the proper<br>
> owners, so the hit ratio will be pretty bad. Using the SKIP_REMOTE_LOOKUP<br>
> flag on reads, we'll avoid the extra RPC in Infinispan, but that will make<br>
> the hit ratio even worse. E.g. in a 4-nodes cluster with numOwners=2, the<br>
> hit ratio will never go above 50%.<br>
><br>
> I don't think anyone would use a cache knowing that its hit ratio can never<br>
> get above 50%, so we should just save ourselves some effort and stop<br>
> supporting PFER in DIST mode.<br>
><br>
> Cheers<br>
> Dan<br>
><br>
><br>
</div></div><div class="HOEnZb"><div class="h5">> _______________________________________________<br>
> infinispan-dev mailing list<br>
> <a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
> <a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
<br>
_______________________________________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
</div></div></blockquote></div><br></div>