[infinispan-dev] DIST.retrieveFromRemoteSource

[Dan Berindei]

On Wed, Jan 25, 2012 at 4:22 PM, Mircea Markus <mircea.markus at jboss.com> wrote:
>
> One node might be busy doing GC and stay unresponsive for a whole
>
> second or longer, another one might be actually crashed and you didn't
>
> know that yet, these are unlikely but possible.
>
> All these are possible but I would rather consider them as exceptional
> situations, possibly handled by a retry logic. We should *not* optimise for
> that these situations IMO.
>

As Sanne pointed out, an exceptional situation on a node becomes
ordinary with 100s or 1000s of nodes.
So the default policy should scale the initial number of requests with
numOwners.

>
> More likely, a rehash is in progress, you could then be asking a node
>
> which doesn't yet (or anymore) have the value.
>
>
> this is a consistency issue and I think we can find a way to handle it some
> other way.
>

With the current state transfer we always send ClusteredGetCommands to
the old owners (and only the old owners). If a node didn't receive the
entire state, it means that state transfer hasn't finished yet and the
CH will not return it as an owner. But the CH could also return owners
that are no longer members of the cluster, so we have to check for
that before picking one owner to send the command to.

In Sanne's non-blocking state transfer proposal I think a new owner
may have to ask the old owner for the key value, so it would still
never return null. But it might be less expensive to ask the old owner
directly (assuming it's safe from a consistency POV).

>
> All good reasons for which imho it makes sense to send out "a couple"
>
> of requests in parallel, but I'd unlikely want to send more than 2,
>
> and I agree often 1 might be enough.
>
> Maybe it should even optimize for the most common case: send out just
>
> one, have a more aggressive timeout and in case of trouble ask for the
>
> next node.
>
> +1
>

-1 for aggressive timeouts... you're going to do the same work as you
do now, except you're going to wait a bit between sending requests. If
you're really unlucky the first target will return first but you'll
ignore its response because the timeout already expired.

>
> In addition, sending a single request might spare us some Future,
>
> await+notify messing in terms of CPU cost of sending the request.
>
> it's the remote OOB thread that's the most costly resource imo.
>

I don't think the OOB thread is that costly, it doesn't block on
anything (not even on state transfer!) so the most expensive part is
reading the key and writing the value. BTW Sanne, we may want to run
Transactional with a smaller payload size ;)

We could implement our own GroupRequest that sends the requests in
parallel instead implementing FutureCollator on top of UnicastRequest
and save some of that overhead on the caller.

I think we already have a JIRA to make PutKeyValueCommands return the
previous value, that would eliminate lots of GetKeyValueCommands and
it would actually improve the performance of puts - we should probably
make this a priority.

>
> I think I agree on all points, it makes more sense.
> Just that in a large cluster, let's say
> 1000 nodes, maybe I want 20 owners as a sweet spot for read/write
> performance tradeoff, and with such high numbers I guess doing 2-3
> gets in parallel might make sense as those "unlikely" events, suddenly
> are an almost certain.. especially the rehash in progress.
>
> So I'd propose a separate configuration option for # parallel get
> events, and one to define a "try next node" policy. Or this policy
> should be the whole strategy, and the #gets one of the options for the
> default implementation.
>
> Agreed that having a configurable remote get policy makes sense.
> We already have a JIRA for this[1], I'll start working on it as the
> performance results are hunting me.

I'd rather focus on implementing one remote get policy that works
instead of making it configurable - even if we make it configurable
we'll have to focus our optimizations on the default policy.

Keep in mind that we also want to introduce eventual consistency - I
think that's going to eliminate our optimization opportunity here
because we'll need to get the values from a majority of owners (if not
all the owners).

> I'd like to have Dan's input on this as well first, as he has worked with
> remote gets and I still don't know why null results are not considered valid
> :)

Pre-5.0 during state transfer an owner could return null to mean "I'm
not sure", so the caller would ignore it unless every target returned
null.
That's no longer necessary, but it wasn't broken so I didn't fix it...

Cheers
Dan

>
> [1] https://issues.jboss.org/browse/ISPN-825
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev