[infinispan-dev] New algorithm to handle remote commands

[Dan Berindei]

Thanks Pedro, this looks great.

However, I don't think it's ok to treat CommitCommands/Pessimistic
PrepareCommands as RemoteLockCommands just because they may send L1
invalidation commands. It's true that those commands will block, but
there's no need to wait for any other command before doing the L1
invalidation. In fact, the non-tx writes on backup owners, which you
consider to be non-blocking, can also send L1 invalidation commands (see
L1NonTxInterceptor.invalidateL1).

On the other hand, one of the good things that the remote executor did was
to allow queueing lots of commands with a higher topology id, when one of
the nodes receives the new topology much later than the others. We still
have to consider each TopologyAffectedCommand as potentially blocking and
put it through the remote executor.

And InvalidateL1Commands are also TopologyAffectedCommands, so there's
still a potential for deadlock when L1 is enabled and we have maxThreads
write commands blocked sending L1 invalidations and those L1 invalidation
commands are stuck in the remote executor's queue on another node. And with
(very) unlucky timing the remote executor might not even get to create
maxThreads threads before the deadlock appears. I wonder if we could write
a custom executor that checks what the first task in the queue is every
second or so, and creates a bunch of new threads if the first task in the
queue hasn't changed.

You're right about the remote executor getting full as well, we're lacking
any feedback mechanism to tell the sender to slow down, except for blocking
the OOB thread. I wonder if we could tell JGroups somehow to discard the
message from inside MessageDispatcher.handle (e.g. throw a
DiscardMessageException), so the sender has to retransmit it and we don't
block the OOB thread. That should allow us to set a size limit on the
BlockingTaskAwareExecutor's blockedTasks collection as well. Bela, WDYT?

Cheers
Dan


On Wed, Sep 17, 2014 at 7:17 PM, Pedro Ruivo <pedro at infinispan.org> wrote:

> new link:
> https://github.com/infinispan/infinispan/wiki/Remote-Command-Handler
>
> On 09/17/2014 05:08 PM, Pedro Ruivo wrote:
> > Hi,
> >
> > I've just wrote on the wiki a new algorithm to better handle the remote
> > commands. You can find it in [1].
> >
> > If you have questions, suggestion or just want to discuss some aspect,
> > please do in thread. I'll update the wiki page based on this discussion
> >
> > Thanks.
> >
> > Cheers,
> > Pedro
> >
> > [1]
> https://github.com/infinispan/infinispan/wiki/Remote-Command-Handler-(Work-In-Progress..
> .)
> >
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20140918/f7fcfe79/attachment-0001.html