[infinispan-dev] New algorithm to handle remote commands

[Dan Berindei]

On Thu, Sep 18, 2014 at 3:09 PM, Bela Ban <bban at redhat.com> wrote:

>
>
> On 18/09/14 13:03, Dan Berindei wrote:
> > Thanks Pedro, this looks great.
> >
> > However, I don't think it's ok to treat CommitCommands/Pessimistic
> > PrepareCommands as RemoteLockCommands just because they may send L1
> > invalidation commands. It's true that those commands will block, but
> > there's no need to wait for any other command before doing the L1
> > invalidation. In fact, the non-tx writes on backup owners, which you
> > consider to be non-blocking, can also send L1 invalidation commands (see
> > L1NonTxInterceptor.invalidateL1).
> >
> > On the other hand, one of the good things that the remote executor did
> > was to allow queueing lots of commands with a higher topology id, when
> > one of the nodes receives the new topology much later than the others.
> > We still have to consider each TopologyAffectedCommand as potentially
> > blocking and put it through the remote executor.
> >
> > And InvalidateL1Commands are also TopologyAffectedCommands, so there's
> > still a potential for deadlock when L1 is enabled and we have maxThreads
> > write commands blocked sending L1 invalidations and those L1
> > invalidation commands are stuck in the remote executor's queue on
> > another node. And with (very) unlucky timing the remote executor might
> > not even get to create maxThreads threads before the deadlock appears. I
> > wonder if we could write a custom executor that checks what the first
> > task in the queue is every second or so, and creates a bunch of new
> > threads if the first task in the queue hasn't changed.
> >
> > You're right about the remote executor getting full as well, we're
> > lacking any feedback mechanism to tell the sender to slow down, except
> > for blocking the OOB thread.
>
> JGroups sends credits back to the sender *after* the message has been
> delivered into the application. If the application is slow in processing
> the messages, or blocks for some time, then the sender will not receive
> enough credits and thus also slow down, or even block.
>

> > I wonder if we could tell JGroups somehow
> > to discard the message from inside MessageDispatcher.handle (e.g. throw
> > a DiscardMessageException), so the sender has to retransmit it
>
> At this point, JGroups considers the message *delivered* (as it has
> passed the UNICAST or NAKACK protocols), and it won't get resent. You
> cannot discard it either, as this will be a message loss. However, if
> you can tolerate loss, all is fine. E.g. if you discard a topo message
> with a lower ID, I don't think any harm is done in Infinispan. (?). To
> discard or not, is Infinispan's decision.
> The other thing is to block, but this can have an impact back into the
> JGroups thread pools.
>

Right, I was hoping the message would be marked as delivered only after
Infinispan finished processing the message (i.e. when up() returns in
UNICAST/NAKACK).

Perhaps we could delay sending the credits instead? When process a message
on our internal thread pool, it would be nice if we could tell JGroups to
send credits back only when we really finished processing the message.


> > and we
> > don't block the OOB thread. That should allow us to set a size limit on
> > the BlockingTaskAwareExecutor's blockedTasks collection as well. Bela,
> WDYT?
> >
> > Cheers
> > Dan
> >
> >
> > On Wed, Sep 17, 2014 at 7:17 PM, Pedro Ruivo <pedro at infinispan.org
> > <mailto:pedro at infinispan.org>> wrote:
> >
> >     new link:
> >     https://github.com/infinispan/infinispan/wiki/Remote-Command-Handler
> >
> >     On 09/17/2014 05:08 PM, Pedro Ruivo wrote:
> >      > Hi,
> >      >
> >      > I've just wrote on the wiki a new algorithm to better handle the
> >     remote
> >      > commands. You can find it in [1].
> >      >
> >      > If you have questions, suggestion or just want to discuss some
> >     aspect,
> >      > please do in thread. I'll update the wiki page based on this
> >     discussion
> >      >
> >      > Thanks.
> >      >
> >      > Cheers,
> >      > Pedro
> >      >
> >      >
> >     [1]
> https://github.com/infinispan/infinispan/wiki/Remote-Command-Handler-(Work-In-Progress..
> .)
> >      >
> >     _______________________________________________
> >     infinispan-dev mailing list
> >     infinispan-dev at lists.jboss.org <mailto:
> infinispan-dev at lists.jboss.org>
> >     https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
> >
> >
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
>
> --
> Bela Ban, JGroups lead (http://www.jgroups.org)
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20140918/87765182/attachment-0001.html