Hi, I've just wrote on the wiki a new algorithm to better handle the remote commands. You can find it in [1]. If you have questions, suggestion or just want to discuss some aspect, please do in thread. I'll update the wiki page based on this discussion Thanks. Cheers, Pedro [1]https://github.com/infinispan/infinispan/wiki/Remote-Command-Handler-(W....)

Thanks Pedro, this looks great. However, I don't think it's ok to treat CommitCommands/Pessimistic PrepareCommands as RemoteLockCommands just because they may send L1 invalidation commands. It's true that those commands will block, but there's no need to wait for any other command before doing the L1 invalidation. In fact, the non-tx writes on backup owners, which you consider to be non-blocking, can also send L1 invalidation commands (see L1NonTxInterceptor.invalidateL1). On the other hand, one of the good things that the remote executor did was to allow queueing lots of commands with a higher topology id, when one of the nodes receives the new topology much later than the others. We still have to consider each TopologyAffectedCommand as potentially blocking and put it through the remote executor. And InvalidateL1Commands are also TopologyAffectedCommands, so there's still a potential for deadlock when L1 is enabled and we have maxThreads write commands blocked sending L1 invalidations and those L1 invalidation commands are stuck in the remote executor's queue on another node. And with (very) unlucky timing the remote executor might not even get to create maxThreads threads before the deadlock appears. I wonder if we could write a custom executor that checks what the first task in the queue is every second or so, and creates a bunch of new threads if the first task in the queue hasn't changed. You're right about the remote executor getting full as well, we're lacking any feedback mechanism to tell the sender to slow down, except for blocking the OOB thread.

I wonder if we could tell JGroups somehow to discard the message from inside MessageDispatcher.handle (e.g. throw a DiscardMessageException), so the sender has to retransmit it

and we don't block the OOB thread. That should allow us to set a size limit on the BlockingTaskAwareExecutor's blockedTasks collection as well. Bela, WDYT? Cheers Dan On Wed, Sep 17, 2014 at 7:17 PM, Pedro Ruivo <pedro(a)infinispan.org <mailto:pedro@infinispan.org>> wrote: new link: https://github.com/infinispan/infinispan/wiki/Remote-Command-Handler On 09/17/2014 05:08 PM, Pedro Ruivo wrote: > Hi, > > I've just wrote on the wiki a new algorithm to better handle the remote > commands. You can find it in [1]. > > If you have questions, suggestion or just want to discuss some aspect, > please do in thread. I'll update the wiki page based on this discussion > > Thanks. > > Cheers, > Pedro > > [1]https://github.com/infinispan/infinispan/wiki/Remote-Command-Handler-(W....) > _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org <mailto:infinispan-dev@lists.jboss.org> https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On Thu, Sep 18, 2014 at 3:09 PM, Bela Ban <bban(a)redhat.com <mailto:bban@redhat.com>> wrote: On 18/09/14 13:03, Dan Berindei wrote: > Thanks Pedro, this looks great. > > However, I don't think it's ok to treat CommitCommands/Pessimistic > PrepareCommands as RemoteLockCommands just because they may send L1 > invalidation commands. It's true that those commands will block, but > there's no need to wait for any other command before doing the L1 > invalidation. In fact, the non-tx writes on backup owners, which you > consider to be non-blocking, can also send L1 invalidation commands (see > L1NonTxInterceptor.invalidateL1). > > On the other hand, one of the good things that the remote executor did > was to allow queueing lots of commands with a higher topology id, when > one of the nodes receives the new topology much later than the others. > We still have to consider each TopologyAffectedCommand as potentially > blocking and put it through the remote executor. > > And InvalidateL1Commands are also TopologyAffectedCommands, so there's > still a potential for deadlock when L1 is enabled and we have maxThreads > write commands blocked sending L1 invalidations and those L1 > invalidation commands are stuck in the remote executor's queue on > another node. And with (very) unlucky timing the remote executor might > not even get to create maxThreads threads before the deadlock appears. I > wonder if we could write a custom executor that checks what the first > task in the queue is every second or so, and creates a bunch of new > threads if the first task in the queue hasn't changed. > > You're right about the remote executor getting full as well, we're > lacking any feedback mechanism to tell the sender to slow down, except > for blocking the OOB thread. JGroups sends credits back to the sender *after* the message has been delivered into the application. If the application is slow in processing the messages, or blocks for some time, then the sender will not receive enough credits and thus also slow down, or even block. > I wonder if we could tell JGroups somehow > to discard the message from inside MessageDispatcher.handle (e.g. throw > a DiscardMessageException), so the sender has to retransmit it At this point, JGroups considers the message *delivered* (as it has passed the UNICAST or NAKACK protocols), and it won't get resent. You cannot discard it either, as this will be a message loss. However, if you can tolerate loss, all is fine. E.g. if you discard a topo message with a lower ID, I don't think any harm is done in Infinispan. (?). To discard or not, is Infinispan's decision. The other thing is to block, but this can have an impact back into the JGroups thread pools. Right, I was hoping the message would be marked as delivered only after Infinispan finished processing the message (i.e. when up() returns in UNICAST/NAKACK).

Perhaps we could delay sending the credits instead? When process a message on our internal thread pool, it would be nice if we could tell JGroups to send credits back only when we really finished processing the message.

On 09/18/2014 12:03 PM, Dan Berindei wrote: > Thanks Pedro, this looks great. > > However, I don't think it's ok to treat CommitCommands/Pessimistic > PrepareCommands as RemoteLockCommands just because they may send L1 > invalidation commands. It's true that those commands will block, but > there's no need to wait for any other command before doing the L1 > invalidation. In fact, the non-tx writes on backup owners, which you > consider to be non-blocking, can also send L1 invalidation commands (see > L1NonTxInterceptor.invalidateL1). They are not treated as RemoteLockCommands. I just said that they are processed in the remote executor service (need to double check what I wrote in the wiki). Unfortunately, I haven't think about the L1 in that scenario... :(

> > On the other hand, one of the good things that the remote executor did > was to allow queueing lots of commands with a higher topology id, when > one of the nodes receives the new topology much later than the others. > We still have to consider each TopologyAffectedCommand as potentially > blocking and put it through the remote executor. > > And InvalidateL1Commands are also TopologyAffectedCommands, so there's > still a potential for deadlock when L1 is enabled and we have maxThreads > write commands blocked sending L1 invalidations and those L1 > invalidation commands are stuck in the remote executor's queue on > another node. And with (very) unlucky timing the remote executor might > not even get to create maxThreads threads before the deadlock appears. I > wonder if we could write a custom executor that checks what the first > task in the queue is every second or so, and creates a bunch of new > threads if the first task in the queue hasn't changed. I need to think a little more about it. So, a single put can originate: 1 RPC to the primary owner (to lock) X RPC to invalidate L1 from the primary owner R RPC for the primary owner to the backups owner Y RPC to invalidate L1 from the backup owner is this correct?

any suggestions are welcome. > > You're right about the remote executor getting full as well, we're > lacking any feedback mechanism to tell the sender to slow down, except > for blocking the OOB thread. I wonder if we could tell JGroups somehow > to discard the message from inside MessageDispatcher.handle (e.g. throw > a DiscardMessageException), so the sender has to retransmit it and we > don't block the OOB thread. That should allow us to set a size limit on > the BlockingTaskAwareExecutor's blockedTasks collection as well. Bela, WDYT? Even if we have a way to tell the JGroups to resend the message, we have no idea if the executor service is full or not. We allow a user to inject their own implementation of it.

> > Cheers > Dan > > > On Wed, Sep 17, 2014 at 7:17 PM, Pedro Ruivo <pedro(a)infinispan.org > <mailto:pedro@infinispan.org>> wrote: > > new link: > https://github.com/infinispan/infinispan/wiki/Remote-Command-Handler > > On 09/17/2014 05:08 PM, Pedro Ruivo wrote: > > Hi, > > > > I've just wrote on the wiki a new algorithm to better handle the > remote > > commands. You can find it in [1]. > > > > If you have questions, suggestion or just want to discuss some > aspect, > > please do in thread. I'll update the wiki page based on this > discussion > > > > Thanks. > > > > Cheers, > > Pedro > > > > > [1] https://github.com/infinispan/infinispan/wiki/Remote-Command-Handler-(Wor.... .) > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org <mailto: infinispan-dev(a)lists.jboss.org&gt; > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

> > And InvalidateL1Commands are also TopologyAffectedCommands, so there's > still a potential for deadlock when L1 is enabled and we have maxThreads > write commands blocked sending L1 invalidations and those L1 > invalidation commands are stuck in the remote executor's queue on > another node. And with (very) unlucky timing the remote executor might > not even get to create maxThreads threads before the deadlock appears. I > wonder if we could write a custom executor that checks what the first > task in the queue is every second or so, and creates a bunch of new > threads if the first task in the queue hasn't changed.

"Read Command : No changes are made. When a read command is delivered, it is processed directly in the JGroups executor service." - you might want to check if here's a force write lock flag present as well