[infinispan-dev] DefaultExecutorFactory and rejection policy

[Dan Berindei]

On Thu, Mar 14, 2013 at 2:31 PM, Radim Vansa <rvansa at redhat.com> wrote:

>
>
> ----- Original Message -----
> | From: "Dan Berindei" <dan.berindei at gmail.com>
> | To: "infinispan -Dev List" <infinispan-dev at lists.jboss.org>
> | Sent: Thursday, March 14, 2013 10:03:10 AM
> | Subject: Re: [infinispan-dev] DefaultExecutorFactory and rejection policy
> |
> | On Thu, Mar 14, 2013 at 9:32 AM, Radim Vansa < rvansa at redhat.com >
> | wrote:
> |
> |
> | | Blocking OOB threads is the thing we want to avoid, remember?
> |
> | Well, you have to block somewhere...
> |
> | I like Adrian's solution, because it's a lot better than
> | CallerRunsPolicy: it's blocking the OOB thread until any other
> | command finishes executing, not until one particular command
> | finishes executing.
>
> I don't like caller-runs policy either. OOB threads shouldn't be waiting
> for anything and executing a command within OOB thread could cause that. In
> our problem, it would only increase the OOB threadpool size by the ispn
> thread pool size and cause some overhead to it. We should always have some
> OOB threads able to process the responses.
>
>
I agree that the OOB threads should not block, as much as possible. The
problem is that the internal thread pool needs to have some limits, too,
and when we get to those limits we'll have to block. We can't discard the
message and wait for the sender to retransmit it, like JGroups does.

We can increase those limits a lot by using a queue instead of extra
threads, but queueing needs to be smart (e.g. can't queue a CommitCommand
when there is a PrepareCommand waiting for a lock).


 |
> | | ...
> |
> | I don't think you can throttle on the sender, because you don't know
> | how many threads the recipient should allocate per sender in the
> | Infinispan thread pool.
>
> You don't need to know how many threads exactly are executing on the
> receiver, because you would have an unbounded queue there which will,
> sooner or later, process the messages. The semaphore is there because of
> throttling so that you never overwhelm the recipient with requests.
>
>
You still need a number to initialize the semaphore...



> I don't insist on semaphore-like synchronization, but the receiver should
> provide some feedback that it's not able to process more messages and the
> sender should be responsible for abiding it. AND, the feedback should be
> provided in a way that the node is still able to process other messages.


Well, sticking all the commands in the ISPN thread pool queue and delaying
their processing would be giving feedback to the sender: while the command
is waiting in the queue, the thread that sent it can't send another command
:)



> If it is a jammed signal message broadcast after the queue length grows
> beyond some limit, and then all messages on senders should be postponed, or
> divine intervention, it's just a performance issue. But if there a
> situation where the node is not able to process any more messages (on
> JGroups level), the OOB issue won't be solved, because there may be a reply
> other message is waiting for that is never processed.
>
>
Even if we do implement a jammed signal, the other nodes will not respond
to the signal instantaneously. So we still have the possibility of not
having enough threads in the ISPN thread pool to process all the incoming
messages, or even enough room in the queue.


>
> |
> | E.g. in non-tx concurrent mode, for each user-initiated request, the
> | primary owner sends out numOwners requests to the other nodes. If
> | the cache is also replicated, you have the coordinator sending
> | clusterSize requests to the other nodes for each user-initiated
> | request. (So for each request that a node receives from the
> | coordinator, it will receive exactly 0 requests from the other
> | nodes.)
> |
> |
> | If you start considering that each cache should have its own
> | semaphore, because they need different numbers of threads, then
> | finding an appropriate initial value is pretty much impossible.
> |
>
> Sorry, I miss the point here. It's not necessary to be exactly able to
> tell how many messages may go from one node to another, you can just use a
> common sense to limit the maximum amount of request that should be
> processed concurrently between the two nodes until we say "Hey, give the
> poor node a break, it's not responding because it's busy and another
> message would not help right now".
>
>
My point is that there is no "common sense" limit for the maximum amount of
requests that should be processed concurrently between two nodes, because
some nodes send a lot more requests than others, and limiting them
artificially would just slow things down.

We already have a throttling mechanism in the HotRod server: the number of
HotRod worker threads. If throttling without a feedback from the recipient
was a good idea, you could just lower the number of HotRod threads and
solve the problem :)

Cheers
Dan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20130320/939d56c0/attachment-0001.html