Investigation:
------------
When I looked at UNICAST3, I saw a lot of missing messages on the
receive side and unacked messages on the send side. This caused me to
look into the (mainly OOB) thread pools and - voila - maxed out !

I learned from Pedro that the Infinispan internal thread pool (with a
default of 32 threads) can be configured, so I increased it to 300 and
increased the OOB pools as well.

This mitigated the problem somewhat, but when I increased the requester
threads to 100, I had the same problem again. Apparently, the Infinispan
internal thread pool uses a rejection policy of "run" and thus uses the
JGroups (OOB) thread when exhausted.

We can't use another rejection policy in the remote executor because the message won't be re-delivered by JGroups, and we can't use a queue either.

Can't we just send response "Node is busy" and cancel the operation? (at least in cases where this is possible - we can't do that safely for CommitCommand, but usually it could be doable, right?) And what's the problem with queues, besides that these can grow out of memory?

Radim

-- 
Radim Vansa <rvansa@redhat.com>
JBoss DataGrid QA