Investigation:
------------
When I looked at UNICAST3, I saw a lot of missing messages on the
receive side and unacked messages on the send side. This caused me to
look into the (mainly OOB) thread pools and - voila - maxed out !
I learned from Pedro that the Infinispan internal thread pool (with a
default of 32 threads) can be configured, so I increased it to 300 and
increased the OOB pools as well.
This mitigated the problem somewhat, but when I increased the
requester
threads to 100, I had the same problem again. Apparently, the
Infinispan
internal thread pool uses a rejection policy of "run" and thus
uses the
JGroups (OOB) thread when exhausted.
We can't use another rejection policy in the remote executor because
the message won't be re-delivered by JGroups, and we can't use a queue
either.
Can't we just send response "Node is busy" and cancel the operation? (at
least in cases where this is possible - we can't do that safely for
CommitCommand, but usually it could be doable, right?) And what's the
problem with queues, besides that these can grow out of memory?
Radim
--
Radim Vansa <rvansa(a)redhat.com>
JBoss DataGrid QA