[ https://issues.jboss.org/browse/ISPN-5246?page=com.atlassian.jira.plugin.... ] Dan Berindei commented on ISPN-5246: ------------------------------------ [~rvansa] no need for a JGroups JIRA, because the default in JGroups was always to have a queue of 500 elements for the internal thread pool. Basically this is about changing the Infinispan embedded/server defaults to match the JGroups defaults, except with a higher number of active threads. The reason to make the change now is this error in a user's log: {noformat} 2015-02-23 17:39:15.562 ERROR [org.jgroups.protocols.UDP] (unicast receiver,shared=udp) null: failed submitting DONT_BUNDLE message to thread pool: java.util.concurrent.RejectedExecutionException: Task org.jgroups.protocols.TP$SingleMessageHandler@445f02d7 rejected from java.util.concurrent.ThreadPoolExecutor@ccce5c2[Running, pool size = 20, active threads = 19, queued tasks = 0, completed tasks = 442418]. Msg: MFC: REPLENISH, UNICAST3: DATA, seqno=11045, UDP: [cluster_name=clustered] {noformat} It shows the INTERNAL-flagged messages are processed very quickly, as the number of active threads is already 19 by the time the message is logged, but still it's possible for a MFC REPLENISH message to be dropped. And until the REPLENISH message is resent, after 500 milliseconds or more, all the threads trying to send broadcasts are blocked. Unfortunately, when a cluster with a lot of defined caches starts up, the coordinator needs to send a lot of broadcast messages in response to unicast messages from other nodes (e.g. a CH_UPDATE message in response to a REBALANCE_CONFIRM message), and blocking all of them quickly fills up the remote executor and then the OOB thread pools. -- This message was sent by Atlassian JIRA (v6.3.11#6341)