[infinispan-issues] [JBoss JIRA] (ISPN-5246) Re-enable the queue for the JGroups internal thread pool in the default configuration

[Dan Berindei (JIRA)]

    [ https://issues.jboss.org/browse/ISPN-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045355#comment-13045355 ] 

Dan Berindei commented on ISPN-5246:
------------------------------------

[~rvansa] no need for a JGroups JIRA, because the default in JGroups was always to have a queue of 500 elements for the internal thread pool. Basically this is about changing the Infinispan embedded/server defaults to match the JGroups defaults, except with a higher number of active threads. 

The reason to make the change now is this error in a user's log:

{noformat}
2015-02-23 17:39:15.562 ERROR [org.jgroups.protocols.UDP] (unicast receiver,shared=udp) null: failed submitting DONT_BUNDLE message to thread pool: java.util.concurrent.RejectedExecutionException: Task org.jgroups.protocols.TP$SingleMessageHandler at 445f02d7 rejected from java.util.concurrent.ThreadPoolExecutor at ccce5c2[Running, pool size = 20, active threads = 19, queued tasks = 0, completed tasks = 442418]. Msg: MFC: REPLENISH, UNICAST3: DATA, seqno=11045, UDP: [cluster_name=clustered]
{noformat}

It shows the INTERNAL-flagged messages are processed very quickly, as the number of active threads is already 19 by the time the message is logged, but still it's possible for a MFC REPLENISH message to be dropped. And until the REPLENISH message is resent, after 500 milliseconds or more, all the threads trying to send broadcasts are blocked. 

Unfortunately, when a cluster with a lot of defined caches starts up, the coordinator needs to send a lot of broadcast messages in response to unicast messages from other nodes (e.g. a CH_UPDATE message in response to a REBALANCE_CONFIRM message), and blocking all of them quickly fills up the remote executor and then the OOB thread pools.


> Re-enable the queue for the JGroups internal thread pool in the default configuration
> -------------------------------------------------------------------------------------
>
>                 Key: ISPN-5246
>                 URL: https://issues.jboss.org/browse/ISPN-5246
>             Project: Infinispan
>          Issue Type: Task
>          Components: Configuration
>    Affects Versions: 7.2.0.Alpha1
>            Reporter: Dan Berindei
>            Assignee: Dan Berindei
>            Priority: Critical
>             Fix For: 7.2.0.Beta1, 7.2.0.Final
>
>
> Need to change {{default-jgroups-udp/tcp.xml}} for embedded and {{jgroups-defaults.xml}} for server.



--
This message was sent by Atlassian JIRA
(v6.3.11#6341)