[jboss-jira] [JBoss JIRA] (WFLY-9524) Messaging - default max client threads

[Jeff Mesnil (JIRA)]

    [ https://issues.jboss.org/browse/WFLY-9524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490452#comment-13490452 ] 

Jeff Mesnil commented on WFLY-9524:
-----------------------------------

Ok, I am convinced we need to support this use case.
Ideally, I would like to have the messaging-activemq subsystem adapts its configuration depending on the deployment use case:
* Cloud/VM deployment with small number of CPU cores
* Metal deployment with large number of CPU cores

Please note that I intend to modify the messaging-activemq subsystem to use thread pools from WildFly (as described in WFLY-5838). So it might make more sense to fix this issue by ensuring that the WildFly thread pools are correctly sized for this use case.

We need to have num-of-artemis-client-threads > num-of-ra-sessions * numbers-of-deployed-mdbs to avoid resource starvation.

* num-of-ra-sessions is controlled by Artemis maxSession activation config property (defaults to 15) and it is not configurable from WildFly management model.
* num-of-artemis-client-threads is managed by /subsystem=messaging-activemq's global-client-thread-pool-max-size attribute (default to 8 * CPU core).
* numbers-of-deployed-mdbs is the number of MDBs that are deployed multiplied by the pool size (managed by /subsystem=ejb3/strict-max-bean-instance-pool=mdb-strict-max-pool which defaults to 4 * CPU core)

This constraint depends on the number of deployed MDB and can not be enforced during the management model validation. We could add a warning whenever a MDB is deployed and we run the risk of resource starvation.

We could also change the default value of the num-of-artemis-client-threads so that it does not depend on the number of CPU cores. E.g. using the default value of 64.

With a default value of 64, we can deploy 1 MDB without resource starvation (that leaves 4 Artemis thread to do some stuff).
For deployment on metal hardware with more CPU cores, this default value is not great.

Maybe we should make the default value of the num-of-artemis-client-threads "smarter" and similar to the logic from the strict-max-bean-instance-pool and derives the actual value from CPU count.
E.g.

if ($cpu_core_count < 4) {
   value = 64
} else {
  value = 8 * $cpu-core-count
}

[~dmlloyd] Would it make sense to have the new WildFly thread pools use such derived size?





 

> Messaging - default max client threads
> --------------------------------------
>
>                 Key: WFLY-9524
>                 URL: https://issues.jboss.org/browse/WFLY-9524
>             Project: WildFly
>          Issue Type: Enhancement
>          Components: JMS
>    Affects Versions: 11.0.0.Final
>            Reporter: Martin Styk
>            Assignee: Jeff Mesnil
>
> By default, size of client thread pool is configured as 8 * number of CPU cores.
> On 1 CPU machine, with default configuration and MDB deployed on server, resource starvation is possible
> On one CPU there are by default
> * 8 ActiveMQ client threads (8 * CPU count)
> * 4 MDB instances (mdb-strict-max-pool = 4 * CPU count)
> * 15 JCA RA sessions consuming messages from queue (default value of maxSession)
> Consuming of messages by MDB gets stuck because all 8 client threads are awaiting large message completion and there is no other thread to handle other tasks.
> {code:title=Client thread waiting for large message completion}
> "Thread-7 (ActiveMQ-client-global-threads)" #475 daemon prio=5 os_prio=0
> tid=0x000000000590f000 nid=0x7c7a in Object.wait() [0x00007fda43413000]
>       java.lang.Thread.State: TIMED_WAITING (on object monitor)
>            at java.lang.Object.wait(Native Method)
>            at
>            org.apache.activemq.artemis.core.client.impl.LargeMessageControllerImpl.waitCompletion(LargeMessageControllerImpl.java:302)
>            - locked <0x00000000b0083e88> (a
>            org.apache.activemq.artemis.core.client.impl.LargeMessageControllerImpl)
>            at
>            org.apache.activemq.artemis.core.client.impl.LargeMessageControllerImpl.saveBuffer(LargeMessageControllerImpl.java:276)
>            - locked <0x00000000b0083e88> (a
>            org.apache.activemq.artemis.core.client.impl.LargeMessageControllerImpl)
>            at
>            org.apache.activemq.artemis.core.client.impl.ClientLargeMessageImpl.checkBuffer(ClientLargeMessageImpl.java:159)
>            at
>            org.apache.activemq.artemis.core.client.impl.ClientLargeMessageImpl.checkCompletion(ClientLargeMessageImpl.java:84)
>            at
>            org.apache.activemq.artemis.jms.client.ActiveMQMessage.doBeforeReceive(ActiveMQMessage.java:786)
>            at
>            org.apache.activemq.artemis.jms.client.ActiveMQTextMessage.doBeforeReceive(ActiveMQTextMessage.java:110)
>            at
>            org.apache.activemq.artemis.ra.inflow.ActiveMQMessageHandler.onMessage(ActiveMQMessageHandler.java:295)
>            at
>            org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1001)
>            at
>            org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:49)
>            at
>            org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1124)
>            at
>            org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:122)
>            at
>            java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>            at
>            java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>            at java.lang.Thread.run(Thread.java:748)
> {code}
> This can be fixed by adjusting default values for these parameters.
> In this case, we need more client threads than JCA RA sessions (maxSession). 
> To avoid resource starvation, number of client threads must be greater than sum of {{maxSession}} for each MDB deployed on server.
> We should check number of client threads required by deployments (MDBs), and at least print warning message that size of client thread pool may be insufficient.



--
This message was sent by Atlassian JIRA
(v7.5.0#75005)