8:38 a.m.
[
https://issues.jboss.org/browse/WFLY-9524?page=com.atlassian.jira.plugin....
]
Jeff Mesnil commented on WFLY-9524:
-----------------------------------
Ok, I am convinced we need to support this use case.
Ideally, I would like to have the messaging-activemq subsystem adapts its configuration
depending on the deployment use case:
* Cloud/VM deployment with small number of CPU cores
* Metal deployment with large number of CPU cores
Please note that I intend to modify the messaging-activemq subsystem to use thread pools
from WildFly (as described in WFLY-5838). So it might make more sense to fix this issue by
ensuring that the WildFly thread pools are correctly sized for this use case.
We need to have num-of-artemis-client-threads > num-of-ra-sessions *
numbers-of-deployed-mdbs to avoid resource starvation.
* num-of-ra-sessions is controlled by Artemis maxSession activation config property
(defaults to 15) and it is not configurable from WildFly management model.
* num-of-artemis-client-threads is managed by /subsystem=messaging-activemq's
global-client-thread-pool-max-size attribute (default to 8 * CPU core).
* numbers-of-deployed-mdbs is the number of MDBs that are deployed multiplied by the pool
size (managed by /subsystem=ejb3/strict-max-bean-instance-pool=mdb-strict-max-pool which
defaults to 4 * CPU core)
This constraint depends on the number of deployed MDB and can not be enforced during the
management model validation. We could add a warning whenever a MDB is deployed and we run
the risk of resource starvation.
We could also change the default value of the num-of-artemis-client-threads so that it
does not depend on the number of CPU cores. E.g. using the default value of 64.
With a default value of 64, we can deploy 1 MDB without resource starvation (that leaves 4
Artemis thread to do some stuff).
For deployment on metal hardware with more CPU cores, this default value is not great.
Maybe we should make the default value of the num-of-artemis-client-threads
"smarter" and similar to the logic from the strict-max-bean-instance-pool and
derives the actual value from CPU count.
E.g.
if ($cpu_core_count < 4) {
value = 64
} else {
value = 8 * $cpu-core-count
}
[~dmlloyd] Would it make sense to have the new WildFly thread pools use such derived
size?
Messaging - default max client threads
--------------------------------------
Key: WFLY-9524
URL: https://issues.jboss.org/browse/WFLY-9524
Project: WildFly
Issue Type: Enhancement
Components: JMS
Affects Versions: 11.0.0.Final
Reporter: Martin Styk
Assignee: Jeff Mesnil
By default, size of client thread pool is configured as 8 * number of CPU cores.
On 1 CPU machine, with default configuration and MDB deployed on server, resource
starvation is possible
On one CPU there are by default
* 8 ActiveMQ client threads (8 * CPU count)
* 4 MDB instances (mdb-strict-max-pool = 4 * CPU count)
* 15 JCA RA sessions consuming messages from queue (default value of maxSession)
Consuming of messages by MDB gets stuck because all 8 client threads are awaiting large
message completion and there is no other thread to handle other tasks.
{code:title=Client thread waiting for large message completion}
"Thread-7 (ActiveMQ-client-global-threads)" #475 daemon prio=5 os_prio=0
tid=0x000000000590f000 nid=0x7c7a in Object.wait() [0x00007fda43413000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.apache.activemq.artemis.core.client.impl.LargeMessageControllerImpl.waitCompletion(LargeMessageControllerImpl.java:302)
- locked <0x00000000b0083e88> (a
org.apache.activemq.artemis.core.client.impl.LargeMessageControllerImpl)
at
org.apache.activemq.artemis.core.client.impl.LargeMessageControllerImpl.saveBuffer(LargeMessageControllerImpl.java:276)
- locked <0x00000000b0083e88> (a
org.apache.activemq.artemis.core.client.impl.LargeMessageControllerImpl)
at
org.apache.activemq.artemis.core.client.impl.ClientLargeMessageImpl.checkBuffer(ClientLargeMessageImpl.java:159)
at
org.apache.activemq.artemis.core.client.impl.ClientLargeMessageImpl.checkCompletion(ClientLargeMessageImpl.java:84)
at
org.apache.activemq.artemis.jms.client.ActiveMQMessage.doBeforeReceive(ActiveMQMessage.java:786)
at
org.apache.activemq.artemis.jms.client.ActiveMQTextMessage.doBeforeReceive(ActiveMQTextMessage.java:110)
at
org.apache.activemq.artemis.ra.inflow.ActiveMQMessageHandler.onMessage(ActiveMQMessageHandler.java:295)
at
org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1001)
at
org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:49)
at
org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1124)
at
org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:122)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
{code}
This can be fixed by adjusting default values for these parameters.
In this case, we need more client threads than JCA RA sessions (maxSession).
To avoid resource starvation, number of client threads must be greater than sum of
{{maxSession}} for each MDB deployed on server.
We should check number of client threads required by deployments (MDBs), and at least
print warning message that size of client thread pool may be insufficient.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
3164
days inactive
3164
days old
0 comments
1 participants
participants (1)
-
Jeff Mesnil (JIRA)