[JBoss JIRA] Created: (JBMESSAGING-1456) Messages stuck in being-delivered state in cluster
by Justin Bertram (JIRA)
Messages stuck in being-delivered state in cluster
--------------------------------------------------
Key: JBMESSAGING-1456
URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1456
Project: JBoss Messaging
Issue Type: Bug
Reporter: Justin Bertram
Assignee: Tim Fox
Priority: Critical
Messages become "stuck" in being-delivered state when clients use a clustered XA connection factory in a cluster of at least 2 nodes.
JBoss setup:
-2 nodes of JBoss EAP 4.3 CP02
-commented out "ClusterPullConnectionFactory" in messaging-service.xml to prevent message redistribution and eliminate the "message suckers" as the potential culprit
-MySQL backend using the default mysql-persistence-service.xml (from <JBOSS_HOME>/docs/examples/jms)
Client setup:
-both nodes have a client which is a separate process (i.e. not inside JBoss)
-clients are Spring based
-one client produces and consumes, the other client just consumes
-both clients use the ClusteredXAConnectionFactory from the default connection-factories-service.xml
-both clients publish to and consume from "queue/testDistributedQueue"
-clients are configured to send persistent messages, use AUTO_ACKNOWLEDGE, and transacted sessions
Symptoms of the issue:
-when running the clients I watch the JMX-Console for the "queue/testDistributedQueue"
-as the consumers pull messages off the queue I can see the MessageCount and DeliveringCount go to 0 every so often
-after a period of time (usually a few hours) the MessageCount and DeliveringCount never go back to 0
-I "kill" the clients and wait for the DeliveringCount to go to 0, but it never does
-after the clients are killed the ConsumerCount for the queue will drop, but never to 0 when messages are "stuck"
-a thread dump reveals at least one JBM server session that is apparently stuck (it never goes away) - ostensibly this is the consumer that is showing in the JMX-Console for "queue/testDistributedQueue"
-a "killall -3 java" doesn't produce anything from the clients so I know their dead
-nothing is in any DLQ or expiry queue
-the database contains as many rows in the JBM_MSG and JBM_MSG_REF tables as the DeliveringCount in the JMX-Console
-rebooting the node with the stuck messages frees the messages to be consumed (i.e. un-sticks them)
Other notes:
-nothing else is happening on either node but running the client and running JBoss
-this only appears to happen when a clustered connection factory is used. I tested using a normal connection factory and after 24 hours couldn't reproduce a stuck message.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 3 months
[JBoss JIRA] Created: (EJBTHREE-1330) EJB timer service should use a thread pool to avoid OOM
by Galder Zamarreno (JIRA)
EJB timer service should use a thread pool to avoid OOM
-------------------------------------------------------
Key: EJBTHREE-1330
URL: http://jira.jboss.com/jira/browse/EJBTHREE-1330
Project: EJB 3.0
Issue Type: Bug
Components: pool
Affects Versions: AS 4.2.2.GA
Reporter: Galder Zamarreno
Assigned To: Galder Zamarreno
Priority: Minor
The default EJB timer service used by the EJB3 layer is based on
org.jboss.ejb3.timerservice.jboss.JBossTimerServiceFactory which delegates
to the standard org.jboss.ejb.txtimer.EJBTimerService.
For EJB3 beans using the EJB timer service the thread local pool should not be used.
Since the current EJB timer service creates a new thread for each timer being created, the
thread local pool will create a matching instance of the bean for that thread. Thus the number
of active instances in total can effectively grow unchecked and thus an OOM will occur.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 3 months
[JBoss JIRA] Created: (JGRP-1002) Merge: make the coordinator remain the same if possible after a merge
by Bela Ban (JIRA)
Merge: make the coordinator remain the same if possible after a merge
---------------------------------------------------------------------
Key: JGRP-1002
URL: https://jira.jboss.org/jira/browse/JGRP-1002
Project: JGroups
Issue Type: Feature Request
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 3.0
When we have {C, A, B} and split into {C,A} and {B}, then a merge will produce {A,B,C}. Therefore the coord is moved from C to A. For certain apps, e.g. JBoss singletons, this is not good, as we're moving the singleton services from a perfectly healthy node to a different node.
GOAL: come up with a deterministic policy of picking a coord such that we minimize coord switching if the existing coord is still member of the cluster.
This depends on or is related to the flexible coord pick policy JIRA issue
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 5 months