[jboss-user] [JBoss Messaging] - Messages getting stuck in delivery on a clustered queue

[victorstar]

Hi guys,

We are doing a pilot project adding JBM to existing EJB application.
Configuration is as following:
JBoss 4.2.3, JBM 1.4.2.GA-SP1, JBossRemoting 2.5.1
EJB3 application is running on a cluster of 7 servers.
All servers have the same application farmed across them.
We have a clustered queue and using clustered connection factory.
Initially everything seemed to work fine
But what we noticed is that during peak server load times (nightly) some messages are getting "stuck" in the queues with MessageCount=DeliveringCount=non-zero. This numbers stays non-zero until the node is restarted. The symptoms are very similar to the open Jira issue https://jira.jboss.org/jira/browse/JBMESSAGING-1456
This issue links to the JBoss Remoting issue https://jira.jboss.org/jira/browse/JBREM-1112 which seems to be fixed in 2.5.1 release.
We've upgraded to this release but the issue seems to stay there.

While reading the Jira issues above and looking at the sympthoms it seems that the issue may be caused by network timeouts. So one thing we've done is we bumped the following values up in remoting-bisocket-service.xml file:

  | <attribute name="clientLeasePeriod" isParam="true">15000</attribute>
  | <attribute name="validatorPingPeriod" isParam="true">15000</attribute>
  | <attribute name="validatorPingTimeout" isParam="true">10000</attribute>
  | <attribute name="timeout" isParam="true">120000</attribute>
  | 

This (or maybe upgrade to the latest JBoss Remoting) seemed to help somewhat. If previous night saw us loosing all messages in the queue, last night had just 10% of them stuck. 
And actually it's a bit different tonight. We only see 4 messages stuck with MessageCount=DeliveringCount=non-zero but on one of the nodes we see a bunch of messages just sitting there with MessageCount=non-zero, DeliveringCount=0 and ConsumerCount=0. These messages are just piling up. While checking other nodes, four of them have same 0 ConsumerCount, two nodes have ConsumerCount=2 and one node has ConsumerCount=6 (the first one).

With all nodes having same code deployed with MDB listening on the queue why would the ConsumerCount be zero? And more importantly: why would the messages go to the node that doesn't have the consumers?
With this piece I think we might be just missing something in our configuration. I'll post our config below and hopefully you guys can point to what we're doing wrong. 

As for the messages "stuck" I'll try bumping timeouts even higher but this doesn't seem to be a proper solution. I would definitely appreciate a good advice here.

Our config:
Clustered queue defined in destinations-service.xml like this:

  |    <mbean code="org.jboss.jms.server.destination.QueueService"
  |       name="jboss.messaging.destination:service=Queue,name=/queue/recommendationSentQueue"
  |       xmbean-dd="xmdesc/Queue-xmbean.xml">
  |       <depends optional-attribute-name="ServerPeer">jboss.messaging:service=ServerPeer</depends>
  |       <depends>jboss.messaging:service=PostOffice</depends>
  |       <attribute name="Clustered">true</attribute>
  |       <attribute name="RedeliveryDelay">1000</attribute>
  |    </mbean>
  | 

MDB is annotated like this:

  | @MessageDriven(activationConfig =
  | {
  |       @ActivationConfigProperty(propertyName="destinationType", propertyValue="javax.jms.Queue"),
  |       @ActivationConfigProperty(propertyName="destination", propertyValue="/queue/recommendationSentQueue"),
  |       @ActivationConfigProperty(propertyName="DLQMaxResent", propertyValue="10")
  | })
  | 

JmsXA connection factory that we're using in the code is pointing to ClusteredXAConnectionFactory in hajndi-jms-ds.xml : 

  |   <mbean code="org.jboss.jms.jndi.JMSProviderLoader"
  |    name="jboss.messaging:service=JMSProviderLoader,name=HAJNDIJMSProvider">
  |     <attribute name="ProviderName">DefaultJMSProvider</attribute>
  |     <attribute name="ProviderAdapterClass">
  |       org.jboss.jms.jndi.JNDIProviderAdapter
  |     </attribute>
  |     <!-- The combined connection factory -->
  |     <attribute name="FactoryRef">ClusteredXAConnectionFactory</attribute>
  |     <!-- The queue connection factory -->
  |     <attribute name="QueueFactoryRef">ClusteredXAConnectionFactory</attribute>
  |     <!-- The topic factory -->
  |     <attribute name="TopicFactoryRef">ClusteredXAConnectionFactory</attribute>
  |     <!-- Access JMS via HAJNDI -->
  |     <attribute name="Properties">
  |        java.naming.factory.initial=org.jnp.interfaces.NamingContextFactory
  |        java.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces
  |        java.naming.provider.url=${jboss.bind.address:localhost}:1100
  |        jnp.disableDiscovery=false
  |        jnp.partitionName=${jboss.partition.name:DefaultPartition}
  |        jnp.discoveryGroup=${jboss.partition.udpGroup:230.0.0.4}
  |        jnp.discoveryPort=1102
  |        jnp.discoveryTTL=16
  |        jnp.discoveryTimeout=5000
  |        jnp.maxRetries=1
  |     </attribute>
  |   </mbean>
  | 
  |   <!-- The server session pool for Message Driven Beans -->
  |   <mbean code="org.jboss.jms.asf.ServerSessionPoolLoader"
  |    name="jboss.messaging:service=ServerSessionPoolMBean,name=StdJMSPool">
  |     <depends optional-attribute-name="XidFactory">jboss:service=XidFactory</depends>
  |     <attribute name="PoolName">StdJMSPool</attribute>
  |     <attribute name="PoolFactoryClass">
  |       org.jboss.jms.asf.StdServerSessionPoolFactory
  |     </attribute>
  |   </mbean>
  | 
  |   <!-- JMS XA Resource adapter, use this to get transacted JMS in beans -->
  |   <tx-connection-factory>
  |     <jndi-name>JmsXA</jndi-name>
  |     <xa-transaction/>
  |     <rar-name>jms-ra.rar</rar-name>
  |     <connection-definition>org.jboss.resource.adapter.jms.JmsConnectionFactory</connection-definition>
  |     <config-property name="SessionDefaultType" type="java.lang.String">javax.jms.Topic</config-property>
  |     <config-property name="JmsProviderAdapterJNDI" type="java.lang.String">java:/DefaultJMSProvider</config-property>
  |     <max-pool-size>20</max-pool-size>
  |     <security-domain-and-application>JmsXARealm</security-domain-and-application>
  |   </tx-connection-factory>
  | 

Thank you guys!
Looking forward to hear any ideas or advice.

Victor


View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4230528#4230528

Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4230528