12:47 p.m.
Gurvinderpal Narula [http://community.jboss.org/people/groovenarula] created the
discussion
"Messages Stuck in 'Delivering State'."
To view the discussion, visit: http://community.jboss.org/message/588306#588306
--------------------------------------------------------------
Our system consists of one virtualized VMWare guest server running Red Hat Enterprise
Server 64 bit, Java HotSpot(TM) 64-Bit Server VM
and JBoss 5.1.0.GA. The machine runs a single instance of JBoss with Postgresql (running
on a different server).
The JBOSS instance is dedicated to processing JMS Messages for 3 tomcat based web
application clients again running on different servers.
We have made no other changes to JBOSS Messaging Postgresql Presistence manager's
configuration other then disabling 'ChannelFactoryName'
<!-- <depends
optional-attribute-name="ChannelFactoryName">jboss.jgroups:service=ChannelFactory</depends>
-->
(referred the following : http://community.jboss.org/message/204473#204473
http://community.jboss.org/message/204473#204473 )
The JMS clients are all Spring based web applications that consumer messages from one of
the queues on the jboss server.
We deployed this configuration to production this Monday. So far the server has been very
stable. The only issue that we've run into is that there are (now) about 7 messages
that are stuck in the 'Delivering' state since the last 3 days. So basically
everytime we look at the queue's status using the jmx-console, it shows MessageCount
is stuck at 7 and DeliveringCount is also stuck at 7. The tomcat servers have recevied
several other messages since we've noticed these stuck messages. We tried to remove
these messages using the 'removeAllMessages' operation, but we then saw an error
in the jboss logs basically stating that messages that are in a DeliveringState cannot be
removed.
I would like to know if someone can please help me find a way to delete these messages.
Also we have not been able to reproduce this behavior in our development / test
enviroment. So we're wondering if this could have been caused by some intermittent
communication failure (network not available etc) between the Jboss Servers and Tomcat
Servers. Does anyone know of any such issues with the JBM version (I believe it's
1.4.3 GA) bundled with JBOSS AS 5.1.0.
We do plan to upgrade to HornetQ in the near future (there is a critical bug in the
current Final version of HornetQ that we're wait to be resolved in the next version
before we migrate over the HornetQ).
So if anyboby can shed some light on what could cause messages to end up in a
DeliveryState and what steps I can take to remove these messages from the queue, it will
be much appreciated.
Thanks in advance.
Gurvinder
--------------------------------------------------------------
Reply to this message by going to Community
[http://community.jboss.org/message/588306#588306]
Start a new discussion in JBoss Messaging at Community
[http://community.jboss.org/choose-container!input.jspa?contentType=1&...]
8:16 p.m.
New subject: [JBoss Messaging] - Re: Messages Stuck in 'Delivering State'.
Gurvinderpal Narula [http://community.jboss.org/people/groovenarula] created the
discussion
"Re: Messages Stuck in 'Delivering State'."
To view the discussion, visit: http://community.jboss.org/message/589595#589595
--------------------------------------------------------------
Can someone from the community please help ? We are now upto 29 messages and we keep
getting more messages stuck in the queue. I would really appreciate it if someone can help
me with how to diagnose the cuase and what I can do to 'release' these stuck
messages.
Thanks
Gurvinder
--------------------------------------------------------------
Reply to this message by going to Community
[http://community.jboss.org/message/589595#589595]
Start a new discussion in JBoss Messaging at Community
[http://community.jboss.org/choose-container!input.jspa?contentType=1&...]
8:46 p.m.
New subject: [JBoss Messaging] - Re: Messages Stuck in 'Delivering State'.
Anders Welen [http://community.jboss.org/people/Welle] created the discussion
"Re: Messages Stuck in 'Delivering State'."
To view the discussion, visit: http://community.jboss.org/message/589599#589599
--------------------------------------------------------------
Can it be that some of the threads that reads the messages are hanging? I would make a
threaddump and browse through iu
Not much of a help, though...
--------------------------------------------------------------
Reply to this message by going to Community
[http://community.jboss.org/message/589599#589599]
Start a new discussion in JBoss Messaging at Community
[http://community.jboss.org/choose-container!input.jspa?contentType=1&...]
2:48 p.m.
New subject: [JBoss Messaging] - Re: Messages Stuck in 'Delivering State'.
Gurvinderpal Narula [http://community.jboss.org/people/groovenarula] created the
discussion
"Re: Messages Stuck in 'Delivering State'."
To view the discussion, visit: http://community.jboss.org/message/589743#589743
--------------------------------------------------------------
Well a server restart flushed out the messages that were in delivering state. But I'm
now wondering if this is something we are going to have to do on a regular basis....
--------------------------------------------------------------
Reply to this message by going to Community
[http://community.jboss.org/message/589743#589743]
Start a new discussion in JBoss Messaging at Community
[http://community.jboss.org/choose-container!input.jspa?contentType=1&...]
3:10 p.m.
New subject: [JBoss Messaging] - Re: Messages Stuck in 'Delivering State'.
Anders Welen [http://community.jboss.org/people/Welle] created the discussion
"Re: Messages Stuck in 'Delivering State'."
To view the discussion, visit: http://community.jboss.org/message/589754#589754
--------------------------------------------------------------
The reason why a restart "solved it" is because you stop the deliver and the
message "returns" to the queue/topic and then re-read after the restart by a
thread that didn't hang.
You should not have to restart the server!
--------------------------------------------------------------
Reply to this message by going to Community
[http://community.jboss.org/message/589754#589754]
Start a new discussion in JBoss Messaging at Community
[http://community.jboss.org/choose-container!input.jspa?contentType=1&...]
5:34 p.m.
New subject: [JBoss Messaging] - Re: Messages Stuck in 'Delivering State'.
Ryan Hochstetler [http://community.jboss.org/people/ryanhos] created the discussion
"Re: Messages Stuck in 'Delivering State'."
To view the discussion, visit: http://community.jboss.org/message/590685#590685
--------------------------------------------------------------
Gurvinderpal,
We're experiencing the "stuck in delivering" issue also, though on a
clustered setup. It occurs on queues consumed by MDBs and by JBoss ESB. We don't yet
have a solution, but I'm closing-in on one. I'm wodering if it isn't a
remoting timeout issue when the server is under high load (we're doing load testing)
or during a stop-the-world GC. This poster has had success increasing the remoting
timeouts: http://community.jboss.org/message/442276#442276
http://community.jboss.org/message/442276#442276
The next time you notice this, check the consumer count on the queue. Is it the expected
value, or is it zero? (I believe MDB-consumed queues should always have one consumer.)
What about the +delivering count?+ Does it match the number of stuck messages? What
happens if you stop and start the MDB consuming that queue via the jmx-console? The
+delivering+ messages will disappear, but do the messages actually get consumed? (have
they perhaps already been consumed? Can you prove this via the logs?) The more evidence
and information we gather about this, the closer we get to a solution. However, most of
JBoss' attention has been turned toward HornetQ (according to my limited powers of
observation). Even if you find the exact source of the problem, a solution may not arrive
quickly or at all, unless you've paid-up your JBoss credits.
--------------------------------------------------------------
Reply to this message by going to Community
[http://community.jboss.org/message/590685#590685]
Start a new discussion in JBoss Messaging at Community
[http://community.jboss.org/choose-container!input.jspa?contentType=1&...]
7:35 a.m.
New subject: [JBoss Messaging] - Re: Messages Stuck in 'Delivering State'.
KVMKrishna Reddy [http://community.jboss.org/people/dasmurali] created the discussion
"Re: Messages Stuck in 'Delivering State'."
To view the discussion, visit: http://community.jboss.org/message/596100#596100
--------------------------------------------------------------
Hi,
We are alos facing the same issue with JBoss Messaging (JBoss app server 5.1.0). After
processing some messages, the Messages are "stuck" in the queue with both
"MessageCount" and "DeliveryCount" as NON ZERO values and
"Consumer Count" as ONE(we had one MDB deployed in the same server).
We are using JBoss Messaging(1.4.3.GA) and JBoss remote(2.5.1) comes by default with JBoss
5.1.0.GA app server.
App server is running on 64 bit linux m/c with 2GB RAM.
Please find the below thread dump at the time of "delivery stuck".
<code>
*Thread: WorkManager(2)-13* : priority:5, demon:true, threadId:102, threadState:WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1767)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
java.lang.Thread.run(Thread.java:595)
*Thread: WorkManager(2)-14* : priority:5, demon:true, threadId:103, threadState:WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1767)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
java.lang.Thread.run(Thread.java:595)
*Thread: WorkManager(2)-15* : priority:5, demon:true, threadId:104, threadState:WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1767)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
java.lang.Thread.run(Thread.java:595)
*Thread: WorkManager(2)-16* : priority:5, demon:true, threadId:105, threadState:WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1767)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
java.lang.Thread.run(Thread.java:595)
*Thread: WorkManager(2)-17* : priority:5, demon:true, threadId:106, threadState:WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1767)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
java.lang.Thread.run(Thread.java:595)
*Thread: WorkManager(2)-18* : priority:5, demon:true, threadId:107, threadState:WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1767)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
java.lang.Thread.run(Thread.java:595)
*Thread: WorkManager(2)-19* : priority:5, demon:true, threadId:108, threadState:WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1767)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
java.lang.Thread.run(Thread.java:595)
</code>
I am thinking, these are MDB (deployed in the same server) threads waiting to pick the
messages. Not sure why they all are in "WAITING" state.
The same application is working damn good in JBoss 4.0.4 server.
Thanks and Regards
Murali Reddy
--------------------------------------------------------------
Reply to this message by going to Community
[http://community.jboss.org/message/596100#596100]
Start a new discussion in JBoss Messaging at Community
[http://community.jboss.org/choose-container!input.jspa?contentType=1&...]
4839
days inactive
4878
days old
6 comments
4 participants
participants (4)
-
Anders Welen -
Gurvinderpal Narula
-
KVMKrishna Reddy
-
Ryan Hochstetler