[JBoss JIRA] Created: (JBESB-3473) Message Loss or premature execution termination in an action during shutdown
6:52 a.m.
Message Loss or premature execution termination in an action during shutdown
----------------------------------------------------------------------------
Key: JBESB-3473
URL: https://jira.jboss.org/browse/JBESB-3473
Project: JBoss ESB
Issue Type: Bug
Security Level: Public (Everyone can see)
Affects Versions: 5.0
Environment: RHEL 64bit
Reporter: Dave Siracusa
I'm able to reproduce JMS message loss or premature action termination during
shutdown. I enqueued several thousand mesages into an ESB application. While the
messages were being processed I toggle (stop/start) lifecycle state for the acting jms
listeners via JMX in order to simulate shutdown. I noticed periodic warnings in the
serverlog (listing 1). The final tally indicated message loss. Via code inspection the
bug appears to be self-evident.
Basically when the doStop method (JMXConsole) is called it sets state to STOPPING and
immediately terminates the executor thread (MessageAwareListener.java). If the thread was
currently processing bad things happen.
I'm using transcated queues and it appears the JMS message transaction evaporates and
the message is not returned to the queue. I added some code to
AbstractThreadedManagedLifecycle.java to remedy the issue. It uses the default
terminationPeriod (60 seconds), unless it is specified in the jboss-esb.xml file.
AbstractThreadedManagedLifecycle.java
---------------------------------------
protected void doStop()
throws ManagedLifecycleException
{
runningLock.lock() ;
try
{
if (isRunning())
{
setRunning(ManagedLifecycleThreadStatae.STOPPING) ;
}
// Dave Siracusa -start
if (!waitUntilStopped())
{
throw new ManagedLifecycleException("Thread still active!") ;
} // Dave Siracusa - end
}
finally
{
runningLock.unlock() ;
}
}
MessageAwareListener.java
---------------------------
protected void doStop()
throws ManagedLifecycleException
{
super.doStop();
_execService.shutdown() ;
}
Listing 1
----------
2010-09-01 15:04:40,924 WARN
[org.jboss.soa.esb.listeners.lifecycle.AbstractThreadedManagedLifecycle] (Thread-200)
Unexpected error from doRun()
java.util.concurrent.RejectedExecutionException
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecutionThreadPoolExecutor.java:1768)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.jboss.soa.esb.listeners.message.MessageAwareListener.waitForEventAndProcess(MessageAwareListener.java:359)
at
org.jboss.soa.esb.listeners.message.MessageAwareListener.doRun(MessageAwareListener.java:253)
at
org.jboss.soa.esb.listeners.lifecycle.AbstractThreadedManagedLifecycle.run(AbstractThreadedManagedLifecycle.java:115)
at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:48 a.m.
New subject: [JBoss JIRA] Updated: (JBESB-3473) Message Loss or premature execution termination in an action during shutdown
[
https://jira.jboss.org/browse/JBESB-3473?page=com.atlassian.jira.plugin.s...
]
David Boeren updated JBESB-3473:
--------------------------------
Message Loss or premature execution termination in an action during
shutdown
----------------------------------------------------------------------------
Key: JBESB-3473
URL: https://jira.jboss.org/browse/JBESB-3473
Project: JBoss ESB
Issue Type: Bug
Security Level: Public(Everyone can see)
Affects Versions: 5.0
Environment: RHEL 64bit
Reporter: Dave Siracusa
I'm able to reproduce JMS message loss or premature action termination during
shutdown. I enqueued several thousand mesages into an ESB application. While the
messages were being processed I toggle (stop/start) lifecycle state for the acting jms
listeners via JMX in order to simulate shutdown. I noticed periodic warnings in the
serverlog (listing 1). The final tally indicated message loss. Via code inspection the
bug appears to be self-evident.
Basically when the doStop method (JMXConsole) is called it sets state to STOPPING and
immediately terminates the executor thread (MessageAwareListener.java). If the thread was
currently processing bad things happen.
I'm using transcated queues and it appears the JMS message transaction evaporates and
the message is not returned to the queue. I added some code to
AbstractThreadedManagedLifecycle.java to remedy the issue. It uses the default
terminationPeriod (60 seconds), unless it is specified in the jboss-esb.xml file.
AbstractThreadedManagedLifecycle.java
---------------------------------------
protected void doStop()
throws ManagedLifecycleException
{
runningLock.lock() ;
try
{
if (isRunning())
{
setRunning(ManagedLifecycleThreadStatae.STOPPING) ;
}
// Dave Siracusa -start
if (!waitUntilStopped())
{
throw new ManagedLifecycleException("Thread still active!") ;
} // Dave Siracusa - end
}
finally
{
runningLock.unlock() ;
}
}
MessageAwareListener.java
---------------------------
protected void doStop()
throws ManagedLifecycleException
{
super.doStop();
_execService.shutdown() ;
}
Listing 1
----------
2010-09-01 15:04:40,924 WARN
[org.jboss.soa.esb.listeners.lifecycle.AbstractThreadedManagedLifecycle] (Thread-200)
Unexpected error from doRun()
java.util.concurrent.RejectedExecutionException
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecutionThreadPoolExecutor.java:1768)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.jboss.soa.esb.listeners.message.MessageAwareListener.waitForEventAndProcess(MessageAwareListener.java:359)
at
org.jboss.soa.esb.listeners.message.MessageAwareListener.doRun(MessageAwareListener.java:253)
at
org.jboss.soa.esb.listeners.lifecycle.AbstractThreadedManagedLifecycle.run(AbstractThreadedManagedLifecycle.java:115)
at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:03 a.m.
New subject: [JBoss JIRA] Updated: (JBESB-3473) Message Loss or premature execution termination in an action during shutdown
[
https://jira.jboss.org/browse/JBESB-3473?page=com.atlassian.jira.plugin.s...
]
Kevin Conner updated JBESB-3473:
--------------------------------
Assignee: Kevin Conner
Fix Version/s: 4.9 CP1
Affects Version/s: 4.9
(was: 5.0)
Message Loss or premature execution termination in an action during
shutdown
----------------------------------------------------------------------------
Key: JBESB-3473
URL: https://jira.jboss.org/browse/JBESB-3473
Project: JBoss ESB
Issue Type: Bug
Security Level: Public(Everyone can see)
Affects Versions: 4.9
Environment: RHEL 64bit
Reporter: Dave Siracusa
Assignee: Kevin Conner
Fix For: 4.9 CP1
I'm able to reproduce JMS message loss or premature action termination during
shutdown. I enqueued several thousand mesages into an ESB application. While the
messages were being processed I toggle (stop/start) lifecycle state for the acting jms
listeners via JMX in order to simulate shutdown. I noticed periodic warnings in the
serverlog (listing 1). The final tally indicated message loss. Via code inspection the
bug appears to be self-evident.
Basically when the doStop method (JMXConsole) is called it sets state to STOPPING and
immediately terminates the executor thread (MessageAwareListener.java). If the thread was
currently processing bad things happen.
I'm using transcated queues and it appears the JMS message transaction evaporates and
the message is not returned to the queue. I added some code to
AbstractThreadedManagedLifecycle.java to remedy the issue. It uses the default
terminationPeriod (60 seconds), unless it is specified in the jboss-esb.xml file.
AbstractThreadedManagedLifecycle.java
---------------------------------------
protected void doStop()
throws ManagedLifecycleException
{
runningLock.lock() ;
try
{
if (isRunning())
{
setRunning(ManagedLifecycleThreadStatae.STOPPING) ;
}
// Dave Siracusa -start
if (!waitUntilStopped())
{
throw new ManagedLifecycleException("Thread still active!") ;
} // Dave Siracusa - end
}
finally
{
runningLock.unlock() ;
}
}
MessageAwareListener.java
---------------------------
protected void doStop()
throws ManagedLifecycleException
{
super.doStop();
_execService.shutdown() ;
}
Listing 1
----------
2010-09-01 15:04:40,924 WARN
[org.jboss.soa.esb.listeners.lifecycle.AbstractThreadedManagedLifecycle] (Thread-200)
Unexpected error from doRun()
java.util.concurrent.RejectedExecutionException
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecutionThreadPoolExecutor.java:1768)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.jboss.soa.esb.listeners.message.MessageAwareListener.waitForEventAndProcess(MessageAwareListener.java:359)
at
org.jboss.soa.esb.listeners.message.MessageAwareListener.doRun(MessageAwareListener.java:253)
at
org.jboss.soa.esb.listeners.lifecycle.AbstractThreadedManagedLifecycle.run(AbstractThreadedManagedLifecycle.java:115)
at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:17 a.m.
New subject: [JBoss JIRA] Commented: (JBESB-3473) Message Loss or premature execution termination in an action during shutdown
[
https://jira.jboss.org/browse/JBESB-3473?page=com.atlassian.jira.plugin.s...
]
Kevin Conner commented on JBESB-3473:
-------------------------------------
Changing the MessageAwareListener code to the following should be sufficient.
if (null != message)
{
final Message pipelineMessage = message ;
final TransactionalRunner txRunner ;
try
{
final Object txHandle = transactionStrategy.suspend();
txRunner = new TransactionalRunner(pickUpCourier, pipelineMessage, txHandle);
}
catch (TransactionStrategyException ex)
{
_logger.warn("Caught transaction related exception: ", ex);
cleanCourier(pickUpCourier);
rollbackTransaction();
return ;
}
updateThreadCount(+1);
try
{
_execService.execute(txRunner);
}
catch (final RejectedExecutionException ree)
{
txRunner.run() ;
}
}
I'll trying to get a test case now, will then test and commit.
Also, using jms-jca-provider rather than jms-provider would use the JCA inflow codebase
and not the MessageAwareListener codebase.
Message Loss or premature execution termination in an action during
shutdown
----------------------------------------------------------------------------
Key: JBESB-3473
URL: https://jira.jboss.org/browse/JBESB-3473
Project: JBoss ESB
Issue Type: Bug
Security Level: Public(Everyone can see)
Affects Versions: 4.9
Environment: RHEL 64bit
Reporter: Dave Siracusa
Assignee: Kevin Conner
Fix For: 4.9 CP1
I'm able to reproduce JMS message loss or premature action termination during
shutdown. I enqueued several thousand mesages into an ESB application. While the
messages were being processed I toggle (stop/start) lifecycle state for the acting jms
listeners via JMX in order to simulate shutdown. I noticed periodic warnings in the
serverlog (listing 1). The final tally indicated message loss. Via code inspection the
bug appears to be self-evident.
Basically when the doStop method (JMXConsole) is called it sets state to STOPPING and
immediately terminates the executor thread (MessageAwareListener.java). If the thread was
currently processing bad things happen.
I'm using transcated queues and it appears the JMS message transaction evaporates and
the message is not returned to the queue. I added some code to
AbstractThreadedManagedLifecycle.java to remedy the issue. It uses the default
terminationPeriod (60 seconds), unless it is specified in the jboss-esb.xml file.
AbstractThreadedManagedLifecycle.java
---------------------------------------
protected void doStop()
throws ManagedLifecycleException
{
runningLock.lock() ;
try
{
if (isRunning())
{
setRunning(ManagedLifecycleThreadStatae.STOPPING) ;
}
// Dave Siracusa -start
if (!waitUntilStopped())
{
throw new ManagedLifecycleException("Thread still active!") ;
} // Dave Siracusa - end
}
finally
{
runningLock.unlock() ;
}
}
MessageAwareListener.java
---------------------------
protected void doStop()
throws ManagedLifecycleException
{
super.doStop();
_execService.shutdown() ;
}
Listing 1
----------
2010-09-01 15:04:40,924 WARN
[org.jboss.soa.esb.listeners.lifecycle.AbstractThreadedManagedLifecycle] (Thread-200)
Unexpected error from doRun()
java.util.concurrent.RejectedExecutionException
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecutionThreadPoolExecutor.java:1768)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.jboss.soa.esb.listeners.message.MessageAwareListener.waitForEventAndProcess(MessageAwareListener.java:359)
at
org.jboss.soa.esb.listeners.message.MessageAwareListener.doRun(MessageAwareListener.java:253)
at
org.jboss.soa.esb.listeners.lifecycle.AbstractThreadedManagedLifecycle.run(AbstractThreadedManagedLifecycle.java:115)
at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:02 p.m.
New subject: [JBoss JIRA] Commented: (JBESB-3473) Message Loss or premature execution termination in an action during shutdown
[
https://jira.jboss.org/browse/JBESB-3473?page=com.atlassian.jira.plugin.s...
]
Dave Siracusa commented on JBESB-3473:
--------------------------------------
Hi Kevin
I may be reading the code incorrectly.... if so please disregard the following:
Does your fix tolerate the premature terminatation, and roll back to prevent loss?
If so, I prefer a graceful shutdown. Not all actions can be protected via XA.
After all there is a service attribute to convey shutdown preferences
(terminationPeriod).
--Dave
Message Loss or premature execution termination in an action during
shutdown
----------------------------------------------------------------------------
Key: JBESB-3473
URL: https://jira.jboss.org/browse/JBESB-3473
Project: JBoss ESB
Issue Type: Bug
Security Level: Public(Everyone can see)
Affects Versions: 4.9
Environment: RHEL 64bit
Reporter: Dave Siracusa
Assignee: Kevin Conner
Fix For: 4.9 CP1
I'm able to reproduce JMS message loss or premature action termination during
shutdown. I enqueued several thousand mesages into an ESB application. While the
messages were being processed I toggle (stop/start) lifecycle state for the acting jms
listeners via JMX in order to simulate shutdown. I noticed periodic warnings in the
serverlog (listing 1). The final tally indicated message loss. Via code inspection the
bug appears to be self-evident.
Basically when the doStop method (JMXConsole) is called it sets state to STOPPING and
immediately terminates the executor thread (MessageAwareListener.java). If the thread was
currently processing bad things happen.
I'm using transcated queues and it appears the JMS message transaction evaporates and
the message is not returned to the queue. I added some code to
AbstractThreadedManagedLifecycle.java to remedy the issue. It uses the default
terminationPeriod (60 seconds), unless it is specified in the jboss-esb.xml file.
AbstractThreadedManagedLifecycle.java
---------------------------------------
protected void doStop()
throws ManagedLifecycleException
{
runningLock.lock() ;
try
{
if (isRunning())
{
setRunning(ManagedLifecycleThreadStatae.STOPPING) ;
}
// Dave Siracusa -start
if (!waitUntilStopped())
{
throw new ManagedLifecycleException("Thread still active!") ;
} // Dave Siracusa - end
}
finally
{
runningLock.unlock() ;
}
}
MessageAwareListener.java
---------------------------
protected void doStop()
throws ManagedLifecycleException
{
super.doStop();
_execService.shutdown() ;
}
Listing 1
----------
2010-09-01 15:04:40,924 WARN
[org.jboss.soa.esb.listeners.lifecycle.AbstractThreadedManagedLifecycle] (Thread-200)
Unexpected error from doRun()
java.util.concurrent.RejectedExecutionException
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecutionThreadPoolExecutor.java:1768)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.jboss.soa.esb.listeners.message.MessageAwareListener.waitForEventAndProcess(MessageAwareListener.java:359)
at
org.jboss.soa.esb.listeners.message.MessageAwareListener.doRun(MessageAwareListener.java:253)
at
org.jboss.soa.esb.listeners.lifecycle.AbstractThreadedManagedLifecycle.run(AbstractThreadedManagedLifecycle.java:115)
at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:32 a.m.
New subject: [JBoss JIRA] Commented: (JBESB-3473) Message Loss or premature execution termination in an action during shutdown
[
https://jira.jboss.org/browse/JBESB-3473?page=com.atlassian.jira.plugin.s...
]
Kevin Conner commented on JBESB-3473:
-------------------------------------
There is already a graceful shutdown and the txrunner will handle the last message if it
is missed because of the race.
Message Loss or premature execution termination in an action during
shutdown
----------------------------------------------------------------------------
Key: JBESB-3473
URL: https://jira.jboss.org/browse/JBESB-3473
Project: JBoss ESB
Issue Type: Bug
Security Level: Public(Everyone can see)
Affects Versions: 4.9
Environment: RHEL 64bit
Reporter: Dave Siracusa
Assignee: Kevin Conner
Fix For: 4.9 CP1
I'm able to reproduce JMS message loss or premature action termination during
shutdown. I enqueued several thousand mesages into an ESB application. While the
messages were being processed I toggle (stop/start) lifecycle state for the acting jms
listeners via JMX in order to simulate shutdown. I noticed periodic warnings in the
serverlog (listing 1). The final tally indicated message loss. Via code inspection the
bug appears to be self-evident.
Basically when the doStop method (JMXConsole) is called it sets state to STOPPING and
immediately terminates the executor thread (MessageAwareListener.java). If the thread was
currently processing bad things happen.
I'm using transcated queues and it appears the JMS message transaction evaporates and
the message is not returned to the queue. I added some code to
AbstractThreadedManagedLifecycle.java to remedy the issue. It uses the default
terminationPeriod (60 seconds), unless it is specified in the jboss-esb.xml file.
AbstractThreadedManagedLifecycle.java
---------------------------------------
protected void doStop()
throws ManagedLifecycleException
{
runningLock.lock() ;
try
{
if (isRunning())
{
setRunning(ManagedLifecycleThreadStatae.STOPPING) ;
}
// Dave Siracusa -start
if (!waitUntilStopped())
{
throw new ManagedLifecycleException("Thread still active!") ;
} // Dave Siracusa - end
}
finally
{
runningLock.unlock() ;
}
}
MessageAwareListener.java
---------------------------
protected void doStop()
throws ManagedLifecycleException
{
super.doStop();
_execService.shutdown() ;
}
Listing 1
----------
2010-09-01 15:04:40,924 WARN
[org.jboss.soa.esb.listeners.lifecycle.AbstractThreadedManagedLifecycle] (Thread-200)
Unexpected error from doRun()
java.util.concurrent.RejectedExecutionException
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecutionThreadPoolExecutor.java:1768)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.jboss.soa.esb.listeners.message.MessageAwareListener.waitForEventAndProcess(MessageAwareListener.java:359)
at
org.jboss.soa.esb.listeners.message.MessageAwareListener.doRun(MessageAwareListener.java:253)
at
org.jboss.soa.esb.listeners.lifecycle.AbstractThreadedManagedLifecycle.run(AbstractThreadedManagedLifecycle.java:115)
at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:34 a.m.
New subject: [JBoss JIRA] Commented: (JBESB-3473) Message Loss or premature execution termination in an action during shutdown
[
https://jira.jboss.org/browse/JBESB-3473?page=com.atlassian.jira.plugin.s...
]
Kevin Conner commented on JBESB-3473:
-------------------------------------
BTW the stop method is not intended to wait until everything has stopped, all it does is
signal the start of that process. The destroy/start methods will handle the check for the
stopped state and the terminationPeriod.
Message Loss or premature execution termination in an action during
shutdown
----------------------------------------------------------------------------
Key: JBESB-3473
URL: https://jira.jboss.org/browse/JBESB-3473
Project: JBoss ESB
Issue Type: Bug
Security Level: Public(Everyone can see)
Affects Versions: 4.9
Environment: RHEL 64bit
Reporter: Dave Siracusa
Assignee: Kevin Conner
Fix For: 4.9 CP1
I'm able to reproduce JMS message loss or premature action termination during
shutdown. I enqueued several thousand mesages into an ESB application. While the
messages were being processed I toggle (stop/start) lifecycle state for the acting jms
listeners via JMX in order to simulate shutdown. I noticed periodic warnings in the
serverlog (listing 1). The final tally indicated message loss. Via code inspection the
bug appears to be self-evident.
Basically when the doStop method (JMXConsole) is called it sets state to STOPPING and
immediately terminates the executor thread (MessageAwareListener.java). If the thread was
currently processing bad things happen.
I'm using transcated queues and it appears the JMS message transaction evaporates and
the message is not returned to the queue. I added some code to
AbstractThreadedManagedLifecycle.java to remedy the issue. It uses the default
terminationPeriod (60 seconds), unless it is specified in the jboss-esb.xml file.
AbstractThreadedManagedLifecycle.java
---------------------------------------
protected void doStop()
throws ManagedLifecycleException
{
runningLock.lock() ;
try
{
if (isRunning())
{
setRunning(ManagedLifecycleThreadStatae.STOPPING) ;
}
// Dave Siracusa -start
if (!waitUntilStopped())
{
throw new ManagedLifecycleException("Thread still active!") ;
} // Dave Siracusa - end
}
finally
{
runningLock.unlock() ;
}
}
MessageAwareListener.java
---------------------------
protected void doStop()
throws ManagedLifecycleException
{
super.doStop();
_execService.shutdown() ;
}
Listing 1
----------
2010-09-01 15:04:40,924 WARN
[org.jboss.soa.esb.listeners.lifecycle.AbstractThreadedManagedLifecycle] (Thread-200)
Unexpected error from doRun()
java.util.concurrent.RejectedExecutionException
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecutionThreadPoolExecutor.java:1768)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.jboss.soa.esb.listeners.message.MessageAwareListener.waitForEventAndProcess(MessageAwareListener.java:359)
at
org.jboss.soa.esb.listeners.message.MessageAwareListener.doRun(MessageAwareListener.java:253)
at
org.jboss.soa.esb.listeners.lifecycle.AbstractThreadedManagedLifecycle.run(AbstractThreadedManagedLifecycle.java:115)
at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:30 a.m.
New subject: [JBoss JIRA] Updated: (JBESB-3473) Message Loss or premature execution termination in an action during shutdown
[
https://jira.jboss.org/browse/JBESB-3473?page=com.atlassian.jira.plugin.s...
]
Kevin Conner updated JBESB-3473:
--------------------------------
Component/s: Rosetta
Message Loss or premature execution termination in an action during
shutdown
----------------------------------------------------------------------------
Key: JBESB-3473
URL: https://jira.jboss.org/browse/JBESB-3473
Project: JBoss ESB
Issue Type: Bug
Security Level: Public(Everyone can see)
Components: Rosetta
Affects Versions: 4.9
Environment: RHEL 64bit
Reporter: Dave Siracusa
Assignee: Kevin Conner
Fix For: 4.9 CP1
I'm able to reproduce JMS message loss or premature action termination during
shutdown. I enqueued several thousand mesages into an ESB application. While the
messages were being processed I toggle (stop/start) lifecycle state for the acting jms
listeners via JMX in order to simulate shutdown. I noticed periodic warnings in the
serverlog (listing 1). The final tally indicated message loss. Via code inspection the
bug appears to be self-evident.
Basically when the doStop method (JMXConsole) is called it sets state to STOPPING and
immediately terminates the executor thread (MessageAwareListener.java). If the thread was
currently processing bad things happen.
I'm using transcated queues and it appears the JMS message transaction evaporates and
the message is not returned to the queue. I added some code to
AbstractThreadedManagedLifecycle.java to remedy the issue. It uses the default
terminationPeriod (60 seconds), unless it is specified in the jboss-esb.xml file.
AbstractThreadedManagedLifecycle.java
---------------------------------------
protected void doStop()
throws ManagedLifecycleException
{
runningLock.lock() ;
try
{
if (isRunning())
{
setRunning(ManagedLifecycleThreadStatae.STOPPING) ;
}
// Dave Siracusa -start
if (!waitUntilStopped())
{
throw new ManagedLifecycleException("Thread still active!") ;
} // Dave Siracusa - end
}
finally
{
runningLock.unlock() ;
}
}
MessageAwareListener.java
---------------------------
protected void doStop()
throws ManagedLifecycleException
{
super.doStop();
_execService.shutdown() ;
}
Listing 1
----------
2010-09-01 15:04:40,924 WARN
[org.jboss.soa.esb.listeners.lifecycle.AbstractThreadedManagedLifecycle] (Thread-200)
Unexpected error from doRun()
java.util.concurrent.RejectedExecutionException
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecutionThreadPoolExecutor.java:1768)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.jboss.soa.esb.listeners.message.MessageAwareListener.waitForEventAndProcess(MessageAwareListener.java:359)
at
org.jboss.soa.esb.listeners.message.MessageAwareListener.doRun(MessageAwareListener.java:253)
at
org.jboss.soa.esb.listeners.lifecycle.AbstractThreadedManagedLifecycle.run(AbstractThreadedManagedLifecycle.java:115)
at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:32 a.m.
New subject: [JBoss JIRA] Closed: (JBESB-3473) Message Loss or premature execution termination in an action during shutdown
[
https://jira.jboss.org/browse/JBESB-3473?page=com.atlassian.jira.plugin.s...
]
Kevin Conner closed JBESB-3473.
-------------------------------
Resolution: Done
Updated in revision 35127.
Message Loss or premature execution termination in an action during
shutdown
----------------------------------------------------------------------------
Key: JBESB-3473
URL: https://jira.jboss.org/browse/JBESB-3473
Project: JBoss ESB
Issue Type: Bug
Security Level: Public(Everyone can see)
Components: Rosetta
Affects Versions: 4.9
Environment: RHEL 64bit
Reporter: Dave Siracusa
Assignee: Kevin Conner
Fix For: 4.9 CP1
I'm able to reproduce JMS message loss or premature action termination during
shutdown. I enqueued several thousand mesages into an ESB application. While the
messages were being processed I toggle (stop/start) lifecycle state for the acting jms
listeners via JMX in order to simulate shutdown. I noticed periodic warnings in the
serverlog (listing 1). The final tally indicated message loss. Via code inspection the
bug appears to be self-evident.
Basically when the doStop method (JMXConsole) is called it sets state to STOPPING and
immediately terminates the executor thread (MessageAwareListener.java). If the thread was
currently processing bad things happen.
I'm using transcated queues and it appears the JMS message transaction evaporates and
the message is not returned to the queue. I added some code to
AbstractThreadedManagedLifecycle.java to remedy the issue. It uses the default
terminationPeriod (60 seconds), unless it is specified in the jboss-esb.xml file.
AbstractThreadedManagedLifecycle.java
---------------------------------------
protected void doStop()
throws ManagedLifecycleException
{
runningLock.lock() ;
try
{
if (isRunning())
{
setRunning(ManagedLifecycleThreadStatae.STOPPING) ;
}
// Dave Siracusa -start
if (!waitUntilStopped())
{
throw new ManagedLifecycleException("Thread still active!") ;
} // Dave Siracusa - end
}
finally
{
runningLock.unlock() ;
}
}
MessageAwareListener.java
---------------------------
protected void doStop()
throws ManagedLifecycleException
{
super.doStop();
_execService.shutdown() ;
}
Listing 1
----------
2010-09-01 15:04:40,924 WARN
[org.jboss.soa.esb.listeners.lifecycle.AbstractThreadedManagedLifecycle] (Thread-200)
Unexpected error from doRun()
java.util.concurrent.RejectedExecutionException
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecutionThreadPoolExecutor.java:1768)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.jboss.soa.esb.listeners.message.MessageAwareListener.waitForEventAndProcess(MessageAwareListener.java:359)
at
org.jboss.soa.esb.listeners.message.MessageAwareListener.doRun(MessageAwareListener.java:253)
at
org.jboss.soa.esb.listeners.lifecycle.AbstractThreadedManagedLifecycle.run(AbstractThreadedManagedLifecycle.java:115)
at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:29 a.m.
New subject: [JBoss JIRA] Commented: (JBESB-3473) Message Loss or premature execution termination in an action during shutdown
[
https://jira.jboss.org/browse/JBESB-3473?page=com.atlassian.jira.plugin.s...
]
Dave Siracusa commented on JBESB-3473:
--------------------------------------
Hi - So I'm straight, if the action thread is busy it will / will not be aborted?
W/re/ to shutdown cleanup I've encountered message loss because my services aren't
shutdown in order.
I have not found any way to specify shutdown order within the same ESB.
What I have found is that messages are trying to be delivered to services which are no
longer listening.
I've had to incorporate an orderly shutdown of listeners prior to shutdown, specified
as such (below). In order to eliminate the potential for message loss.
<property name="services">
<!-- List the services in the order they need to be shutdown -->
<!-- They will be started in opposite order -->
<jmx-property name="0"
value="jboss.esb:deployment=bpgateway-1.0.0-SNAPSHOT.esb,gateway-name=Main Gateway
(protected),target-service-category=Business Profile Services,target-service-name=Gateway
Service" />
<jmx-property name="1"
value="jboss.esb:deployment=bpgateway-1.0.0-SNAPSHOT.esb,listener-name=scheduled_catchall_services_listener,service-category=Business
Profile Services,service-name=CatchAll Management Service"/>
<jmx-property name="2"
value="jboss.esb:deployment=bpgateway-1.0.0-SNAPSHOT.esb,listener-name=Main
Gateway,service-category=Business Profile Services,service-name=Gateway
Service"/>
<jmx-property name="3"
value="jboss.esb:deployment=bpgateway-1.0.0-SNAPSHOT.esb,listener-name=Catch
Listener,service-category=Business Profile Services,service-name=CatchAll
Service"/>
<jmx-property name="4"
value="jboss.esb:deployment=bpgateway-1.0.0-SNAPSHOT.esb,listener-name=scheduled_poll_services_listener,service-category=Business
Profile Services,service-name=Poll Listener Service"/>
<jmx-property name="5"
value="jboss.esb:deployment=bpesbservices-1.0.0-SNAPSHOT.esb,gateway-name=Main
Gateway,target-service-category=Business Profile Services,target-service-name=Router
Service" />
<jmx-property name="6"
value="jboss.esb:deployment=bpesbservices-1.0.0-SNAPSHOT.esb,listener-name=Main
Bus,service-category=Business Profile Services,service-name=Router Service"/>
<jmx-property name="7"
value="jboss.esb:deployment=bpesbservices-1.0.0-SNAPSHOT.esb,listener-name=AddressHub
Bus,service-category=Business Profile Services,service-name=AddressHub Service"
/>
<jmx-property name="8"
value="jboss.esb:deployment=bpesbservices-1.0.0-SNAPSHOT.esb,listener-name=BusinessProfile
Bus,service-category=Business Profile Services,service-name=BusinessProfile Service"
/>
<jmx-property name="9"
value="jboss.esb:deployment=bpesbservices-1.0.0-SNAPSHOT.esb,listener-name=NullWork
Bus,service-category=Business Profile Services,service-name=Null Work Service" />
<jmx-property name="10"
value="jboss.esb:deployment=bpesbservices-1.0.0-SNAPSHOT.esb,listener-name=SimulateWork
Bus,service-category=Business Profile Services,service-name=Simulate Work Service"
/>
<jmx-property name="11"
value="jboss.esb:deployment=bpesbservices-1.0.0-SNAPSHOT.esb,listener-name=Lost
Message Service,service-category=Business Profile Services,service-name=Monitor
Service"/>
<jmx-property name="12"
value="jboss.esb:deployment=bpesbservices-1.0.0-SNAPSHOT.esb,listener-name=Lost
Bus,service-category=Business Profile Services,service-name=Lost Message
Service"/>
<jmx-property name="13"
value="jboss.esb:deployment=bpesbservices-1.0.0-SNAPSHOT.esb,listener-name=ResponseHub
Bus,service-category=Business Profile Services,service-name=SendResponse Service"
/>
<jmx-property name="14"
value="jboss.esb:deployment=bpesbservices-1.0.0-SNAPSHOT.esb,listener-name=scheduled_listener,service-category=Business
Profile Services,service-name=Audit Service"/>
</property>
Message Loss or premature execution termination in an action during
shutdown
----------------------------------------------------------------------------
Key: JBESB-3473
URL: https://jira.jboss.org/browse/JBESB-3473
Project: JBoss ESB
Issue Type: Bug
Security Level: Public(Everyone can see)
Components: Rosetta
Affects Versions: 4.9
Environment: RHEL 64bit
Reporter: Dave Siracusa
Assignee: Kevin Conner
Fix For: 4.9 CP1
I'm able to reproduce JMS message loss or premature action termination during
shutdown. I enqueued several thousand mesages into an ESB application. While the
messages were being processed I toggle (stop/start) lifecycle state for the acting jms
listeners via JMX in order to simulate shutdown. I noticed periodic warnings in the
serverlog (listing 1). The final tally indicated message loss. Via code inspection the
bug appears to be self-evident.
Basically when the doStop method (JMXConsole) is called it sets state to STOPPING and
immediately terminates the executor thread (MessageAwareListener.java). If the thread was
currently processing bad things happen.
I'm using transcated queues and it appears the JMS message transaction evaporates and
the message is not returned to the queue. I added some code to
AbstractThreadedManagedLifecycle.java to remedy the issue. It uses the default
terminationPeriod (60 seconds), unless it is specified in the jboss-esb.xml file.
AbstractThreadedManagedLifecycle.java
---------------------------------------
protected void doStop()
throws ManagedLifecycleException
{
runningLock.lock() ;
try
{
if (isRunning())
{
setRunning(ManagedLifecycleThreadStatae.STOPPING) ;
}
// Dave Siracusa -start
if (!waitUntilStopped())
{
throw new ManagedLifecycleException("Thread still active!") ;
} // Dave Siracusa - end
}
finally
{
runningLock.unlock() ;
}
}
MessageAwareListener.java
---------------------------
protected void doStop()
throws ManagedLifecycleException
{
super.doStop();
_execService.shutdown() ;
}
Listing 1
----------
2010-09-01 15:04:40,924 WARN
[org.jboss.soa.esb.listeners.lifecycle.AbstractThreadedManagedLifecycle] (Thread-200)
Unexpected error from doRun()
java.util.concurrent.RejectedExecutionException
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecutionThreadPoolExecutor.java:1768)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.jboss.soa.esb.listeners.message.MessageAwareListener.waitForEventAndProcess(MessageAwareListener.java:359)
at
org.jboss.soa.esb.listeners.message.MessageAwareListener.doRun(MessageAwareListener.java:253)
at
org.jboss.soa.esb.listeners.lifecycle.AbstractThreadedManagedLifecycle.run(AbstractThreadedManagedLifecycle.java:115)
at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5000
days inactive
5009
days old
9 comments
3 participants
participants (3)
-
Dave Siracusa (JIRA) -
David Boeren (JIRA)
-
Kevin Conner (JIRA)