[esb-issues] [JBoss JIRA] Commented: (JBESB-3473) Message Loss or premature execution termination in an action during shutdown
Dave Siracusa (JIRA)
jira-events at lists.jboss.org
Mon Sep 13 10:29:12 EDT 2010
[ https://jira.jboss.org/browse/JBESB-3473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12550548#action_12550548 ]
Dave Siracusa commented on JBESB-3473:
--------------------------------------
Hi - So I'm straight, if the action thread is busy it will / will not be aborted?
W/re/ to shutdown cleanup I've encountered message loss because my services aren't shutdown in order.
I have not found any way to specify shutdown order within the same ESB.
What I have found is that messages are trying to be delivered to services which are no longer listening.
I've had to incorporate an orderly shutdown of listeners prior to shutdown, specified as such (below). In order to eliminate the potential for message loss.
<property name="services">
<!-- List the services in the order they need to be shutdown -->
<!-- They will be started in opposite order -->
<jmx-property name="0" value="jboss.esb:deployment=bpgateway-1.0.0-SNAPSHOT.esb,gateway-name=Main Gateway (protected),target-service-category=Business Profile Services,target-service-name=Gateway Service" />
<jmx-property name="1" value="jboss.esb:deployment=bpgateway-1.0.0-SNAPSHOT.esb,listener-name=scheduled_catchall_services_listener,service-category=Business Profile Services,service-name=CatchAll Management Service"/>
<jmx-property name="2" value="jboss.esb:deployment=bpgateway-1.0.0-SNAPSHOT.esb,listener-name=Main Gateway,service-category=Business Profile Services,service-name=Gateway Service"/>
<jmx-property name="3" value="jboss.esb:deployment=bpgateway-1.0.0-SNAPSHOT.esb,listener-name=Catch Listener,service-category=Business Profile Services,service-name=CatchAll Service"/>
<jmx-property name="4" value="jboss.esb:deployment=bpgateway-1.0.0-SNAPSHOT.esb,listener-name=scheduled_poll_services_listener,service-category=Business Profile Services,service-name=Poll Listener Service"/>
<jmx-property name="5" value="jboss.esb:deployment=bpesbservices-1.0.0-SNAPSHOT.esb,gateway-name=Main Gateway,target-service-category=Business Profile Services,target-service-name=Router Service" />
<jmx-property name="6" value="jboss.esb:deployment=bpesbservices-1.0.0-SNAPSHOT.esb,listener-name=Main Bus,service-category=Business Profile Services,service-name=Router Service"/>
<jmx-property name="7" value="jboss.esb:deployment=bpesbservices-1.0.0-SNAPSHOT.esb,listener-name=AddressHub Bus,service-category=Business Profile Services,service-name=AddressHub Service" />
<jmx-property name="8" value="jboss.esb:deployment=bpesbservices-1.0.0-SNAPSHOT.esb,listener-name=BusinessProfile Bus,service-category=Business Profile Services,service-name=BusinessProfile Service" />
<jmx-property name="9" value="jboss.esb:deployment=bpesbservices-1.0.0-SNAPSHOT.esb,listener-name=NullWork Bus,service-category=Business Profile Services,service-name=Null Work Service" />
<jmx-property name="10" value="jboss.esb:deployment=bpesbservices-1.0.0-SNAPSHOT.esb,listener-name=SimulateWork Bus,service-category=Business Profile Services,service-name=Simulate Work Service" />
<jmx-property name="11" value="jboss.esb:deployment=bpesbservices-1.0.0-SNAPSHOT.esb,listener-name=Lost Message Service,service-category=Business Profile Services,service-name=Monitor Service"/>
<jmx-property name="12" value="jboss.esb:deployment=bpesbservices-1.0.0-SNAPSHOT.esb,listener-name=Lost Bus,service-category=Business Profile Services,service-name=Lost Message Service"/>
<jmx-property name="13" value="jboss.esb:deployment=bpesbservices-1.0.0-SNAPSHOT.esb,listener-name=ResponseHub Bus,service-category=Business Profile Services,service-name=SendResponse Service" />
<jmx-property name="14" value="jboss.esb:deployment=bpesbservices-1.0.0-SNAPSHOT.esb,listener-name=scheduled_listener,service-category=Business Profile Services,service-name=Audit Service"/>
</property>
> Message Loss or premature execution termination in an action during shutdown
> ----------------------------------------------------------------------------
>
> Key: JBESB-3473
> URL: https://jira.jboss.org/browse/JBESB-3473
> Project: JBoss ESB
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Components: Rosetta
> Affects Versions: 4.9
> Environment: RHEL 64bit
> Reporter: Dave Siracusa
> Assignee: Kevin Conner
> Fix For: 4.9 CP1
>
>
> I'm able to reproduce JMS message loss or premature action termination during shutdown. I enqueued several thousand mesages into an ESB application. While the messages were being processed I toggle (stop/start) lifecycle state for the acting jms listeners via JMX in order to simulate shutdown. I noticed periodic warnings in the serverlog (listing 1). The final tally indicated message loss. Via code inspection the bug appears to be self-evident.
> Basically when the doStop method (JMXConsole) is called it sets state to STOPPING and immediately terminates the executor thread (MessageAwareListener.java). If the thread was currently processing bad things happen.
> I'm using transcated queues and it appears the JMS message transaction evaporates and the message is not returned to the queue. I added some code to AbstractThreadedManagedLifecycle.java to remedy the issue. It uses the default terminationPeriod (60 seconds), unless it is specified in the jboss-esb.xml file.
> AbstractThreadedManagedLifecycle.java
> ---------------------------------------
> protected void doStop()
> throws ManagedLifecycleException
> {
> runningLock.lock() ;
> try
> {
> if (isRunning())
> {
> setRunning(ManagedLifecycleThreadStatae.STOPPING) ;
> }
> // Dave Siracusa -start
> if (!waitUntilStopped())
> {
> throw new ManagedLifecycleException("Thread still active!") ;
> } // Dave Siracusa - end
> }
> finally
> {
> runningLock.unlock() ;
> }
> }
> MessageAwareListener.java
> ---------------------------
> protected void doStop()
> throws ManagedLifecycleException
> {
> super.doStop();
> _execService.shutdown() ;
> }
> Listing 1
> ----------
> 2010-09-01 15:04:40,924 WARN [org.jboss.soa.esb.listeners.lifecycle.AbstractThreadedManagedLifecycle] (Thread-200) Unexpected error from doRun()
> java.util.concurrent.RejectedExecutionException
> at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecutionThreadPoolExecutor.java:1768)
> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
> at org.jboss.soa.esb.listeners.message.MessageAwareListener.waitForEventAndProcess(MessageAwareListener.java:359)
> at org.jboss.soa.esb.listeners.message.MessageAwareListener.doRun(MessageAwareListener.java:253)
> at org.jboss.soa.esb.listeners.lifecycle.AbstractThreadedManagedLifecycle.run(AbstractThreadedManagedLifecycle.java:115)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
More information about the esb-issues
mailing list