[jboss-dev] Shutting down the app server and removing components
Brian Stansberry
brian.stansberry at redhat.com
Fri Feb 20 10:59:24 EST 2009
Manik Surtani wrote:
>
<snip/>
>>> Guys
>>> While running some basic state transfer tests on AS 5.0, we've seen
>>> some stack traces when shutting down a server
>>>
>>> http://hudson.qa.jboss.com/hudson/view/EAP%20Perf+Fail/job/as5-random-failover-http-buddy-async-jvmkill/10/console-perf01/
>>>
>>
<snip/>
> pertinent log and stack trace
> snips below:
>
I believe the issue here is when the AjpProtocol logs the "Stopping
Coyote AJP/1.3" message you see, what it's basically doing is closing
the ServerSocket from which regular Sockets are accepted. AFAICT if
there's a it doesn't disrupt or in any way signal any of the threads
processing existing sockets that have already been accepted. Those
threads will keep reading requests off the socket as long as there are any.
I don't know if closing the ServerSocket will cause mod_xx to promptly
mark the worker as unavailable, or if it waits until 1) it needs to
create a new connection to the worker or 2) one of the existing
connections has a failure. Either way, in-flight requests are still
active after the "Stopping AJP/1.3" message is logged.
>
> [JBoss] 05:47:33,394 INFO [ServerImpl] Shutting down the server,
> blockingShutdown: false
> [JBoss] 05:47:33,397 INFO [ServerImpl] Server exit(0) called
> [JBoss] 05:47:33,398 INFO [ServerImpl] Runtime shutdown hook called,
> forceHalt: true
> [JBoss] 05:47:33,406 INFO [Http11Protocol] Pausing Coyote HTTP/1.1 on
> http-perf01%2F10.16.88.179-8080
> [JBoss] 05:47:33,407 INFO [Http11Protocol] Stopping Coyote HTTP/1.1 on
> http-perf01%2F10.16.88.179-8080
> [JBoss] 05:47:33,408 INFO [AjpProtocol] Pausing Coyote AJP/1.3 on
> ajp-perf01%2F10.16.88.179-8009
> [JBoss] 05:47:33,409 INFO [AjpProtocol] Stopping Coyote AJP/1.3 on
> ajp-perf01%2F10.16.88.179-8009
> [JBoss] 05:47:33,460 INFO [ConnectionFactory]
> org.jboss.jms.server.connectionfactory.ConnectionFactory at 1c42860 undeployed
> [JBoss] 05:47:33,484 INFO [ConnectionFactory]
> org.jboss.jms.server.connectionfactory.ConnectionFactory at 38f0e7 undeployed
> [JBoss] 05:47:33,500 INFO [ConnectionFactory]
> org.jboss.jms.server.connectionfactory.ConnectionFactory at 4e6c9d undeployed
> [JBoss] 05:47:33,530 INFO [TomcatDeployment] undeploy,
> ctxPath=/web-console, vfsUrl=management/console-mgr.sar/web-console.war
> [JBoss] 05:47:33,584 INFO [QueueService] Queue[/queue/ExpiryQueue] stopped
> [JBoss] 05:47:33,585 INFO [QueueService] Queue[/queue/DLQ] stopped
> [JBoss] 05:47:33,595 INFO [QuartzScheduler] Scheduler
> JBossEJB3QuartzScheduler_$_NON_CLUSTERED shutting down.
> [JBoss] 05:47:33,595 INFO [QuartzScheduler] Scheduler
> JBossEJB3QuartzScheduler_$_NON_CLUSTERED paused.
> [JBoss] 05:47:33,596 INFO [QuartzScheduler] Scheduler
> JBossEJB3QuartzScheduler_$_NON_CLUSTERED shutdown complete.
> [JBoss] 05:47:34,645 INFO [ConnectionFactoryBindingService] Unbound
> ConnectionManager
> 'jboss.jca:service=ConnectionFactoryBinding,name=JmsXA' from JNDI name
> 'java:JmsXA'
> [JBoss] 05:47:34,658 INFO [ServerPeer] JMS ServerPeer[0] stopped
> [JBoss] 05:47:34,726 INFO [TomcatDeployment] undeploy,
> ctxPath=/invoker, vfsUrl=httpha-invoker.sar/invoker.war
> [JBoss] 05:47:34,799 INFO [HttpReplTestPartition] Stopping partition
> HttpReplTestPartition
> [JBoss] 05:47:34,802 INFO [RPCManagerImpl] Disconnecting and closing
> the Channel
> [JBoss] 05:47:34,874 INFO [RPCManagerImpl] Stopping the RpcDispatcher
> [JBoss] 05:47:34,888 INFO [HttpReplTestPartition] Partition
> HttpReplTestPartition stopped.
> [JBoss] 05:47:34,891 INFO [HttpReplTestPartition] Partition
> HttpReplTestPartition destroyed.
> [JBoss] 05:47:34,893 INFO [TomcatDeployment] undeploy, ctxPath=/st,
> vfsUrl=SessionTest.war
> [JBoss] 05:47:34,931 INFO [RPCManagerImpl] Disconnecting and closing
> the Channel
> [JBoss] 05:47:35,002 INFO [RPCManagerImpl] Stopping the RpcDispatcher
> [JBoss] 05:47:35,451 INFO [TomcatDeployment] undeploy,
> ctxPath=/jbossws, vfsUrl=jbossws.sar/jbossws-management.war
> [JBoss] 05:47:35,454 INFO [TomcatDeployment] undeploy, ctxPath=/juddi,
> vfsUrl=juddi-service.sar/juddi.war
> [JBoss] 05:47:35,455 INFO [RegistryServlet] jUDDI Stopping: Cleaning up
> existing resources.
> [JBoss] 05:47:35,456 INFO [TomcatDeployment] undeploy, ctxPath=/,
> vfsUrl=ROOT.war
> [JBoss] 05:47:35,458 INFO [TomcatDeployment] undeploy,
> ctxPath=/jmx-console, vfsUrl=jmx-console.war
> [JBoss] 05:47:35,462 INFO [StandardService] Stopping service jboss.web
> [JBoss] 05:47:35,598 INFO [MailService] Mail service 'java:/Mail'
> removed from JNDI
> [JBoss] 05:47:35,632 INFO [SnmpAgentService] SNMP agent stopped
> [JBoss] 05:47:35,811 INFO [ConnectionFactoryBindingService] Unbound
> ConnectionManager 'jboss.jca:service=DataSourceBinding,name=DefaultDS'
> from JNDI name 'java:DefaultDS'
> [JBoss] 05:47:35,841 INFO [QuartzScheduler] Scheduler
> DefaultQuartzScheduler_$_NON_CLUSTERED shutting down.
> [JBoss] 05:47:35,841 INFO [QuartzScheduler] Scheduler
> DefaultQuartzScheduler_$_NON_CLUSTERED paused.
> [JBoss] 05:47:35,845 INFO [SimpleThreadPool] There are still 45 worker
> threads active. See javadoc runInThread(Runnable) for a possible
> explanation
> [JBoss] 05:47:35,845 INFO [QuartzScheduler] Scheduler
> DefaultQuartzScheduler_$_NON_CLUSTERED shutdown complete.
> [JBoss] 05:47:35,957 INFO [TransactionManagerService] Stopping
> transaction recovery manager
> [JBoss] 05:47:36,213 INFO [HypersonicDatabase] Database standalone
> closed clean
> [JBoss] 05:47:36,602 ERROR [CoyoteAdapter] An exception or error
> occurred in the container during the request processing
> [JBoss] java.lang.RuntimeException: DistributedCacheManager is null.
> [JBoss] at
> org.jboss.web.tomcat.service.session.ClusteredSession.establishDistributedCacheManager(ClusteredSession.java:1802)
>
> [JBoss] at
> org.jboss.web.tomcat.service.session.ClusteredSession.setManager(ClusteredSession.java:506)
>
> [JBoss] at
> org.jboss.web.tomcat.service.session.ClusteredSession.<init>(ClusteredSession.java:399)
>
> [JBoss] at
> org.jboss.web.tomcat.service.session.SessionBasedClusteredSession.<init>(SessionBasedClusteredSession.java:56)
>
> [JBoss] at
> org.jboss.web.tomcat.service.session.JBossCacheManager.createEmptyClusteredSession(JBossCacheManager.java:1410)
>
> [JBoss] at
> org.jboss.web.tomcat.service.session.JBossCacheManager.loadSession(JBossCacheManager.java:1491)
>
> [JBoss] at
> org.jboss.web.tomcat.service.session.JBossCacheManager.findSession(JBossCacheManager.java:462)
>
> [JBoss] at
> org.apache.catalina.connector.Request.doGetSession(Request.java:2322)
> [JBoss] at
> org.apache.catalina.connector.Request.getSession(Request.java:2096)
> [JBoss] at
> org.jboss.web.tomcat.service.session.JvmRouteValve.checkJvmRoute(JvmRouteValve.java:94)
>
> [JBoss] at
> org.jboss.web.tomcat.service.session.JvmRouteValve.invoke(JvmRouteValve.java:84)
>
> [JBoss] at
> org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:92)
>
> [JBoss] at
> org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)
>
> [JBoss] at
> org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)
>
> [JBoss] at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>
> [JBoss] at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>
> [JBoss] at
> org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
>
> [JBoss] at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>
> [JBoss] at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)
> [JBoss] at
> org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:436)
> [JBoss] at
> org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:384)
>
> [JBoss] at
> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
> [JBoss] at java.lang.Thread.run(Thread.java:619)
>
>
>>
>>
>>> I haven't dug into this in much detail, but it seems as though the
>>> CoyoteAdapter is still accepting HTTP requests even after the webapps
>>> and other clustering components have been stopped/removed? Is this a
>>> known bug in AS 5.0?
>>> Thoughts?
>>> Cheers
>>> Manik
>>> --
>>> Manik Surtani
>>> Lead, JBoss Cache
>>> http://www.jbosscache.org
>>> manik at jboss.org
>>> ------------------------------------------------------------------------
>>> _______________________________________________
>>> jboss-development mailing list
>>> jboss-development at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/jboss-development
>>
>> --
>> Galder Zamarreño
>> Sr. Software Maintenance Engineer
>> JBoss, a division of Red Hat
>> _______________________________________________
>> jboss-development mailing list
>> jboss-development at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/jboss-development
>
> --
> Manik Surtani
> Lead, JBoss Cache
> http://www.jbosscache.org
> manik at jboss.org
>
>
>
>
>
> _______________________________________________
> jboss-development mailing list
> jboss-development at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/jboss-development
--
Brian Stansberry
Lead, AS Clustering
JBoss, a division of Red Hat
brian.stansberry at redhat.com
More information about the jboss-development
mailing list