Dude, Manik Surtani wrote: > Guys > While running some basic state transfer tests on AS 5.0, we've seen > some stack traces when shutting down a server > http://hudson.qa.jboss.com/hudson/view/EAP%20Perf+Fail/job/as5-random-fai... I believe that's an internal link. Public only has access to http://hudson.jboss.org/hudson/ and there's no "EAP Perf+Fail" there.

> I haven't dug into this in much detail, but it seems as though the > CoyoteAdapter is still accepting HTTP requests even after the > webapps and other clustering components have been stopped/removed? > Is this a known bug in AS 5.0? > Thoughts? > Cheers > Manik > -- > Manik Surtani > Lead, JBoss Cache > http://www.jbosscache.org > manik(a)jboss.org > ------------------------------------------------------------------------ > _______________________________________________ > jboss-development mailing list > jboss-development(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-development -- Galder Zamarreño Sr. Software Maintenance Engineer JBoss, a division of Red Hat _______________________________________________ jboss-development mailing list jboss-development(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-development

Manik Surtani wrote: > <snip/> >>> Guys >>> While running some basic state transfer tests on AS 5.0, we've seen >>> some stack traces when shutting down a server >>> >>> http://hudson.qa.jboss.com/hudson/view/EAP%20Perf+Fail/job/as5-random-fai... >>> >> <snip/> > pertinent log and stack trace > snips below: > I believe the issue here is when the AjpProtocol logs the "Stopping Coyote AJP/1.3" message you see, what it's basically doing is closing the ServerSocket from which regular Sockets are accepted. AFAICT if there's a it doesn't disrupt or in any way signal any of the threads processing existing sockets that have already been accepted. Those threads will keep reading requests off the socket as long as there are any. I don't know if closing the ServerSocket will cause mod_xx to promptly mark the worker as unavailable, or if it waits until 1) it needs to create a new connection to the worker or 2) one of the existing connections has a failure. Either way, in-flight requests are still active after the "Stopping AJP/1.3" message is logged. > > [JBoss] 05:47:33,394 INFO [ServerImpl] Shutting down the server, > blockingShutdown: false > [JBoss] 05:47:33,397 INFO [ServerImpl] Server exit(0) called > [JBoss] 05:47:33,398 INFO [ServerImpl] Runtime shutdown hook called, > forceHalt: true > [JBoss] 05:47:33,406 INFO [Http11Protocol] Pausing Coyote HTTP/1.1 on > http-perf01%2F10.16.88.179-8080 > [JBoss] 05:47:33,407 INFO [Http11Protocol] Stopping Coyote HTTP/1.1 on > http-perf01%2F10.16.88.179-8080 > [JBoss] 05:47:33,408 INFO [AjpProtocol] Pausing Coyote AJP/1.3 on > ajp-perf01%2F10.16.88.179-8009 > [JBoss] 05:47:33,409 INFO [AjpProtocol] Stopping Coyote AJP/1.3 on > ajp-perf01%2F10.16.88.179-8009 > [JBoss] 05:47:33,460 INFO [ConnectionFactory] > org.jboss.jms.server.connectionfactory.ConnectionFactory@1c42860 undeployed > [JBoss] 05:47:33,484 INFO [ConnectionFactory] > org.jboss.jms.server.connectionfactory.ConnectionFactory@38f0e7 undeployed > [JBoss] 05:47:33,500 INFO [ConnectionFactory] > org.jboss.jms.server.connectionfactory.ConnectionFactory@4e6c9d undeployed > [JBoss] 05:47:33,530 INFO [TomcatDeployment] undeploy, > ctxPath=/web-console, vfsUrl=management/console-mgr.sar/web-console.war > [JBoss] 05:47:33,584 INFO [QueueService] Queue[/queue/ExpiryQueue] stopped > [JBoss] 05:47:33,585 INFO [QueueService] Queue[/queue/DLQ] stopped > [JBoss] 05:47:33,595 INFO [QuartzScheduler] Scheduler > JBossEJB3QuartzScheduler_$_NON_CLUSTERED shutting down. > [JBoss] 05:47:33,595 INFO [QuartzScheduler] Scheduler > JBossEJB3QuartzScheduler_$_NON_CLUSTERED paused. > [JBoss] 05:47:33,596 INFO [QuartzScheduler] Scheduler > JBossEJB3QuartzScheduler_$_NON_CLUSTERED shutdown complete. > [JBoss] 05:47:34,645 INFO [ConnectionFactoryBindingService] Unbound > ConnectionManager > 'jboss.jca:service=ConnectionFactoryBinding,name=JmsXA' from JNDI name > 'java:JmsXA' > [JBoss] 05:47:34,658 INFO [ServerPeer] JMS ServerPeer[0] stopped > [JBoss] 05:47:34,726 INFO [TomcatDeployment] undeploy, > ctxPath=/invoker, vfsUrl=httpha-invoker.sar/invoker.war > [JBoss] 05:47:34,799 INFO [HttpReplTestPartition] Stopping partition > HttpReplTestPartition > [JBoss] 05:47:34,802 INFO [RPCManagerImpl] Disconnecting and closing > the Channel > [JBoss] 05:47:34,874 INFO [RPCManagerImpl] Stopping the RpcDispatcher > [JBoss] 05:47:34,888 INFO [HttpReplTestPartition] Partition > HttpReplTestPartition stopped. > [JBoss] 05:47:34,891 INFO [HttpReplTestPartition] Partition > HttpReplTestPartition destroyed. > [JBoss] 05:47:34,893 INFO [TomcatDeployment] undeploy, ctxPath=/st, > vfsUrl=SessionTest.war > [JBoss] 05:47:34,931 INFO [RPCManagerImpl] Disconnecting and closing > the Channel > [JBoss] 05:47:35,002 INFO [RPCManagerImpl] Stopping the RpcDispatcher > [JBoss] 05:47:35,451 INFO [TomcatDeployment] undeploy, > ctxPath=/jbossws, vfsUrl=jbossws.sar/jbossws-management.war > [JBoss] 05:47:35,454 INFO [TomcatDeployment] undeploy, ctxPath=/juddi, > vfsUrl=juddi-service.sar/juddi.war > [JBoss] 05:47:35,455 INFO [RegistryServlet] jUDDI Stopping: Cleaning up > existing resources. > [JBoss] 05:47:35,456 INFO [TomcatDeployment] undeploy, ctxPath=/, > vfsUrl=ROOT.war > [JBoss] 05:47:35,458 INFO [TomcatDeployment] undeploy, > ctxPath=/jmx-console, vfsUrl=jmx-console.war > [JBoss] 05:47:35,462 INFO [StandardService] Stopping service jboss.web > [JBoss] 05:47:35,598 INFO [MailService] Mail service 'java:/Mail' > removed from JNDI > [JBoss] 05:47:35,632 INFO [SnmpAgentService] SNMP agent stopped > [JBoss] 05:47:35,811 INFO [ConnectionFactoryBindingService] Unbound > ConnectionManager 'jboss.jca:service=DataSourceBinding,name=DefaultDS' > from JNDI name 'java:DefaultDS' > [JBoss] 05:47:35,841 INFO [QuartzScheduler] Scheduler > DefaultQuartzScheduler_$_NON_CLUSTERED shutting down. > [JBoss] 05:47:35,841 INFO [QuartzScheduler] Scheduler > DefaultQuartzScheduler_$_NON_CLUSTERED paused. > [JBoss] 05:47:35,845 INFO [SimpleThreadPool] There are still 45 worker > threads active. See javadoc runInThread(Runnable) for a possible > explanation > [JBoss] 05:47:35,845 INFO [QuartzScheduler] Scheduler > DefaultQuartzScheduler_$_NON_CLUSTERED shutdown complete. > [JBoss] 05:47:35,957 INFO [TransactionManagerService] Stopping > transaction recovery manager > [JBoss] 05:47:36,213 INFO [HypersonicDatabase] Database standalone > closed clean > [JBoss] 05:47:36,602 ERROR [CoyoteAdapter] An exception or error > occurred in the container during the request processing > [JBoss] java.lang.RuntimeException: DistributedCacheManager is null. > [JBoss] at > org.jboss.web.tomcat.service.session.ClusteredSession.establishDistributedCacheManager(ClusteredSession.java:1802) > > [JBoss] at > org.jboss.web.tomcat.service.session.ClusteredSession.setManager(ClusteredSession.java:506) > > [JBoss] at > org.jboss.web.tomcat.service.session.ClusteredSession.<init>(ClusteredSession.java:399) > > [JBoss] at > org.jboss.web.tomcat.service.session.SessionBasedClusteredSession.<init>(SessionBasedClusteredSession.java:56) > > [JBoss] at > org.jboss.web.tomcat.service.session.JBossCacheManager.createEmptyClusteredSession(JBossCacheManager.java:1410) > > [JBoss] at > org.jboss.web.tomcat.service.session.JBossCacheManager.loadSession(JBossCacheManager.java:1491) > > [JBoss] at > org.jboss.web.tomcat.service.session.JBossCacheManager.findSession(JBossCacheManager.java:462) > > [JBoss] at > org.apache.catalina.connector.Request.doGetSession(Request.java:2322) > [JBoss] at > org.apache.catalina.connector.Request.getSession(Request.java:2096) > [JBoss] at > org.jboss.web.tomcat.service.session.JvmRouteValve.checkJvmRoute(JvmRouteValve.java:94) > > [JBoss] at > org.jboss.web.tomcat.service.session.JvmRouteValve.invoke(JvmRouteValve.java:84) > > [JBoss] at > org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:92) > > [JBoss] at > org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126) > > [JBoss] at > org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70) > > [JBoss] at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) > > [JBoss] at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) > > [JBoss] at > org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158) > > [JBoss] at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > > [JBoss] at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330) > [JBoss] at > org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:436) > [JBoss] at > org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:384) > > [JBoss] at > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447) > [JBoss] at java.lang.Thread.run(Thread.java:619) > > >> >> >>> I haven't dug into this in much detail, but it seems as though the >>> CoyoteAdapter is still accepting HTTP requests even after the webapps >>> and other clustering components have been stopped/removed? Is this a >>> known bug in AS 5.0? >>> Thoughts? >>> Cheers >>> Manik >>> -- >>> Manik Surtani >>> Lead, JBoss Cache >>> http://www.jbosscache.org >>> manik(a)jboss.org >>> ------------------------------------------------------------------------ >>> _______________________________________________ >>> jboss-development mailing list >>> jboss-development(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/jboss-development >> >> -- >> Galder Zamarreño >> Sr. Software Maintenance Engineer >> JBoss, a division of Red Hat >> _______________________________________________ >> jboss-development mailing list >> jboss-development(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jboss-development > > -- > Manik Surtani > Lead, JBoss Cache > http://www.jbosscache.org > manik(a)jboss.org > > > > > > _______________________________________________ > jboss-development mailing list > jboss-development(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-development