Re: [jboss-dev] Re: AS5 testsuite cannot complete.

Hi Brian, I have not seen this problem before so I do not know what's going on. :-( Usually, when a build is in progress, depending on which state it is, workspace may not be available. Most of projects in hudson are configured to clean up workspace every time a new build starts. So if you try to access workspace during clean up time, it wont be accessible. But this is not the case here. Regards, Prabhat Brian Stansberry wrote: > Same issue there as with http://dev45.qa.atl.jboss.com:8585/hudson : > the server never responds when you request > http://hudson.qa.jboss.com/hudson/job/JBoss-AS-5.0.x-TestSuite-sun15-noip... > > > Rajesh Rajasekaran wrote: >> The workspace is not accessible from the public hudson instance. >> Please use the internal hudson instance. >> http://hudson.qa.jboss.com/hudson/view/JBoss%20AS/ >> >> >> Brian Stansberry wrote: >>> Has the workspace for JBoss-AS-5.0.x-TestSuite-sun15-noip been >>> removed? Trying to access it through hudson.jboss.org tells me it >>> has; trying to get it through >>> http://dev45.qa.atl.jboss.com:8585/hudson the server never responds. >>> >>> Need to see the server.log to have any clue what is going on; the >>> console logging isn't detailed enough. >>> >>> Vladimir, assume this one is coming your way. On the surface it >>> looks like a FLUSH issue. >>> >>> Rajesh Rajasekaran wrote: >>>> As seen from the server log the server process is not destroyed >>>> after 45 secs in this case. >>>> This is something to look at in the jboss-test project. >>>> >>>> The delay in the all server shutdown is actually caused by the >>>> server trying to flush all the cache and failing in the process. >>>> The attached server log shows the details. >>>> This goes on for 2 minutes and more before which the hudson kill >>>> kicks in and destroys the server because the build is complete >>>> (failed) by then. >>>> >>>> >>>> >>>> Dimitris Andreadis wrote: >>>>> We still cannot get a decent testsuite run: >>>>> http://dev45.qa.atl.jboss.com:8585/hudson/view/JBoss%20AS/job/JBoss-AS-5.... >>>>> >>>>> >>>>> At some point there is an attempt to stop the server running the >>>>> 'all' configuration: >>>>> >>>>> ---------------------------------- >>>>> tests-aspects: >>>>> [server:stop] Shutting down server: all >>>>> [server:stop] Shutting down server: >>>>> /qa/tools/opt/jdk1.5.0_12/bin/java -cp >>>>> /qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3/bin/shutdown.jar:/qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3/client/jbossall-client.jar:/qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3/client/jboss-common.jar >>>>> org.jboss.Shutdown --server jnp://localhost:1099 --shutdown >>>>> [server:stop] shutdownTimeout will be=45 >>>>> [server:stop] Failed to shutdown server "all" before timeout. >>>>> Destroying the process. >>>>> ---------------------------------- >>>>> >>>>> The server fails to shutdown withing 45 seconds, and so it's >>>>> supposedly killed. The server 'all' log shows we are stuck at >>>>> jboss messaging shutting down (see below). >>>>> >>>>> Then the next server (profileservice test) cannot start because a >>>>> previous instance is already running (see bottom)??? How can this >>>>> be possible if it was killed??? >>>>> >>>>> ----------------------------------- >>>>> 04:31:57,537 INFO [ServerImpl] LifeThread.run exits! >>>>> 04:31:57,538 INFO [ServerImpl] Shutting down the JVM now! >>>>> 04:31:57,539 INFO [DefaultPartition-ClusteredSSOCache] stop(): >>>>> closing the channel >>>>> 04:31:57,540 INFO [DefaultPartition-SFSBCache] stop(): closing >>>>> the channel >>>>> 04:31:57,540 INFO [ServerImpl] JBoss SHUTDOWN >>>>> 04:31:57,544 INFO [DefaultPartition-TomcatCache] stop(): closing >>>>> the channel >>>>> 04:31:57,544 INFO [DefaultPartition-EntityCache] stop(): closing >>>>> the channel >>>>> 04:31:57,544 INFO [DefaultPartition-HAPartitionCache] stop(): >>>>> closing the channel >>>>> 04:31:57,561 INFO [TopicService] Topic[/topic/testTopic] stopped >>>>> 04:31:57,561 INFO [TopicService] Topic[/topic/securedTopic] stopped >>>>> 04:31:57,561 INFO [TopicService] Topic[/topic/testDurableTopic] >>>>> stopped >>>>> 04:31:57,561 INFO [QueueService] Queue[/queue/testQueue] stopped >>>>> 04:31:57,562 INFO [QueueService] Queue[/queue/testObjectMessage] >>>>> stopped >>>>> 04:31:57,567 INFO [QueueService] Queue[/queue/A] stopped >>>>> 04:31:57,567 INFO [QueueService] Queue[/queue/B] stopped >>>>> 04:31:57,568 INFO [QueueService] Queue[/queue/C] stopped >>>>> 04:31:57,568 INFO [QueueService] Queue[/queue/D] stopped >>>>> 04:31:57,568 INFO [QueueService] Queue[/queue/ex] stopped >>>>> 04:31:57,578 INFO [ConnectionFactory] >>>>> org.jboss.jms.server.connectionfactory.ConnectionFactory@176ffeb >>>>> undeployed >>>>> 04:31:57,580 INFO [ConnectionFactory] >>>>> org.jboss.jms.server.connectionfactory.ConnectionFactory@10b3611 >>>>> undeployed >>>>> 04:31:57,581 ERROR [ExceptionUtil] >>>>> org.jboss.jms.server.connectionfactory.ConnectionFactory@c610af >>>>> startService >>>>> java.lang.IllegalStateException: Cannot find replicant to remove: >>>>> CF_jboss.messaging.connectionfactory:service=ConnectionFactory >>>>> at >>>>> org.jboss.jms.server.connectionfactory.ConnectionFactoryJNDIMapper.unregisterConnectionFactory(ConnectionFactoryJNDIMapper.java:266) >>>>> >>>>> at >>>>> org.jboss.jms.server.connectionfactory.ConnectionFactory.stopService(ConnectionFactory.java:187) >>>>> >>>>> at >>>>> org.jboss.system.ServiceMBeanSupport.jbossInternalStop(ServiceMBeanSupport.java:328) >>>>> >>>>> at >>>>> org.jboss.system.ServiceMBeanSupport.stop(ServiceMBeanSupport.java:206) >>>>> >>>>> at sun.reflect.GeneratedMethodAccessor882.invoke(Unknown Source) >>>>> at >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>> >>>>> at java.lang.reflect.Method.invoke(Method.java:585) >>>>> at >>>>> org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157) >>>>> >>>>> at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96) >>>>> at >>>>> org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:138) >>>>> >>>>> at org.jboss.mx.server.Invocation.invoke(Invocation.java:90) >>>>> at >>>>> org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:140) >>>>> >>>>> at org.jboss.mx.server.Invocation.invoke(Invocation.java:90) >>>>> at >>>>> org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) >>>>> >>>>> at >>>>> org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668) >>>>> at >>>>> org.jboss.system.microcontainer.ServiceProxy.invoke(ServiceProxy.java:184) >>>>> >>>>> at $Proxy4.stop(Unknown Source) >>>>> at >>>>> org.jboss.system.microcontainer.StartStopLifecycleAction.uninstallAction(StartStopLifecycleAction.java:56) >>>>> >>>>> at >>>>> org.jboss.system.microcontainer.StartStopLifecycleAction.uninstallAction(StartStopLifecycleAction.java:37) >>>>> >>>>> at >>>>> org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleUninstallAction(SimpleControllerContextAction.java:79) >>>>> >>>>> at >>>>> org.jboss.dependency.plugins.action.AccessControllerContextAction.uninstall(AccessControllerContextAction.java:131) >>>>> >>>>> at >>>>> org.jboss.dependency.plugins.AbstractControllerContextActions.uninstall(AbstractControllerContextActions.java:58) >>>>> >>>>> at >>>>> org.jboss.dependency.plugins.AbstractControllerContext.uninstall(AbstractControllerContext.java:333) >>>>> >>>>> at >>>>> org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:1323) >>>>> >>>>> at >>>>> org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1009) >>>>> >>>>> at >>>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:627) >>>>> >>>>> at >>>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:411) >>>>> >>>>> at >>>>> org.jboss.system.ServiceController.doChange(ServiceController.java:659) >>>>> >>>>> at >>>>> org.jboss.system.ServiceController.stop(ServiceController.java:481) >>>>> at >>>>> org.jboss.system.deployers.ServiceDeployer.stop(ServiceDeployer.java:156) >>>>> >>>>> at >>>>> org.jboss.system.deployers.ServiceDeployer.undeploy(ServiceDeployer.java:136) >>>>> >>>>> at >>>>> org.jboss.system.deployers.ServiceDeployer.undeploy(ServiceDeployer.java:46) >>>>> >>>>> at >>>>> org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.undeploy(AbstractSimpleRealDeployer.java:73) >>>>> >>>>> at >>>>> org.jboss.deployers.plugins.deployers.DeployerWrapper.undeploy(DeployerWrapper.java:187) >>>>> >>>>> at >>>>> org.jboss.deployers.plugins.deployers.DeployersImpl.doUninstallParentLast(DeployersImpl.java:947) >>>>> >>>>> at >>>>> org.jboss.deployers.plugins.deployers.DeployersImpl.doUninstallParentLast(DeployersImpl.java:940) >>>>> >>>>> at >>>>> org.jboss.deployers.plugins.deployers.DeployersImpl.uninstall(DeployersImpl.java:902) >>>>> >>>>> at >>>>> org.jboss.dependency.plugins.AbstractControllerContext.uninstall(AbstractControllerContext.java:333) >>>>> >>>>> at >>>>> org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:1323) >>>>> >>>>> at >>>>> org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1009) >>>>> >>>>> at >>>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:627) >>>>> >>>>> at >>>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:411) >>>>> >>>>> at >>>>> org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:420) >>>>> >>>>> at >>>>> org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:354) >>>>> >>>>> at >>>>> org.jboss.deployers.plugins.main.MainDeployerImpl.shutdown(MainDeployerImpl.java:382) >>>>> >>>>> at >>>>> org.jboss.system.server.profileservice.ProfileServiceBootstrap.shutdown(ProfileServiceBootstrap.java:151) >>>>> >>>>> at >>>>> org.jboss.bootstrap.AbstractServerImpl.shutdownServer(AbstractServerImpl.java:482) >>>>> >>>>> at >>>>> org.jboss.bootstrap.AbstractServerImpl$ShutdownHook.run(AbstractServerImpl.java:778) >>>>> >>>>> 04:31:57,582 WARN [ConnectionFactory] Stopping failed >>>>> jboss.messaging.connectionfactory:service=ConnectionFactory >>>>> java.lang.IllegalStateException: Cannot find replicant to remove: >>>>> CF_jboss.messaging.connectionfactory:service=ConnectionFactory >>>>> at >>>>> org.jboss.jms.server.connectionfactory.ConnectionFactoryJNDIMapper.unregisterConnectionFactory(ConnectionFactoryJNDIMapper.java:266) >>>>> >>>>> at >>>>> org.jboss.jms.server.connectionfactory.ConnectionFactory.stopService(ConnectionFactory.java:187) >>>>> >>>>> at >>>>> org.jboss.system.ServiceMBeanSupport.jbossInternalStop(ServiceMBeanSupport.java:328) >>>>> >>>>> at >>>>> org.jboss.system.ServiceMBeanSupport.stop(ServiceMBeanSupport.java:206) >>>>> >>>>> at sun.reflect.GeneratedMethodAccessor882.invoke(Unknown Source) >>>>> at >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>> >>>>> at java.lang.reflect.Method.invoke(Method.java:585) >>>>> at >>>>> org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157) >>>>> >>>>> at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96) >>>>> at >>>>> org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:138) >>>>> >>>>> at org.jboss.mx.server.Invocation.invoke(Invocation.java:90) >>>>> at >>>>> org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:140) >>>>> >>>>> at org.jboss.mx.server.Invocation.invoke(Invocation.java:90) >>>>> at >>>>> org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) >>>>> >>>>> at >>>>> org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668) >>>>> at >>>>> org.jboss.system.microcontainer.ServiceProxy.invoke(ServiceProxy.java:184) >>>>> >>>>> at $Proxy4.stop(Unknown Source) >>>>> at >>>>> org.jboss.system.microcontainer.StartStopLifecycleAction.uninstallAction(StartStopLifecycleAction.java:56) >>>>> >>>>> at >>>>> org.jboss.system.microcontainer.StartStopLifecycleAction.uninstallAction(StartStopLifecycleAction.java:37) >>>>> >>>>> at >>>>> org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleUninstallAction(SimpleControllerContextAction.java:79) >>>>> >>>>> at >>>>> org.jboss.dependency.plugins.action.AccessControllerContextAction.uninstall(AccessControllerContextAction.java:131) >>>>> >>>>> at >>>>> org.jboss.dependency.plugins.AbstractControllerContextActions.uninstall(AbstractControllerContextActions.java:58) >>>>> >>>>> at >>>>> org.jboss.dependency.plugins.AbstractControllerContext.uninstall(AbstractControllerContext.java:333) >>>>> >>>>> at >>>>> org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:1323) >>>>> >>>>> at >>>>> org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1009) >>>>> >>>>> at >>>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:627) >>>>> >>>>> at >>>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:411) >>>>> >>>>> at >>>>> org.jboss.system.ServiceController.doChange(ServiceController.java:659) >>>>> >>>>> at >>>>> org.jboss.system.ServiceController.stop(ServiceController.java:481) >>>>> at >>>>> org.jboss.system.deployers.ServiceDeployer.stop(ServiceDeployer.java:156) >>>>> >>>>> at >>>>> org.jboss.system.deployers.ServiceDeployer.undeploy(ServiceDeployer.java:136) >>>>> >>>>> at >>>>> org.jboss.system.deployers.ServiceDeployer.undeploy(ServiceDeployer.java:46) >>>>> >>>>> at >>>>> org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.undeploy(AbstractSimpleRealDeployer.java:73) >>>>> >>>>> at >>>>> org.jboss.deployers.plugins.deployers.DeployerWrapper.undeploy(DeployerWrapper.java:187) >>>>> >>>>> at >>>>> org.jboss.deployers.plugins.deployers.DeployersImpl.doUninstallParentLast(DeployersImpl.java:947) >>>>> >>>>> at >>>>> org.jboss.deployers.plugins.deployers.DeployersImpl.doUninstallParentLast(DeployersImpl.java:940) >>>>> >>>>> at >>>>> org.jboss.deployers.plugins.deployers.DeployersImpl.uninstall(DeployersImpl.java:902) >>>>> >>>>> at >>>>> org.jboss.dependency.plugins.AbstractControllerContext.uninstall(AbstractControllerContext.java:333) >>>>> >>>>> at >>>>> org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:1323) >>>>> >>>>> at >>>>> org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1009) >>>>> >>>>> at >>>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:627) >>>>> >>>>> at >>>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:411) >>>>> >>>>> at >>>>> org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:420) >>>>> >>>>> at >>>>> org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:354) >>>>> >>>>> at >>>>> org.jboss.deployers.plugins.main.MainDeployerImpl.shutdown(MainDeployerImpl.java:382) >>>>> >>>>> at >>>>> org.jboss.system.server.profileservice.ProfileServiceBootstrap.shutdown(ProfileServiceBootstrap.java:151) >>>>> >>>>> at >>>>> org.jboss.bootstrap.AbstractServerImpl.shutdownServer(AbstractServerImpl.java:482) >>>>> >>>>> at >>>>> org.jboss.bootstrap.AbstractServerImpl$ShutdownHook.run(AbstractServerImpl.java:778) >>>>> >>>>> 04:31:57,585 INFO [QueueService] Queue[/queue/DLQ] stopped >>>>> 04:31:57,586 INFO [QueueService] Queue[/queue/ExpiryQueue] stopped >>>>> 04:32:20,561 WARN [JChannelFactory] Flush failed at >>>>> 127.0.0.1:58939DefaultPartition-ClusteredSSOCache >>>>> >>>>> ----------------------------------- >>>>> >>>>> >>>>> >>>>> ----------------------------------- >>>>> tests-profileservice: >>>>> [echo] creating profileservice config, >>>>> jboss.dist=/qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3 >>>>> >>>>> [copy] Copying 378 files to >>>>> /qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3/server/profileservice >>>>> >>>>> [copy] Copied 101 empty directories to 1 empty directory >>>>> under >>>>> /qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3/server/profileservice >>>>> >>>>> [copy] Copying 1 file to >>>>> /qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3/server/profileservice/conf >>>>> >>>>> [copy] Copying 1 file to >>>>> /qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3/server/profileservice/deployers >>>>> >>>>> >>>>> BUILD FAILED >>>>> /qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/testsuite/build.xml:834: >>>>> The following error occurred while executing this line: >>>>> /qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/testsuite/build.xml:891: >>>>> Error starting server "profileservice": Found a process already >>>>> listening on:http://localhost:8080 or jnp://localhost:1099 >>>>> >>>>> Total time: 105 minutes 42 seconds >>>>> ----------------------------------- >>>> >>>> >>>> ------------------------------------------------------------------------ >>>> >>>> >>>> _______________________________________________ >>>> jboss-development mailing list >>>> jboss-development(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/jboss-development >>> >> >> _______________________________________________ >> jboss-development mailing list >> jboss-development(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jboss-development > _______________________________________________ jboss-development mailing list jboss-development(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-development