Re: [jboss-dev] Re: AS5 testsuite cannot complete.

The most obvious difference is the noip runs bind to localhost; the others to a 10.x address. What machine is this running on? I'd like to take a look at its networking setup. The operation that's failing sends a multicast msg out to the network and counts on the sending node receiving that message itself. If that receiving of its own message fails, we'd get the behavior we're seeing. Dimitris Andreadis wrote: > Since the JBoss-AS-5.0.x-TestSuite-sun15 suite completes (vs the > JBoss-AS-5.0.x-TestSuite-sun15-noip that is stuck with start/stop > problems) I think we should focus on exactly what's different between > the two setups. > > We've said before that we want to get rid of the 'noip' version, like > in Branch_4_2, so let's do that. > > Brian Stansberry wrote: >> 2.5.0.GA. >> >> This problem is happening as part of a clean shutdown of a server >> that's been running for quite a while. The server is started in the >> 'all' config, but its not for clustering tests, so there shouldn't be >> another server with which it clusters. Without the server log it's >> hard to say much beyond that. I'm trying to reproduce. >> >> Bela Ban wrote: >>> A failed flush could also be caused by reincarnation >>> (http://jira.jboss.com/jira/browse/JGRP-130), especially when >>> killing and restarting a server with a rather long timeout in FD. >>> JGroups 2.5.2/2.6-CR-1 fixed this by simply rejecting a member who >>> crashed, but was not removed, and wants to join the same group under >>> the same address and port. >>> >>> >>> What version of JGroups is AS 5 running with ? >>> >>> Brian Stansberry wrote: >>>> Has the workspace for JBoss-AS-5.0.x-TestSuite-sun15-noip been >>>> removed? Trying to access it through hudson.jboss.org tells me it >>>> has; trying to get it through >>>> http://dev45.qa.atl.jboss.com:8585/hudson the server never responds. >>>> >>>> Need to see the server.log to have any clue what is going on; the >>>> console logging isn't detailed enough. >>>> >>>> Vladimir, assume this one is coming your way. On the surface it >>>> looks like a FLUSH issue. >>>> >>>> Rajesh Rajasekaran wrote: >>>>> As seen from the server log the server process is not destroyed >>>>> after 45 secs in this case. >>>>> This is something to look at in the jboss-test project. >>>>> >>>>> The delay in the all server shutdown is actually caused by the >>>>> server trying to flush all the cache and failing in the process. >>>>> The attached server log shows the details. >>>>> This goes on for 2 minutes and more before which the hudson kill >>>>> kicks in and destroys the server because the build is complete >>>>> (failed) by then. >>>>> >>>>> >>>>> >>>>> Dimitris Andreadis wrote: >>>>>> We still cannot get a decent testsuite run: >>>>>> http://dev45.qa.atl.jboss.com:8585/hudson/view/JBoss%20AS/job/JBoss-AS-5.... >>>>>> >>>>>> >>>>>> At some point there is an attempt to stop the server running the >>>>>> 'all' configuration: >>>>>> >>>>>> ---------------------------------- >>>>>> tests-aspects: >>>>>> [server:stop] Shutting down server: all >>>>>> [server:stop] Shutting down server: >>>>>> /qa/tools/opt/jdk1.5.0_12/bin/java -cp >>>>>> /qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3/bin/shutdown.jar:/qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3/client/jbossall-client.jar:/qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3/client/jboss-common.jar >>>>>> org.jboss.Shutdown --server jnp://localhost:1099 --shutdown >>>>>> [server:stop] shutdownTimeout will be=45 >>>>>> [server:stop] Failed to shutdown server "all" before timeout. >>>>>> Destroying the process. >>>>>> ---------------------------------- >>>>>> >>>>>> The server fails to shutdown withing 45 seconds, and so it's >>>>>> supposedly killed. The server 'all' log shows we are stuck at >>>>>> jboss messaging shutting down (see below). >>>>>> >>>>>> Then the next server (profileservice test) cannot start because a >>>>>> previous instance is already running (see bottom)??? How can this >>>>>> be possible if it was killed??? >>>>>> >>>>>> ----------------------------------- >>>>>> 04:31:57,537 INFO [ServerImpl] LifeThread.run exits! >>>>>> 04:31:57,538 INFO [ServerImpl] Shutting down the JVM now! >>>>>> 04:31:57,539 INFO [DefaultPartition-ClusteredSSOCache] stop(): >>>>>> closing the channel >>>>>> 04:31:57,540 INFO [DefaultPartition-SFSBCache] stop(): closing >>>>>> the channel >>>>>> 04:31:57,540 INFO [ServerImpl] JBoss SHUTDOWN >>>>>> 04:31:57,544 INFO [DefaultPartition-TomcatCache] stop(): closing >>>>>> the channel >>>>>> 04:31:57,544 INFO [DefaultPartition-EntityCache] stop(): closing >>>>>> the channel >>>>>> 04:31:57,544 INFO [DefaultPartition-HAPartitionCache] stop(): >>>>>> closing the channel >>>>>> 04:31:57,561 INFO [TopicService] Topic[/topic/testTopic] stopped >>>>>> 04:31:57,561 INFO [TopicService] Topic[/topic/securedTopic] stopped >>>>>> 04:31:57,561 INFO [TopicService] Topic[/topic/testDurableTopic] >>>>>> stopped >>>>>> 04:31:57,561 INFO [QueueService] Queue[/queue/testQueue] stopped >>>>>> 04:31:57,562 INFO [QueueService] Queue[/queue/testObjectMessage] >>>>>> stopped >>>>>> 04:31:57,567 INFO [QueueService] Queue[/queue/A] stopped >>>>>> 04:31:57,567 INFO [QueueService] Queue[/queue/B] stopped >>>>>> 04:31:57,568 INFO [QueueService] Queue[/queue/C] stopped >>>>>> 04:31:57,568 INFO [QueueService] Queue[/queue/D] stopped >>>>>> 04:31:57,568 INFO [QueueService] Queue[/queue/ex] stopped >>>>>> 04:31:57,578 INFO [ConnectionFactory] >>>>>> org.jboss.jms.server.connectionfactory.ConnectionFactory@176ffeb >>>>>> undeployed >>>>>> 04:31:57,580 INFO [ConnectionFactory] >>>>>> org.jboss.jms.server.connectionfactory.ConnectionFactory@10b3611 >>>>>> undeployed >>>>>> 04:31:57,581 ERROR [ExceptionUtil] >>>>>> org.jboss.jms.server.connectionfactory.ConnectionFactory@c610af >>>>>> startService >>>>>> java.lang.IllegalStateException: Cannot find replicant to remove: >>>>>> CF_jboss.messaging.connectionfactory:service=ConnectionFactory >>>>>> at >>>>>> org.jboss.jms.server.connectionfactory.ConnectionFactoryJNDIMapper.unregisterConnectionFactory(ConnectionFactoryJNDIMapper.java:266) >>>>>> >>>>>> at >>>>>> org.jboss.jms.server.connectionfactory.ConnectionFactory.stopService(ConnectionFactory.java:187) >>>>>> >>>>>> at >>>>>> org.jboss.system.ServiceMBeanSupport.jbossInternalStop(ServiceMBeanSupport.java:328) >>>>>> >>>>>> at >>>>>> org.jboss.system.ServiceMBeanSupport.stop(ServiceMBeanSupport.java:206) >>>>>> >>>>>> at sun.reflect.GeneratedMethodAccessor882.invoke(Unknown Source) >>>>>> at >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>>> >>>>>> at java.lang.reflect.Method.invoke(Method.java:585) >>>>>> at >>>>>> org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157) >>>>>> >>>>>> at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96) >>>>>> at >>>>>> org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:138) >>>>>> >>>>>> at org.jboss.mx.server.Invocation.invoke(Invocation.java:90) >>>>>> at >>>>>> org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:140) >>>>>> >>>>>> at org.jboss.mx.server.Invocation.invoke(Invocation.java:90) >>>>>> at >>>>>> org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) >>>>>> >>>>>> at >>>>>> org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668) >>>>>> at >>>>>> org.jboss.system.microcontainer.ServiceProxy.invoke(ServiceProxy.java:184) >>>>>> >>>>>> at $Proxy4.stop(Unknown Source) >>>>>> at >>>>>> org.jboss.system.microcontainer.StartStopLifecycleAction.uninstallAction(StartStopLifecycleAction.java:56) >>>>>> >>>>>> at >>>>>> org.jboss.system.microcontainer.StartStopLifecycleAction.uninstallAction(StartStopLifecycleAction.java:37) >>>>>> >>>>>> at >>>>>> org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleUninstallAction(SimpleControllerContextAction.java:79) >>>>>> >>>>>> at >>>>>> org.jboss.dependency.plugins.action.AccessControllerContextAction.uninstall(AccessControllerContextAction.java:131) >>>>>> >>>>>> at >>>>>> org.jboss.dependency.plugins.AbstractControllerContextActions.uninstall(AbstractControllerContextActions.java:58) >>>>>> >>>>>> at >>>>>> org.jboss.dependency.plugins.AbstractControllerContext.uninstall(AbstractControllerContext.java:333) >>>>>> >>>>>> at >>>>>> org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:1323) >>>>>> >>>>>> at >>>>>> org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1009) >>>>>> >>>>>> at >>>>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:627) >>>>>> >>>>>> at >>>>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:411) >>>>>> >>>>>> at >>>>>> org.jboss.system.ServiceController.doChange(ServiceController.java:659) >>>>>> >>>>>> at >>>>>> org.jboss.system.ServiceController.stop(ServiceController.java:481) >>>>>> at >>>>>> org.jboss.system.deployers.ServiceDeployer.stop(ServiceDeployer.java:156) >>>>>> >>>>>> at >>>>>> org.jboss.system.deployers.ServiceDeployer.undeploy(ServiceDeployer.java:136) >>>>>> >>>>>> at >>>>>> org.jboss.system.deployers.ServiceDeployer.undeploy(ServiceDeployer.java:46) >>>>>> >>>>>> at >>>>>> org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.undeploy(AbstractSimpleRealDeployer.java:73) >>>>>> >>>>>> at >>>>>> org.jboss.deployers.plugins.deployers.DeployerWrapper.undeploy(DeployerWrapper.java:187) >>>>>> >>>>>> at >>>>>> org.jboss.deployers.plugins.deployers.DeployersImpl.doUninstallParentLast(DeployersImpl.java:947) >>>>>> >>>>>> at >>>>>> org.jboss.deployers.plugins.deployers.DeployersImpl.doUninstallParentLast(DeployersImpl.java:940) >>>>>> >>>>>> at >>>>>> org.jboss.deployers.plugins.deployers.DeployersImpl.uninstall(DeployersImpl.java:902) >>>>>> >>>>>> at >>>>>> org.jboss.dependency.plugins.AbstractControllerContext.uninstall(AbstractControllerContext.java:333) >>>>>> >>>>>> at >>>>>> org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:1323) >>>>>> >>>>>> at >>>>>> org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1009) >>>>>> >>>>>> at >>>>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:627) >>>>>> >>>>>> at >>>>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:411) >>>>>> >>>>>> at >>>>>> org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:420) >>>>>> >>>>>> at >>>>>> org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:354) >>>>>> >>>>>> at >>>>>> org.jboss.deployers.plugins.main.MainDeployerImpl.shutdown(MainDeployerImpl.java:382) >>>>>> >>>>>> at >>>>>> org.jboss.system.server.profileservice.ProfileServiceBootstrap.shutdown(ProfileServiceBootstrap.java:151) >>>>>> >>>>>> at >>>>>> org.jboss.bootstrap.AbstractServerImpl.shutdownServer(AbstractServerImpl.java:482) >>>>>> >>>>>> at >>>>>> org.jboss.bootstrap.AbstractServerImpl$ShutdownHook.run(AbstractServerImpl.java:778) >>>>>> >>>>>> 04:31:57,582 WARN [ConnectionFactory] Stopping failed >>>>>> jboss.messaging.connectionfactory:service=ConnectionFactory >>>>>> java.lang.IllegalStateException: Cannot find replicant to remove: >>>>>> CF_jboss.messaging.connectionfactory:service=ConnectionFactory >>>>>> at >>>>>> org.jboss.jms.server.connectionfactory.ConnectionFactoryJNDIMapper.unregisterConnectionFactory(ConnectionFactoryJNDIMapper.java:266) >>>>>> >>>>>> at >>>>>> org.jboss.jms.server.connectionfactory.ConnectionFactory.stopService(ConnectionFactory.java:187) >>>>>> >>>>>> at >>>>>> org.jboss.system.ServiceMBeanSupport.jbossInternalStop(ServiceMBeanSupport.java:328) >>>>>> >>>>>> at >>>>>> org.jboss.system.ServiceMBeanSupport.stop(ServiceMBeanSupport.java:206) >>>>>> >>>>>> at sun.reflect.GeneratedMethodAccessor882.invoke(Unknown Source) >>>>>> at >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>>> >>>>>> at java.lang.reflect.Method.invoke(Method.java:585) >>>>>> at >>>>>> org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157) >>>>>> >>>>>> at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96) >>>>>> at >>>>>> org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:138) >>>>>> >>>>>> at org.jboss.mx.server.Invocation.invoke(Invocation.java:90) >>>>>> at >>>>>> org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:140) >>>>>> >>>>>> at org.jboss.mx.server.Invocation.invoke(Invocation.java:90) >>>>>> at >>>>>> org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) >>>>>> >>>>>> at >>>>>> org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668) >>>>>> at >>>>>> org.jboss.system.microcontainer.ServiceProxy.invoke(ServiceProxy.java:184) >>>>>> >>>>>> at $Proxy4.stop(Unknown Source) >>>>>> at >>>>>> org.jboss.system.microcontainer.StartStopLifecycleAction.uninstallAction(StartStopLifecycleAction.java:56) >>>>>> >>>>>> at >>>>>> org.jboss.system.microcontainer.StartStopLifecycleAction.uninstallAction(StartStopLifecycleAction.java:37) >>>>>> >>>>>> at >>>>>> org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleUninstallAction(SimpleControllerContextAction.java:79) >>>>>> >>>>>> at >>>>>> org.jboss.dependency.plugins.action.AccessControllerContextAction.uninstall(AccessControllerContextAction.java:131) >>>>>> >>>>>> at >>>>>> org.jboss.dependency.plugins.AbstractControllerContextActions.uninstall(AbstractControllerContextActions.java:58) >>>>>> >>>>>> at >>>>>> org.jboss.dependency.plugins.AbstractControllerContext.uninstall(AbstractControllerContext.java:333) >>>>>> >>>>>> at >>>>>> org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:1323) >>>>>> >>>>>> at >>>>>> org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1009) >>>>>> >>>>>> at >>>>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:627) >>>>>> >>>>>> at >>>>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:411) >>>>>> >>>>>> at >>>>>> org.jboss.system.ServiceController.doChange(ServiceController.java:659) >>>>>> >>>>>> at >>>>>> org.jboss.system.ServiceController.stop(ServiceController.java:481) >>>>>> at >>>>>> org.jboss.system.deployers.ServiceDeployer.stop(ServiceDeployer.java:156) >>>>>> >>>>>> at >>>>>> org.jboss.system.deployers.ServiceDeployer.undeploy(ServiceDeployer.java:136) >>>>>> >>>>>> at >>>>>> org.jboss.system.deployers.ServiceDeployer.undeploy(ServiceDeployer.java:46) >>>>>> >>>>>> at >>>>>> org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.undeploy(AbstractSimpleRealDeployer.java:73) >>>>>> >>>>>> at >>>>>> org.jboss.deployers.plugins.deployers.DeployerWrapper.undeploy(DeployerWrapper.java:187) >>>>>> >>>>>> at >>>>>> org.jboss.deployers.plugins.deployers.DeployersImpl.doUninstallParentLast(DeployersImpl.java:947) >>>>>> >>>>>> at >>>>>> org.jboss.deployers.plugins.deployers.DeployersImpl.doUninstallParentLast(DeployersImpl.java:940) >>>>>> >>>>>> at >>>>>> org.jboss.deployers.plugins.deployers.DeployersImpl.uninstall(DeployersImpl.java:902) >>>>>> >>>>>> at >>>>>> org.jboss.dependency.plugins.AbstractControllerContext.uninstall(AbstractControllerContext.java:333) >>>>>> >>>>>> at >>>>>> org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:1323) >>>>>> >>>>>> at >>>>>> org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1009) >>>>>> >>>>>> at >>>>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:627) >>>>>> >>>>>> at >>>>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:411) >>>>>> >>>>>> at >>>>>> org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:420) >>>>>> >>>>>> at >>>>>> org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:354) >>>>>> >>>>>> at >>>>>> org.jboss.deployers.plugins.main.MainDeployerImpl.shutdown(MainDeployerImpl.java:382) >>>>>> >>>>>> at >>>>>> org.jboss.system.server.profileservice.ProfileServiceBootstrap.shutdown(ProfileServiceBootstrap.java:151) >>>>>> >>>>>> at >>>>>> org.jboss.bootstrap.AbstractServerImpl.shutdownServer(AbstractServerImpl.java:482) >>>>>> >>>>>> at >>>>>> org.jboss.bootstrap.AbstractServerImpl$ShutdownHook.run(AbstractServerImpl.java:778) >>>>>> >>>>>> 04:31:57,585 INFO [QueueService] Queue[/queue/DLQ] stopped >>>>>> 04:31:57,586 INFO [QueueService] Queue[/queue/ExpiryQueue] stopped >>>>>> 04:32:20,561 WARN [JChannelFactory] Flush failed at >>>>>> 127.0.0.1:58939DefaultPartition-ClusteredSSOCache >>>>>> >>>>>> ----------------------------------- >>>>>> >>>>>> >>>>>> >>>>>> ----------------------------------- >>>>>> tests-profileservice: >>>>>> [echo] creating profileservice config, >>>>>> jboss.dist=/qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3 >>>>>> >>>>>> [copy] Copying 378 files to >>>>>> /qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3/server/profileservice >>>>>> >>>>>> [copy] Copied 101 empty directories to 1 empty directory >>>>>> under >>>>>> /qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3/server/profileservice >>>>>> >>>>>> [copy] Copying 1 file to >>>>>> /qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3/server/profileservice/conf >>>>>> >>>>>> [copy] Copying 1 file to >>>>>> /qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3/server/profileservice/deployers >>>>>> >>>>>> >>>>>> BUILD FAILED >>>>>> /qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/testsuite/build.xml:834: >>>>>> The following error occurred while executing this line: >>>>>> /qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/testsuite/build.xml:891: >>>>>> Error starting server "profileservice": Found a process already >>>>>> listening on:http://localhost:8080 or jnp://localhost:1099 >>>>>> >>>>>> Total time: 105 minutes 42 seconds >>>>>> ----------------------------------- >>>>> >>>>> >>>>> ------------------------------------------------------------------------ >>>>> >>>>> >>>>> _______________________________________________ >>>>> jboss-development mailing list >>>>> jboss-development(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/jboss-development >>>> >>> >> > _______________________________________________ > jboss-development mailing list > jboss-development(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-development