[jboss-dev] Re: AS5 testsuite cannot complete.
Brian Stansberry
brian.stansberry at redhat.com
Wed Oct 31 12:28:08 EDT 2007
2.5.0.GA.
This problem is happening as part of a clean shutdown of a server that's
been running for quite a while. The server is started in the 'all'
config, but its not for clustering tests, so there shouldn't be another
server with which it clusters. Without the server log it's hard to say
much beyond that. I'm trying to reproduce.
Bela Ban wrote:
> A failed flush could also be caused by reincarnation
> (http://jira.jboss.com/jira/browse/JGRP-130), especially when killing
> and restarting a server with a rather long timeout in FD. JGroups
> 2.5.2/2.6-CR-1 fixed this by simply rejecting a member who crashed, but
> was not removed, and wants to join the same group under the same address
> and port.
>
>
> What version of JGroups is AS 5 running with ?
>
> Brian Stansberry wrote:
>> Has the workspace for JBoss-AS-5.0.x-TestSuite-sun15-noip been
>> removed? Trying to access it through hudson.jboss.org tells me it
>> has; trying to get it through
>> http://dev45.qa.atl.jboss.com:8585/hudson the server never responds.
>>
>> Need to see the server.log to have any clue what is going on; the
>> console logging isn't detailed enough.
>>
>> Vladimir, assume this one is coming your way. On the surface it looks
>> like a FLUSH issue.
>>
>> Rajesh Rajasekaran wrote:
>>> As seen from the server log the server process is not destroyed after
>>> 45 secs in this case.
>>> This is something to look at in the jboss-test project.
>>>
>>> The delay in the all server shutdown is actually caused by the server
>>> trying to flush all the cache and failing in the process.
>>> The attached server log shows the details.
>>> This goes on for 2 minutes and more before which the hudson kill
>>> kicks in and destroys the server because the build is complete
>>> (failed) by then.
>>>
>>>
>>>
>>> Dimitris Andreadis wrote:
>>>> We still cannot get a decent testsuite run:
>>>> http://dev45.qa.atl.jboss.com:8585/hudson/view/JBoss%20AS/job/JBoss-AS-5.0.x-TestSuite-sun15-noip/115/console
>>>>
>>>>
>>>> At some point there is an attempt to stop the server running the
>>>> 'all' configuration:
>>>>
>>>> ----------------------------------
>>>> tests-aspects:
>>>> [server:stop] Shutting down server: all
>>>> [server:stop] Shutting down server:
>>>> /qa/tools/opt/jdk1.5.0_12/bin/java -cp
>>>> /qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3/bin/shutdown.jar:/qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3/client/jbossall-client.jar:/qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3/client/jboss-common.jar
>>>> org.jboss.Shutdown --server jnp://localhost:1099 --shutdown
>>>> [server:stop] shutdownTimeout will be=45
>>>> [server:stop] Failed to shutdown server "all" before timeout.
>>>> Destroying the process.
>>>> ----------------------------------
>>>>
>>>> The server fails to shutdown withing 45 seconds, and so it's
>>>> supposedly killed. The server 'all' log shows we are stuck at jboss
>>>> messaging shutting down (see below).
>>>>
>>>> Then the next server (profileservice test) cannot start because a
>>>> previous instance is already running (see bottom)??? How can this be
>>>> possible if it was killed???
>>>>
>>>> -----------------------------------
>>>> 04:31:57,537 INFO [ServerImpl] LifeThread.run exits!
>>>> 04:31:57,538 INFO [ServerImpl] Shutting down the JVM now!
>>>> 04:31:57,539 INFO [DefaultPartition-ClusteredSSOCache] stop():
>>>> closing the channel
>>>> 04:31:57,540 INFO [DefaultPartition-SFSBCache] stop(): closing the
>>>> channel
>>>> 04:31:57,540 INFO [ServerImpl] JBoss SHUTDOWN
>>>> 04:31:57,544 INFO [DefaultPartition-TomcatCache] stop(): closing
>>>> the channel
>>>> 04:31:57,544 INFO [DefaultPartition-EntityCache] stop(): closing
>>>> the channel
>>>> 04:31:57,544 INFO [DefaultPartition-HAPartitionCache] stop():
>>>> closing the channel
>>>> 04:31:57,561 INFO [TopicService] Topic[/topic/testTopic] stopped
>>>> 04:31:57,561 INFO [TopicService] Topic[/topic/securedTopic] stopped
>>>> 04:31:57,561 INFO [TopicService] Topic[/topic/testDurableTopic]
>>>> stopped
>>>> 04:31:57,561 INFO [QueueService] Queue[/queue/testQueue] stopped
>>>> 04:31:57,562 INFO [QueueService] Queue[/queue/testObjectMessage]
>>>> stopped
>>>> 04:31:57,567 INFO [QueueService] Queue[/queue/A] stopped
>>>> 04:31:57,567 INFO [QueueService] Queue[/queue/B] stopped
>>>> 04:31:57,568 INFO [QueueService] Queue[/queue/C] stopped
>>>> 04:31:57,568 INFO [QueueService] Queue[/queue/D] stopped
>>>> 04:31:57,568 INFO [QueueService] Queue[/queue/ex] stopped
>>>> 04:31:57,578 INFO [ConnectionFactory]
>>>> org.jboss.jms.server.connectionfactory.ConnectionFactory at 176ffeb
>>>> undeployed
>>>> 04:31:57,580 INFO [ConnectionFactory]
>>>> org.jboss.jms.server.connectionfactory.ConnectionFactory at 10b3611
>>>> undeployed
>>>> 04:31:57,581 ERROR [ExceptionUtil]
>>>> org.jboss.jms.server.connectionfactory.ConnectionFactory at c610af
>>>> startService
>>>> java.lang.IllegalStateException: Cannot find replicant to remove:
>>>> CF_jboss.messaging.connectionfactory:service=ConnectionFactory
>>>> at
>>>> org.jboss.jms.server.connectionfactory.ConnectionFactoryJNDIMapper.unregisterConnectionFactory(ConnectionFactoryJNDIMapper.java:266)
>>>>
>>>> at
>>>> org.jboss.jms.server.connectionfactory.ConnectionFactory.stopService(ConnectionFactory.java:187)
>>>>
>>>> at
>>>> org.jboss.system.ServiceMBeanSupport.jbossInternalStop(ServiceMBeanSupport.java:328)
>>>>
>>>> at
>>>> org.jboss.system.ServiceMBeanSupport.stop(ServiceMBeanSupport.java:206)
>>>> at sun.reflect.GeneratedMethodAccessor882.invoke(Unknown Source)
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>
>>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>>> at
>>>> org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)
>>>>
>>>> at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)
>>>> at
>>>> org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:138)
>>>>
>>>> at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)
>>>> at
>>>> org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:140)
>>>>
>>>> at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)
>>>> at
>>>> org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
>>>>
>>>> at
>>>> org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
>>>> at
>>>> org.jboss.system.microcontainer.ServiceProxy.invoke(ServiceProxy.java:184)
>>>>
>>>> at $Proxy4.stop(Unknown Source)
>>>> at
>>>> org.jboss.system.microcontainer.StartStopLifecycleAction.uninstallAction(StartStopLifecycleAction.java:56)
>>>>
>>>> at
>>>> org.jboss.system.microcontainer.StartStopLifecycleAction.uninstallAction(StartStopLifecycleAction.java:37)
>>>>
>>>> at
>>>> org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleUninstallAction(SimpleControllerContextAction.java:79)
>>>>
>>>> at
>>>> org.jboss.dependency.plugins.action.AccessControllerContextAction.uninstall(AccessControllerContextAction.java:131)
>>>>
>>>> at
>>>> org.jboss.dependency.plugins.AbstractControllerContextActions.uninstall(AbstractControllerContextActions.java:58)
>>>>
>>>> at
>>>> org.jboss.dependency.plugins.AbstractControllerContext.uninstall(AbstractControllerContext.java:333)
>>>>
>>>> at
>>>> org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:1323)
>>>>
>>>> at
>>>> org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1009)
>>>>
>>>> at
>>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:627)
>>>>
>>>> at
>>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:411)
>>>>
>>>> at
>>>> org.jboss.system.ServiceController.doChange(ServiceController.java:659)
>>>> at
>>>> org.jboss.system.ServiceController.stop(ServiceController.java:481)
>>>> at
>>>> org.jboss.system.deployers.ServiceDeployer.stop(ServiceDeployer.java:156)
>>>>
>>>> at
>>>> org.jboss.system.deployers.ServiceDeployer.undeploy(ServiceDeployer.java:136)
>>>>
>>>> at
>>>> org.jboss.system.deployers.ServiceDeployer.undeploy(ServiceDeployer.java:46)
>>>>
>>>> at
>>>> org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.undeploy(AbstractSimpleRealDeployer.java:73)
>>>>
>>>> at
>>>> org.jboss.deployers.plugins.deployers.DeployerWrapper.undeploy(DeployerWrapper.java:187)
>>>>
>>>> at
>>>> org.jboss.deployers.plugins.deployers.DeployersImpl.doUninstallParentLast(DeployersImpl.java:947)
>>>>
>>>> at
>>>> org.jboss.deployers.plugins.deployers.DeployersImpl.doUninstallParentLast(DeployersImpl.java:940)
>>>>
>>>> at
>>>> org.jboss.deployers.plugins.deployers.DeployersImpl.uninstall(DeployersImpl.java:902)
>>>>
>>>> at
>>>> org.jboss.dependency.plugins.AbstractControllerContext.uninstall(AbstractControllerContext.java:333)
>>>>
>>>> at
>>>> org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:1323)
>>>>
>>>> at
>>>> org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1009)
>>>>
>>>> at
>>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:627)
>>>>
>>>> at
>>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:411)
>>>>
>>>> at
>>>> org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:420)
>>>>
>>>> at
>>>> org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:354)
>>>>
>>>> at
>>>> org.jboss.deployers.plugins.main.MainDeployerImpl.shutdown(MainDeployerImpl.java:382)
>>>>
>>>> at
>>>> org.jboss.system.server.profileservice.ProfileServiceBootstrap.shutdown(ProfileServiceBootstrap.java:151)
>>>>
>>>> at
>>>> org.jboss.bootstrap.AbstractServerImpl.shutdownServer(AbstractServerImpl.java:482)
>>>>
>>>> at
>>>> org.jboss.bootstrap.AbstractServerImpl$ShutdownHook.run(AbstractServerImpl.java:778)
>>>>
>>>> 04:31:57,582 WARN [ConnectionFactory] Stopping failed
>>>> jboss.messaging.connectionfactory:service=ConnectionFactory
>>>> java.lang.IllegalStateException: Cannot find replicant to remove:
>>>> CF_jboss.messaging.connectionfactory:service=ConnectionFactory
>>>> at
>>>> org.jboss.jms.server.connectionfactory.ConnectionFactoryJNDIMapper.unregisterConnectionFactory(ConnectionFactoryJNDIMapper.java:266)
>>>>
>>>> at
>>>> org.jboss.jms.server.connectionfactory.ConnectionFactory.stopService(ConnectionFactory.java:187)
>>>>
>>>> at
>>>> org.jboss.system.ServiceMBeanSupport.jbossInternalStop(ServiceMBeanSupport.java:328)
>>>>
>>>> at
>>>> org.jboss.system.ServiceMBeanSupport.stop(ServiceMBeanSupport.java:206)
>>>> at sun.reflect.GeneratedMethodAccessor882.invoke(Unknown Source)
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>
>>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>>> at
>>>> org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)
>>>>
>>>> at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)
>>>> at
>>>> org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:138)
>>>>
>>>> at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)
>>>> at
>>>> org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:140)
>>>>
>>>> at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)
>>>> at
>>>> org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
>>>>
>>>> at
>>>> org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
>>>> at
>>>> org.jboss.system.microcontainer.ServiceProxy.invoke(ServiceProxy.java:184)
>>>>
>>>> at $Proxy4.stop(Unknown Source)
>>>> at
>>>> org.jboss.system.microcontainer.StartStopLifecycleAction.uninstallAction(StartStopLifecycleAction.java:56)
>>>>
>>>> at
>>>> org.jboss.system.microcontainer.StartStopLifecycleAction.uninstallAction(StartStopLifecycleAction.java:37)
>>>>
>>>> at
>>>> org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleUninstallAction(SimpleControllerContextAction.java:79)
>>>>
>>>> at
>>>> org.jboss.dependency.plugins.action.AccessControllerContextAction.uninstall(AccessControllerContextAction.java:131)
>>>>
>>>> at
>>>> org.jboss.dependency.plugins.AbstractControllerContextActions.uninstall(AbstractControllerContextActions.java:58)
>>>>
>>>> at
>>>> org.jboss.dependency.plugins.AbstractControllerContext.uninstall(AbstractControllerContext.java:333)
>>>>
>>>> at
>>>> org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:1323)
>>>>
>>>> at
>>>> org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1009)
>>>>
>>>> at
>>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:627)
>>>>
>>>> at
>>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:411)
>>>>
>>>> at
>>>> org.jboss.system.ServiceController.doChange(ServiceController.java:659)
>>>> at
>>>> org.jboss.system.ServiceController.stop(ServiceController.java:481)
>>>> at
>>>> org.jboss.system.deployers.ServiceDeployer.stop(ServiceDeployer.java:156)
>>>>
>>>> at
>>>> org.jboss.system.deployers.ServiceDeployer.undeploy(ServiceDeployer.java:136)
>>>>
>>>> at
>>>> org.jboss.system.deployers.ServiceDeployer.undeploy(ServiceDeployer.java:46)
>>>>
>>>> at
>>>> org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.undeploy(AbstractSimpleRealDeployer.java:73)
>>>>
>>>> at
>>>> org.jboss.deployers.plugins.deployers.DeployerWrapper.undeploy(DeployerWrapper.java:187)
>>>>
>>>> at
>>>> org.jboss.deployers.plugins.deployers.DeployersImpl.doUninstallParentLast(DeployersImpl.java:947)
>>>>
>>>> at
>>>> org.jboss.deployers.plugins.deployers.DeployersImpl.doUninstallParentLast(DeployersImpl.java:940)
>>>>
>>>> at
>>>> org.jboss.deployers.plugins.deployers.DeployersImpl.uninstall(DeployersImpl.java:902)
>>>>
>>>> at
>>>> org.jboss.dependency.plugins.AbstractControllerContext.uninstall(AbstractControllerContext.java:333)
>>>>
>>>> at
>>>> org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:1323)
>>>>
>>>> at
>>>> org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1009)
>>>>
>>>> at
>>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:627)
>>>>
>>>> at
>>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:411)
>>>>
>>>> at
>>>> org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:420)
>>>>
>>>> at
>>>> org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:354)
>>>>
>>>> at
>>>> org.jboss.deployers.plugins.main.MainDeployerImpl.shutdown(MainDeployerImpl.java:382)
>>>>
>>>> at
>>>> org.jboss.system.server.profileservice.ProfileServiceBootstrap.shutdown(ProfileServiceBootstrap.java:151)
>>>>
>>>> at
>>>> org.jboss.bootstrap.AbstractServerImpl.shutdownServer(AbstractServerImpl.java:482)
>>>>
>>>> at
>>>> org.jboss.bootstrap.AbstractServerImpl$ShutdownHook.run(AbstractServerImpl.java:778)
>>>>
>>>> 04:31:57,585 INFO [QueueService] Queue[/queue/DLQ] stopped
>>>> 04:31:57,586 INFO [QueueService] Queue[/queue/ExpiryQueue] stopped
>>>> 04:32:20,561 WARN [JChannelFactory] Flush failed at
>>>> 127.0.0.1:58939DefaultPartition-ClusteredSSOCache
>>>>
>>>> -----------------------------------
>>>>
>>>>
>>>>
>>>> -----------------------------------
>>>> tests-profileservice:
>>>> [echo] creating profileservice config,
>>>> jboss.dist=/qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3
>>>>
>>>> [copy] Copying 378 files to
>>>> /qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3/server/profileservice
>>>>
>>>> [copy] Copied 101 empty directories to 1 empty directory under
>>>> /qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3/server/profileservice
>>>>
>>>> [copy] Copying 1 file to
>>>> /qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3/server/profileservice/conf
>>>>
>>>> [copy] Copying 1 file to
>>>> /qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3/server/profileservice/deployers
>>>>
>>>>
>>>> BUILD FAILED
>>>> /qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/testsuite/build.xml:834:
>>>> The following error occurred while executing this line:
>>>> /qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/testsuite/build.xml:891:
>>>> Error starting server "profileservice": Found a process already
>>>> listening on:http://localhost:8080 or jnp://localhost:1099
>>>>
>>>> Total time: 105 minutes 42 seconds
>>>> -----------------------------------
>>>
>>>
>>> ------------------------------------------------------------------------
>>>
>>> _______________________________________________
>>> jboss-development mailing list
>>> jboss-development at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/jboss-development
>>
>
--
Brian Stansberry
Lead, AS Clustering
JBoss, a division of Red Hat
brian.stansberry at redhat.com
More information about the jboss-development
mailing list