[jboss-dev] Re: AS5 testsuite cannot complete.
Dimitris Andreadis
dandread at redhat.com
Thu Nov 1 07:17:19 EDT 2007
Since the JBoss-AS-5.0.x-TestSuite-sun15 suite completes (vs the
JBoss-AS-5.0.x-TestSuite-sun15-noip that is stuck with start/stop problems) I think we
should focus on exactly what's different between the two setups.
We've said before that we want to get rid of the 'noip' version, like in Branch_4_2, so
let's do that.
Brian Stansberry wrote:
> 2.5.0.GA.
>
> This problem is happening as part of a clean shutdown of a server that's
> been running for quite a while. The server is started in the 'all'
> config, but its not for clustering tests, so there shouldn't be another
> server with which it clusters. Without the server log it's hard to say
> much beyond that. I'm trying to reproduce.
>
> Bela Ban wrote:
>> A failed flush could also be caused by reincarnation
>> (http://jira.jboss.com/jira/browse/JGRP-130), especially when killing
>> and restarting a server with a rather long timeout in FD. JGroups
>> 2.5.2/2.6-CR-1 fixed this by simply rejecting a member who crashed,
>> but was not removed, and wants to join the same group under the same
>> address and port.
>>
>>
>> What version of JGroups is AS 5 running with ?
>>
>> Brian Stansberry wrote:
>>> Has the workspace for JBoss-AS-5.0.x-TestSuite-sun15-noip been
>>> removed? Trying to access it through hudson.jboss.org tells me it
>>> has; trying to get it through
>>> http://dev45.qa.atl.jboss.com:8585/hudson the server never responds.
>>>
>>> Need to see the server.log to have any clue what is going on; the
>>> console logging isn't detailed enough.
>>>
>>> Vladimir, assume this one is coming your way. On the surface it looks
>>> like a FLUSH issue.
>>>
>>> Rajesh Rajasekaran wrote:
>>>> As seen from the server log the server process is not destroyed
>>>> after 45 secs in this case.
>>>> This is something to look at in the jboss-test project.
>>>>
>>>> The delay in the all server shutdown is actually caused by the
>>>> server trying to flush all the cache and failing in the process.
>>>> The attached server log shows the details.
>>>> This goes on for 2 minutes and more before which the hudson kill
>>>> kicks in and destroys the server because the build is complete
>>>> (failed) by then.
>>>>
>>>>
>>>>
>>>> Dimitris Andreadis wrote:
>>>>> We still cannot get a decent testsuite run:
>>>>> http://dev45.qa.atl.jboss.com:8585/hudson/view/JBoss%20AS/job/JBoss-AS-5.0.x-TestSuite-sun15-noip/115/console
>>>>>
>>>>>
>>>>> At some point there is an attempt to stop the server running the
>>>>> 'all' configuration:
>>>>>
>>>>> ----------------------------------
>>>>> tests-aspects:
>>>>> [server:stop] Shutting down server: all
>>>>> [server:stop] Shutting down server:
>>>>> /qa/tools/opt/jdk1.5.0_12/bin/java -cp
>>>>> /qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3/bin/shutdown.jar:/qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3/client/jbossall-client.jar:/qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3/client/jboss-common.jar
>>>>> org.jboss.Shutdown --server jnp://localhost:1099 --shutdown
>>>>> [server:stop] shutdownTimeout will be=45
>>>>> [server:stop] Failed to shutdown server "all" before timeout.
>>>>> Destroying the process.
>>>>> ----------------------------------
>>>>>
>>>>> The server fails to shutdown withing 45 seconds, and so it's
>>>>> supposedly killed. The server 'all' log shows we are stuck at jboss
>>>>> messaging shutting down (see below).
>>>>>
>>>>> Then the next server (profileservice test) cannot start because a
>>>>> previous instance is already running (see bottom)??? How can this
>>>>> be possible if it was killed???
>>>>>
>>>>> -----------------------------------
>>>>> 04:31:57,537 INFO [ServerImpl] LifeThread.run exits!
>>>>> 04:31:57,538 INFO [ServerImpl] Shutting down the JVM now!
>>>>> 04:31:57,539 INFO [DefaultPartition-ClusteredSSOCache] stop():
>>>>> closing the channel
>>>>> 04:31:57,540 INFO [DefaultPartition-SFSBCache] stop(): closing the
>>>>> channel
>>>>> 04:31:57,540 INFO [ServerImpl] JBoss SHUTDOWN
>>>>> 04:31:57,544 INFO [DefaultPartition-TomcatCache] stop(): closing
>>>>> the channel
>>>>> 04:31:57,544 INFO [DefaultPartition-EntityCache] stop(): closing
>>>>> the channel
>>>>> 04:31:57,544 INFO [DefaultPartition-HAPartitionCache] stop():
>>>>> closing the channel
>>>>> 04:31:57,561 INFO [TopicService] Topic[/topic/testTopic] stopped
>>>>> 04:31:57,561 INFO [TopicService] Topic[/topic/securedTopic] stopped
>>>>> 04:31:57,561 INFO [TopicService] Topic[/topic/testDurableTopic]
>>>>> stopped
>>>>> 04:31:57,561 INFO [QueueService] Queue[/queue/testQueue] stopped
>>>>> 04:31:57,562 INFO [QueueService] Queue[/queue/testObjectMessage]
>>>>> stopped
>>>>> 04:31:57,567 INFO [QueueService] Queue[/queue/A] stopped
>>>>> 04:31:57,567 INFO [QueueService] Queue[/queue/B] stopped
>>>>> 04:31:57,568 INFO [QueueService] Queue[/queue/C] stopped
>>>>> 04:31:57,568 INFO [QueueService] Queue[/queue/D] stopped
>>>>> 04:31:57,568 INFO [QueueService] Queue[/queue/ex] stopped
>>>>> 04:31:57,578 INFO [ConnectionFactory]
>>>>> org.jboss.jms.server.connectionfactory.ConnectionFactory at 176ffeb
>>>>> undeployed
>>>>> 04:31:57,580 INFO [ConnectionFactory]
>>>>> org.jboss.jms.server.connectionfactory.ConnectionFactory at 10b3611
>>>>> undeployed
>>>>> 04:31:57,581 ERROR [ExceptionUtil]
>>>>> org.jboss.jms.server.connectionfactory.ConnectionFactory at c610af
>>>>> startService
>>>>> java.lang.IllegalStateException: Cannot find replicant to remove:
>>>>> CF_jboss.messaging.connectionfactory:service=ConnectionFactory
>>>>> at
>>>>> org.jboss.jms.server.connectionfactory.ConnectionFactoryJNDIMapper.unregisterConnectionFactory(ConnectionFactoryJNDIMapper.java:266)
>>>>>
>>>>> at
>>>>> org.jboss.jms.server.connectionfactory.ConnectionFactory.stopService(ConnectionFactory.java:187)
>>>>>
>>>>> at
>>>>> org.jboss.system.ServiceMBeanSupport.jbossInternalStop(ServiceMBeanSupport.java:328)
>>>>>
>>>>> at
>>>>> org.jboss.system.ServiceMBeanSupport.stop(ServiceMBeanSupport.java:206)
>>>>>
>>>>> at sun.reflect.GeneratedMethodAccessor882.invoke(Unknown Source)
>>>>> at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>
>>>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>>>> at
>>>>> org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)
>>>>>
>>>>> at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)
>>>>> at
>>>>> org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:138)
>>>>>
>>>>> at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)
>>>>> at
>>>>> org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:140)
>>>>>
>>>>> at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)
>>>>> at
>>>>> org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
>>>>>
>>>>> at
>>>>> org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
>>>>> at
>>>>> org.jboss.system.microcontainer.ServiceProxy.invoke(ServiceProxy.java:184)
>>>>>
>>>>> at $Proxy4.stop(Unknown Source)
>>>>> at
>>>>> org.jboss.system.microcontainer.StartStopLifecycleAction.uninstallAction(StartStopLifecycleAction.java:56)
>>>>>
>>>>> at
>>>>> org.jboss.system.microcontainer.StartStopLifecycleAction.uninstallAction(StartStopLifecycleAction.java:37)
>>>>>
>>>>> at
>>>>> org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleUninstallAction(SimpleControllerContextAction.java:79)
>>>>>
>>>>> at
>>>>> org.jboss.dependency.plugins.action.AccessControllerContextAction.uninstall(AccessControllerContextAction.java:131)
>>>>>
>>>>> at
>>>>> org.jboss.dependency.plugins.AbstractControllerContextActions.uninstall(AbstractControllerContextActions.java:58)
>>>>>
>>>>> at
>>>>> org.jboss.dependency.plugins.AbstractControllerContext.uninstall(AbstractControllerContext.java:333)
>>>>>
>>>>> at
>>>>> org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:1323)
>>>>>
>>>>> at
>>>>> org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1009)
>>>>>
>>>>> at
>>>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:627)
>>>>>
>>>>> at
>>>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:411)
>>>>>
>>>>> at
>>>>> org.jboss.system.ServiceController.doChange(ServiceController.java:659)
>>>>>
>>>>> at
>>>>> org.jboss.system.ServiceController.stop(ServiceController.java:481)
>>>>> at
>>>>> org.jboss.system.deployers.ServiceDeployer.stop(ServiceDeployer.java:156)
>>>>>
>>>>> at
>>>>> org.jboss.system.deployers.ServiceDeployer.undeploy(ServiceDeployer.java:136)
>>>>>
>>>>> at
>>>>> org.jboss.system.deployers.ServiceDeployer.undeploy(ServiceDeployer.java:46)
>>>>>
>>>>> at
>>>>> org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.undeploy(AbstractSimpleRealDeployer.java:73)
>>>>>
>>>>> at
>>>>> org.jboss.deployers.plugins.deployers.DeployerWrapper.undeploy(DeployerWrapper.java:187)
>>>>>
>>>>> at
>>>>> org.jboss.deployers.plugins.deployers.DeployersImpl.doUninstallParentLast(DeployersImpl.java:947)
>>>>>
>>>>> at
>>>>> org.jboss.deployers.plugins.deployers.DeployersImpl.doUninstallParentLast(DeployersImpl.java:940)
>>>>>
>>>>> at
>>>>> org.jboss.deployers.plugins.deployers.DeployersImpl.uninstall(DeployersImpl.java:902)
>>>>>
>>>>> at
>>>>> org.jboss.dependency.plugins.AbstractControllerContext.uninstall(AbstractControllerContext.java:333)
>>>>>
>>>>> at
>>>>> org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:1323)
>>>>>
>>>>> at
>>>>> org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1009)
>>>>>
>>>>> at
>>>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:627)
>>>>>
>>>>> at
>>>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:411)
>>>>>
>>>>> at
>>>>> org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:420)
>>>>>
>>>>> at
>>>>> org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:354)
>>>>>
>>>>> at
>>>>> org.jboss.deployers.plugins.main.MainDeployerImpl.shutdown(MainDeployerImpl.java:382)
>>>>>
>>>>> at
>>>>> org.jboss.system.server.profileservice.ProfileServiceBootstrap.shutdown(ProfileServiceBootstrap.java:151)
>>>>>
>>>>> at
>>>>> org.jboss.bootstrap.AbstractServerImpl.shutdownServer(AbstractServerImpl.java:482)
>>>>>
>>>>> at
>>>>> org.jboss.bootstrap.AbstractServerImpl$ShutdownHook.run(AbstractServerImpl.java:778)
>>>>>
>>>>> 04:31:57,582 WARN [ConnectionFactory] Stopping failed
>>>>> jboss.messaging.connectionfactory:service=ConnectionFactory
>>>>> java.lang.IllegalStateException: Cannot find replicant to remove:
>>>>> CF_jboss.messaging.connectionfactory:service=ConnectionFactory
>>>>> at
>>>>> org.jboss.jms.server.connectionfactory.ConnectionFactoryJNDIMapper.unregisterConnectionFactory(ConnectionFactoryJNDIMapper.java:266)
>>>>>
>>>>> at
>>>>> org.jboss.jms.server.connectionfactory.ConnectionFactory.stopService(ConnectionFactory.java:187)
>>>>>
>>>>> at
>>>>> org.jboss.system.ServiceMBeanSupport.jbossInternalStop(ServiceMBeanSupport.java:328)
>>>>>
>>>>> at
>>>>> org.jboss.system.ServiceMBeanSupport.stop(ServiceMBeanSupport.java:206)
>>>>>
>>>>> at sun.reflect.GeneratedMethodAccessor882.invoke(Unknown Source)
>>>>> at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>
>>>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>>>> at
>>>>> org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)
>>>>>
>>>>> at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)
>>>>> at
>>>>> org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:138)
>>>>>
>>>>> at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)
>>>>> at
>>>>> org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:140)
>>>>>
>>>>> at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)
>>>>> at
>>>>> org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
>>>>>
>>>>> at
>>>>> org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
>>>>> at
>>>>> org.jboss.system.microcontainer.ServiceProxy.invoke(ServiceProxy.java:184)
>>>>>
>>>>> at $Proxy4.stop(Unknown Source)
>>>>> at
>>>>> org.jboss.system.microcontainer.StartStopLifecycleAction.uninstallAction(StartStopLifecycleAction.java:56)
>>>>>
>>>>> at
>>>>> org.jboss.system.microcontainer.StartStopLifecycleAction.uninstallAction(StartStopLifecycleAction.java:37)
>>>>>
>>>>> at
>>>>> org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleUninstallAction(SimpleControllerContextAction.java:79)
>>>>>
>>>>> at
>>>>> org.jboss.dependency.plugins.action.AccessControllerContextAction.uninstall(AccessControllerContextAction.java:131)
>>>>>
>>>>> at
>>>>> org.jboss.dependency.plugins.AbstractControllerContextActions.uninstall(AbstractControllerContextActions.java:58)
>>>>>
>>>>> at
>>>>> org.jboss.dependency.plugins.AbstractControllerContext.uninstall(AbstractControllerContext.java:333)
>>>>>
>>>>> at
>>>>> org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:1323)
>>>>>
>>>>> at
>>>>> org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1009)
>>>>>
>>>>> at
>>>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:627)
>>>>>
>>>>> at
>>>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:411)
>>>>>
>>>>> at
>>>>> org.jboss.system.ServiceController.doChange(ServiceController.java:659)
>>>>>
>>>>> at
>>>>> org.jboss.system.ServiceController.stop(ServiceController.java:481)
>>>>> at
>>>>> org.jboss.system.deployers.ServiceDeployer.stop(ServiceDeployer.java:156)
>>>>>
>>>>> at
>>>>> org.jboss.system.deployers.ServiceDeployer.undeploy(ServiceDeployer.java:136)
>>>>>
>>>>> at
>>>>> org.jboss.system.deployers.ServiceDeployer.undeploy(ServiceDeployer.java:46)
>>>>>
>>>>> at
>>>>> org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.undeploy(AbstractSimpleRealDeployer.java:73)
>>>>>
>>>>> at
>>>>> org.jboss.deployers.plugins.deployers.DeployerWrapper.undeploy(DeployerWrapper.java:187)
>>>>>
>>>>> at
>>>>> org.jboss.deployers.plugins.deployers.DeployersImpl.doUninstallParentLast(DeployersImpl.java:947)
>>>>>
>>>>> at
>>>>> org.jboss.deployers.plugins.deployers.DeployersImpl.doUninstallParentLast(DeployersImpl.java:940)
>>>>>
>>>>> at
>>>>> org.jboss.deployers.plugins.deployers.DeployersImpl.uninstall(DeployersImpl.java:902)
>>>>>
>>>>> at
>>>>> org.jboss.dependency.plugins.AbstractControllerContext.uninstall(AbstractControllerContext.java:333)
>>>>>
>>>>> at
>>>>> org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:1323)
>>>>>
>>>>> at
>>>>> org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1009)
>>>>>
>>>>> at
>>>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:627)
>>>>>
>>>>> at
>>>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:411)
>>>>>
>>>>> at
>>>>> org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:420)
>>>>>
>>>>> at
>>>>> org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:354)
>>>>>
>>>>> at
>>>>> org.jboss.deployers.plugins.main.MainDeployerImpl.shutdown(MainDeployerImpl.java:382)
>>>>>
>>>>> at
>>>>> org.jboss.system.server.profileservice.ProfileServiceBootstrap.shutdown(ProfileServiceBootstrap.java:151)
>>>>>
>>>>> at
>>>>> org.jboss.bootstrap.AbstractServerImpl.shutdownServer(AbstractServerImpl.java:482)
>>>>>
>>>>> at
>>>>> org.jboss.bootstrap.AbstractServerImpl$ShutdownHook.run(AbstractServerImpl.java:778)
>>>>>
>>>>> 04:31:57,585 INFO [QueueService] Queue[/queue/DLQ] stopped
>>>>> 04:31:57,586 INFO [QueueService] Queue[/queue/ExpiryQueue] stopped
>>>>> 04:32:20,561 WARN [JChannelFactory] Flush failed at
>>>>> 127.0.0.1:58939DefaultPartition-ClusteredSSOCache
>>>>>
>>>>> -----------------------------------
>>>>>
>>>>>
>>>>>
>>>>> -----------------------------------
>>>>> tests-profileservice:
>>>>> [echo] creating profileservice config,
>>>>> jboss.dist=/qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3
>>>>>
>>>>> [copy] Copying 378 files to
>>>>> /qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3/server/profileservice
>>>>>
>>>>> [copy] Copied 101 empty directories to 1 empty directory under
>>>>> /qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3/server/profileservice
>>>>>
>>>>> [copy] Copying 1 file to
>>>>> /qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3/server/profileservice/conf
>>>>>
>>>>> [copy] Copying 1 file to
>>>>> /qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3/server/profileservice/deployers
>>>>>
>>>>>
>>>>> BUILD FAILED
>>>>> /qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/testsuite/build.xml:834:
>>>>> The following error occurred while executing this line:
>>>>> /qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/testsuite/build.xml:891:
>>>>> Error starting server "profileservice": Found a process already
>>>>> listening on:http://localhost:8080 or jnp://localhost:1099
>>>>>
>>>>> Total time: 105 minutes 42 seconds
>>>>> -----------------------------------
>>>>
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>>
>>>> _______________________________________________
>>>> jboss-development mailing list
>>>> jboss-development at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/jboss-development
>>>
>>
>
More information about the jboss-development
mailing list