[jboss-dev] Re: AS5 testsuite cannot complete.

Bela Ban bela at jboss.com
Wed Oct 31 11:05:13 EDT 2007 

A failed flush could also be caused by reincarnation 
(http://jira.jboss.com/jira/browse/JGRP-130), especially when killing 
and restarting a server with a rather long timeout in FD. JGroups 
2.5.2/2.6-CR-1 fixed this by simply rejecting a member who crashed, but 
was not removed, and wants to join the same group under the same address 
and port.


What version of JGroups is AS 5 running with ?

Brian Stansberry wrote:
> Has the workspace for JBoss-AS-5.0.x-TestSuite-sun15-noip been 
> removed?  Trying to access it through hudson.jboss.org tells me it 
> has; trying to get it through 
> http://dev45.qa.atl.jboss.com:8585/hudson the server never responds.
>
> Need to see the server.log to have any clue what is going on; the 
> console logging isn't detailed enough.
>
> Vladimir, assume this one is coming your way. On the surface it looks 
> like a FLUSH issue.
>
> Rajesh Rajasekaran wrote:
>> As seen from the server log the server process is not destroyed after 
>> 45 secs in this case.
>> This is something to look at in the jboss-test project.
>>
>> The delay in the all server shutdown is actually caused by the server 
>> trying to flush all the cache and failing in the process.
>> The attached server log shows the details.
>> This goes on for 2 minutes and more before which the hudson kill 
>> kicks in and destroys the server because the build is complete 
>> (failed) by then.
>>
>>
>>
>> Dimitris Andreadis wrote:
>>> We still cannot get a decent testsuite run:
>>> http://dev45.qa.atl.jboss.com:8585/hudson/view/JBoss%20AS/job/JBoss-AS-5.0.x-TestSuite-sun15-noip/115/console 
>>>
>>>
>>> At some point there is an attempt to stop the server running the 
>>> 'all' configuration:
>>>
>>> ----------------------------------
>>> tests-aspects:
>>> [server:stop] Shutting down server: all
>>> [server:stop] Shutting down server: 
>>> /qa/tools/opt/jdk1.5.0_12/bin/java -cp 
>>> /qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3/bin/shutdown.jar:/qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3/client/jbossall-client.jar:/qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3/client/jboss-common.jar 
>>> org.jboss.Shutdown --server jnp://localhost:1099 --shutdown
>>> [server:stop] shutdownTimeout will be=45
>>> [server:stop] Failed to shutdown server "all" before timeout. 
>>> Destroying the process.
>>> ----------------------------------
>>>
>>> The server fails to shutdown withing 45 seconds, and so it's 
>>> supposedly killed. The server 'all' log shows we are stuck at jboss 
>>> messaging shutting down (see below).
>>>
>>> Then the next server (profileservice test) cannot start because a 
>>> previous instance is already running (see bottom)??? How can this be 
>>> possible if it was killed???
>>>
>>> -----------------------------------
>>> 04:31:57,537 INFO  [ServerImpl] LifeThread.run exits!
>>> 04:31:57,538 INFO  [ServerImpl] Shutting down the JVM now!
>>> 04:31:57,539 INFO  [DefaultPartition-ClusteredSSOCache] stop(): 
>>> closing the channel
>>> 04:31:57,540 INFO  [DefaultPartition-SFSBCache] stop(): closing the 
>>> channel
>>> 04:31:57,540 INFO  [ServerImpl] JBoss SHUTDOWN
>>> 04:31:57,544 INFO  [DefaultPartition-TomcatCache] stop(): closing 
>>> the channel
>>> 04:31:57,544 INFO  [DefaultPartition-EntityCache] stop(): closing 
>>> the channel
>>> 04:31:57,544 INFO  [DefaultPartition-HAPartitionCache] stop(): 
>>> closing the channel
>>> 04:31:57,561 INFO  [TopicService] Topic[/topic/testTopic] stopped
>>> 04:31:57,561 INFO  [TopicService] Topic[/topic/securedTopic] stopped
>>> 04:31:57,561 INFO  [TopicService] Topic[/topic/testDurableTopic] 
>>> stopped
>>> 04:31:57,561 INFO  [QueueService] Queue[/queue/testQueue] stopped
>>> 04:31:57,562 INFO  [QueueService] Queue[/queue/testObjectMessage] 
>>> stopped
>>> 04:31:57,567 INFO  [QueueService] Queue[/queue/A] stopped
>>> 04:31:57,567 INFO  [QueueService] Queue[/queue/B] stopped
>>> 04:31:57,568 INFO  [QueueService] Queue[/queue/C] stopped
>>> 04:31:57,568 INFO  [QueueService] Queue[/queue/D] stopped
>>> 04:31:57,568 INFO  [QueueService] Queue[/queue/ex] stopped
>>> 04:31:57,578 INFO  [ConnectionFactory] 
>>> org.jboss.jms.server.connectionfactory.ConnectionFactory at 176ffeb 
>>> undeployed
>>> 04:31:57,580 INFO  [ConnectionFactory] 
>>> org.jboss.jms.server.connectionfactory.ConnectionFactory at 10b3611 
>>> undeployed
>>> 04:31:57,581 ERROR [ExceptionUtil] 
>>> org.jboss.jms.server.connectionfactory.ConnectionFactory at c610af 
>>> startService
>>> java.lang.IllegalStateException: Cannot find replicant to remove: 
>>> CF_jboss.messaging.connectionfactory:service=ConnectionFactory
>>>     at 
>>> org.jboss.jms.server.connectionfactory.ConnectionFactoryJNDIMapper.unregisterConnectionFactory(ConnectionFactoryJNDIMapper.java:266) 
>>>
>>>     at 
>>> org.jboss.jms.server.connectionfactory.ConnectionFactory.stopService(ConnectionFactory.java:187) 
>>>
>>>     at 
>>> org.jboss.system.ServiceMBeanSupport.jbossInternalStop(ServiceMBeanSupport.java:328) 
>>>
>>>     at 
>>> org.jboss.system.ServiceMBeanSupport.stop(ServiceMBeanSupport.java:206)
>>>     at sun.reflect.GeneratedMethodAccessor882.invoke(Unknown Source)
>>>     at 
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
>>>
>>>     at java.lang.reflect.Method.invoke(Method.java:585)
>>>     at 
>>> org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157) 
>>>
>>>     at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)
>>>     at 
>>> org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:138) 
>>>
>>>     at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)
>>>     at 
>>> org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:140) 
>>>
>>>     at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)
>>>     at 
>>> org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) 
>>>
>>>     at 
>>> org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
>>>     at 
>>> org.jboss.system.microcontainer.ServiceProxy.invoke(ServiceProxy.java:184) 
>>>
>>>     at $Proxy4.stop(Unknown Source)
>>>     at 
>>> org.jboss.system.microcontainer.StartStopLifecycleAction.uninstallAction(StartStopLifecycleAction.java:56) 
>>>
>>>     at 
>>> org.jboss.system.microcontainer.StartStopLifecycleAction.uninstallAction(StartStopLifecycleAction.java:37) 
>>>
>>>     at 
>>> org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleUninstallAction(SimpleControllerContextAction.java:79) 
>>>
>>>     at 
>>> org.jboss.dependency.plugins.action.AccessControllerContextAction.uninstall(AccessControllerContextAction.java:131) 
>>>
>>>     at 
>>> org.jboss.dependency.plugins.AbstractControllerContextActions.uninstall(AbstractControllerContextActions.java:58) 
>>>
>>>     at 
>>> org.jboss.dependency.plugins.AbstractControllerContext.uninstall(AbstractControllerContext.java:333) 
>>>
>>>     at 
>>> org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:1323) 
>>>
>>>     at 
>>> org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1009) 
>>>
>>>     at 
>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:627) 
>>>
>>>     at 
>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:411) 
>>>
>>>     at 
>>> org.jboss.system.ServiceController.doChange(ServiceController.java:659)
>>>     at 
>>> org.jboss.system.ServiceController.stop(ServiceController.java:481)
>>>     at 
>>> org.jboss.system.deployers.ServiceDeployer.stop(ServiceDeployer.java:156) 
>>>
>>>     at 
>>> org.jboss.system.deployers.ServiceDeployer.undeploy(ServiceDeployer.java:136) 
>>>
>>>     at 
>>> org.jboss.system.deployers.ServiceDeployer.undeploy(ServiceDeployer.java:46) 
>>>
>>>     at 
>>> org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.undeploy(AbstractSimpleRealDeployer.java:73) 
>>>
>>>     at 
>>> org.jboss.deployers.plugins.deployers.DeployerWrapper.undeploy(DeployerWrapper.java:187) 
>>>
>>>     at 
>>> org.jboss.deployers.plugins.deployers.DeployersImpl.doUninstallParentLast(DeployersImpl.java:947) 
>>>
>>>     at 
>>> org.jboss.deployers.plugins.deployers.DeployersImpl.doUninstallParentLast(DeployersImpl.java:940) 
>>>
>>>     at 
>>> org.jboss.deployers.plugins.deployers.DeployersImpl.uninstall(DeployersImpl.java:902) 
>>>
>>>     at 
>>> org.jboss.dependency.plugins.AbstractControllerContext.uninstall(AbstractControllerContext.java:333) 
>>>
>>>     at 
>>> org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:1323) 
>>>
>>>     at 
>>> org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1009) 
>>>
>>>     at 
>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:627) 
>>>
>>>     at 
>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:411) 
>>>
>>>     at 
>>> org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:420) 
>>>
>>>     at 
>>> org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:354) 
>>>
>>>     at 
>>> org.jboss.deployers.plugins.main.MainDeployerImpl.shutdown(MainDeployerImpl.java:382) 
>>>
>>>     at 
>>> org.jboss.system.server.profileservice.ProfileServiceBootstrap.shutdown(ProfileServiceBootstrap.java:151) 
>>>
>>>     at 
>>> org.jboss.bootstrap.AbstractServerImpl.shutdownServer(AbstractServerImpl.java:482) 
>>>
>>>     at 
>>> org.jboss.bootstrap.AbstractServerImpl$ShutdownHook.run(AbstractServerImpl.java:778) 
>>>
>>> 04:31:57,582 WARN  [ConnectionFactory] Stopping failed 
>>> jboss.messaging.connectionfactory:service=ConnectionFactory
>>> java.lang.IllegalStateException: Cannot find replicant to remove: 
>>> CF_jboss.messaging.connectionfactory:service=ConnectionFactory
>>>     at 
>>> org.jboss.jms.server.connectionfactory.ConnectionFactoryJNDIMapper.unregisterConnectionFactory(ConnectionFactoryJNDIMapper.java:266) 
>>>
>>>     at 
>>> org.jboss.jms.server.connectionfactory.ConnectionFactory.stopService(ConnectionFactory.java:187) 
>>>
>>>     at 
>>> org.jboss.system.ServiceMBeanSupport.jbossInternalStop(ServiceMBeanSupport.java:328) 
>>>
>>>     at 
>>> org.jboss.system.ServiceMBeanSupport.stop(ServiceMBeanSupport.java:206)
>>>     at sun.reflect.GeneratedMethodAccessor882.invoke(Unknown Source)
>>>     at 
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
>>>
>>>     at java.lang.reflect.Method.invoke(Method.java:585)
>>>     at 
>>> org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157) 
>>>
>>>     at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)
>>>     at 
>>> org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:138) 
>>>
>>>     at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)
>>>     at 
>>> org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:140) 
>>>
>>>     at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)
>>>     at 
>>> org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) 
>>>
>>>     at 
>>> org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
>>>     at 
>>> org.jboss.system.microcontainer.ServiceProxy.invoke(ServiceProxy.java:184) 
>>>
>>>     at $Proxy4.stop(Unknown Source)
>>>     at 
>>> org.jboss.system.microcontainer.StartStopLifecycleAction.uninstallAction(StartStopLifecycleAction.java:56) 
>>>
>>>     at 
>>> org.jboss.system.microcontainer.StartStopLifecycleAction.uninstallAction(StartStopLifecycleAction.java:37) 
>>>
>>>     at 
>>> org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleUninstallAction(SimpleControllerContextAction.java:79) 
>>>
>>>     at 
>>> org.jboss.dependency.plugins.action.AccessControllerContextAction.uninstall(AccessControllerContextAction.java:131) 
>>>
>>>     at 
>>> org.jboss.dependency.plugins.AbstractControllerContextActions.uninstall(AbstractControllerContextActions.java:58) 
>>>
>>>     at 
>>> org.jboss.dependency.plugins.AbstractControllerContext.uninstall(AbstractControllerContext.java:333) 
>>>
>>>     at 
>>> org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:1323) 
>>>
>>>     at 
>>> org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1009) 
>>>
>>>     at 
>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:627) 
>>>
>>>     at 
>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:411) 
>>>
>>>     at 
>>> org.jboss.system.ServiceController.doChange(ServiceController.java:659)
>>>     at 
>>> org.jboss.system.ServiceController.stop(ServiceController.java:481)
>>>     at 
>>> org.jboss.system.deployers.ServiceDeployer.stop(ServiceDeployer.java:156) 
>>>
>>>     at 
>>> org.jboss.system.deployers.ServiceDeployer.undeploy(ServiceDeployer.java:136) 
>>>
>>>     at 
>>> org.jboss.system.deployers.ServiceDeployer.undeploy(ServiceDeployer.java:46) 
>>>
>>>     at 
>>> org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.undeploy(AbstractSimpleRealDeployer.java:73) 
>>>
>>>     at 
>>> org.jboss.deployers.plugins.deployers.DeployerWrapper.undeploy(DeployerWrapper.java:187) 
>>>
>>>     at 
>>> org.jboss.deployers.plugins.deployers.DeployersImpl.doUninstallParentLast(DeployersImpl.java:947) 
>>>
>>>     at 
>>> org.jboss.deployers.plugins.deployers.DeployersImpl.doUninstallParentLast(DeployersImpl.java:940) 
>>>
>>>     at 
>>> org.jboss.deployers.plugins.deployers.DeployersImpl.uninstall(DeployersImpl.java:902) 
>>>
>>>     at 
>>> org.jboss.dependency.plugins.AbstractControllerContext.uninstall(AbstractControllerContext.java:333) 
>>>
>>>     at 
>>> org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:1323) 
>>>
>>>     at 
>>> org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1009) 
>>>
>>>     at 
>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:627) 
>>>
>>>     at 
>>> org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:411) 
>>>
>>>     at 
>>> org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:420) 
>>>
>>>     at 
>>> org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:354) 
>>>
>>>     at 
>>> org.jboss.deployers.plugins.main.MainDeployerImpl.shutdown(MainDeployerImpl.java:382) 
>>>
>>>     at 
>>> org.jboss.system.server.profileservice.ProfileServiceBootstrap.shutdown(ProfileServiceBootstrap.java:151) 
>>>
>>>     at 
>>> org.jboss.bootstrap.AbstractServerImpl.shutdownServer(AbstractServerImpl.java:482) 
>>>
>>>     at 
>>> org.jboss.bootstrap.AbstractServerImpl$ShutdownHook.run(AbstractServerImpl.java:778) 
>>>
>>> 04:31:57,585 INFO  [QueueService] Queue[/queue/DLQ] stopped
>>> 04:31:57,586 INFO  [QueueService] Queue[/queue/ExpiryQueue] stopped
>>> 04:32:20,561 WARN  [JChannelFactory] Flush failed at 
>>> 127.0.0.1:58939DefaultPartition-ClusteredSSOCache
>>>
>>> -----------------------------------
>>>
>>>
>>>
>>> -----------------------------------
>>> tests-profileservice:
>>>      [echo] creating profileservice config, 
>>> jboss.dist=/qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3 
>>>
>>>      [copy] Copying 378 files to 
>>> /qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3/server/profileservice 
>>>
>>>      [copy] Copied 101 empty directories to 1 empty directory under 
>>> /qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3/server/profileservice 
>>>
>>>      [copy] Copying 1 file to 
>>> /qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3/server/profileservice/conf 
>>>
>>>      [copy] Copying 1 file to 
>>> /qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/build/output/jboss-5.0.0.Beta3/server/profileservice/deployers 
>>>
>>>
>>> BUILD FAILED
>>> /qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/testsuite/build.xml:834: 
>>> The following error occurred while executing this line:
>>> /qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15-noip/trunk/testsuite/build.xml:891: 
>>> Error starting server "profileservice": Found a process already 
>>> listening on:http://localhost:8080 or jnp://localhost:1099
>>>
>>> Total time: 105 minutes 42 seconds
>>> -----------------------------------
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> jboss-development mailing list
>> jboss-development at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/jboss-development
>

-- 
Bela Ban
Lead JGroups / Clustering Team
JBoss - a division of Red Hat

More information about the jboss-development mailing list