The latest hudson run (first since last Fri AM) shows a ton of clustering failures. This is not due to the only clustering related commit made since the last good run, the upgrade of JBC/PojoCache to 3.0.0.GA. Reverting that locally doesn't fix the failures, and applying the upgrade to r81421 (last known good rev from Friday) doesn't introduce the failures. So, it's something else. Failures are related to the deployment of a utility sar that is deployed at the start of each test class and undeployed at end. This deploys properly for the first test class and then begins failing due to what looks like a classloader leak problem from the undeploy. If anyone has any idea on what might have changed to lead to this, please fix ASAP; it would definitely suck if all the work since last Friday AM had to be reverted. The failure: 2008-11-24 16:27:47,727 DEBUG [org.jboss.deployers.structure.spi.helpers.AbstractDeploymentContext] (RMI TCP Connection(72)-10.16.93.37) Added component ClusterTestCacheConfigs to vfszip:/qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15/trunk/testsuite/output/lib/jbosscache-helper.sar 2008-11-24 16:27:47,736 DEBUG [org.jboss.deployers.structure.spi.helpers.AbstractDeploymentContext] (RMI TCP Connection(72)-10.16.93.37) Added component ClusterTestCacheHelper to vfszip:/qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15/trunk/testsuite/output/lib/jbosscache-helper.sar 2008-11-24 16:27:47,754 ERROR [org.jboss.kernel.plugins.dependency.AbstractKernelController] (RMI TCP Connection(72)-10.16.93.37) Error installing to Described: name=ClusterTestCacheConfigs state=PreInstall java.lang.RuntimeException: java.lang.IllegalStateException: BaseClassLoader@e89dc2{vfszip:/qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15/trunk/testsuite/output/lib/jbosscache-helper.sar} classLoader is not connected to a domain (probably undeployed?) for class org.jboss.cache.pojo.annotation.Replicable at org.jboss.aop.pointcut.Util.matchesClassExpr(Util.java:137) at org.jboss.aop.pointcut.TypeMatcher.visit(TypeMatcher.java:153) at org.jboss.aop.pointcut.ast.ASTClass.jjtAccept(ASTClass.java:43) at org.jboss.aop.pointcut.TypeMatcher.visit(TypeMatcher.java:73) at org.jboss.aop.pointcut.ast.ASTStart.jjtAccept(ASTStart.java:42) at org.jboss.aop.introduction.InterfaceIntroduction.matches(InterfaceIntroduction.java:274) at org.jboss.aop.AspectManager.applyInterfaceIntroductions(AspectManager.java:1794) at org.jboss.aop.Advisor.initializeInterfaceIntroductions(Advisor.java:497) at org.jboss.aop.proxy.container.ClassProxyContainer.initialise(ClassProxyContainer.java:176) at org.jboss.aop.proxy.container.InstanceProxyContainer.<init>(InstanceProxyContainer.java:56) at org.jboss.aop.proxy.container.InstanceProxyContainer.createInstanceProxyContainer(InstanceProxyContainer.java:68) at org.jboss.aop.proxy.container.ContainerCache.initInstanceContainer(ContainerCache.java:292) at org.jboss.aop.proxy.container.ContainerCache.initialise(ContainerCache.java:93) at org.jboss.aop.proxy.container.ContainerCache.initialise(ContainerCache.java:72) at org.jboss.aop.microcontainer.integration.AOPDependencyBuilder.getDependencies(AOPDependencyBuilder.java:99) at org.jboss.kernel.plugins.dependency.DescribeAction.installActionInternal(DescribeAction.java:56) at org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:54) at org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:42) at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62) at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71) at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51) at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1598) at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:934) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1062) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:984) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:774) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:540) at org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer.deploy(BeanMetaDataDeployer.java:121) at org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer.deploy(BeanMetaDataDeployer.java:51) at org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalDeploy(AbstractSimpleRealDeployer.java:62) at org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:50) at org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:171) at org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(DeployersImpl.java:1439) at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1157) at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1178) at org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:1098) at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1598) at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:934) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1062) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:984) at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:822) at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:553) at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:781) at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:541) at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:827) at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:802) at org.jboss.deployment.MainDeployer.redeploy(MainDeployer.java:588) at sun.reflect.GeneratedMethodAccessor392.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96) at org.jboss.mx.server.Invocation.invoke(Invocation.java:88) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668) at sun.reflect.GeneratedMethodAccessor317.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.jmx.connector.invoker.InvokerAdaptorService.invoke(InvokerAdaptorService.java:270) at sun.reflect.GeneratedMethodAccessor314.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96) at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:138) at org.jboss.mx.server.Invocation.invoke(Invocation.java:90) at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:140) at org.jboss.jmx.connector.invoker.SerializableInterceptor.invoke(SerializableInterceptor.java:74) at org.jboss.mx.server.Invocation.invoke(Invocation.java:90) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668) at org.jboss.invocation.jrmp.server.JRMPProxyFactory.invoke(JRMPProxyFactory.java:180) at sun.reflect.GeneratedMethodAccessor313.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96) at org.jboss.mx.server.Invocation.invoke(Invocation.java:88) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668) at org.jboss.invocation.jrmp.server.JRMPInvoker$MBeanServerAction.invoke(JRMPInvoker.java:855) at org.jboss.invocation.jrmp.server.JRMPInvoker.invoke(JRMPInvoker.java:422) at sun.reflect.GeneratedMethodAccessor312.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:294) at sun.rmi.transport.Transport$1.run(Transport.java:153) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:149) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:466) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:707) at java.lang.Thread.run(Thread.java:595) Caused by: java.lang.IllegalStateException: BaseClassLoader@e89dc2{vfszip:/qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15/trunk/testsuite/output/lib/jbosscache-helper.sar} classLoader is not connected to a domain (probably undeployed?) for class org.jboss.cache.pojo.annotation.Replicable at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:730) at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:378) at java.lang.ClassLoader.loadClass(ClassLoader.java:251) at org.jboss.aop.pointcut.MatcherStrategy.subtypeOf(MatcherStrategy.java:71) at org.jboss.aop.pointcut.Util.subtypeOf(Util.java:409) at org.jboss.aop.pointcut.Util.matchesClassExpr(Util.java:124) ... 93 more -------- Original Message -------- Subject: JBoss-AS-5.0.x-TestSuite-sun15 - Build # 1097 - Unstable! Date: Mon, 24 Nov 2008 18:50:56 -0500 (EST) From: jboss-qa-internal(a)redhat.com To: builds(a)lists.jboss.org JBoss-AS-5.0.x-TestSuite-sun15 - Build # 1097 - Unstable: Check console output at https://hudson.jboss.org/hudson//job/JBoss-AS-5.0.x-TestSuite-sun15/1097 to view the results. Modifications since last build: [pgier] Small change to make the jar creation more clear because the zipgroupfileset task was being used incorrectly. [mmillson] Fix EJB TIMERS table creation for Oracle schema [JBPAPP-1366]. [adrian(a)jboss.org] Fix the eclipse build [pgier] [JBBUILD-498] Update maven assemblies to match ant output. [pgier] Remove some redundant includes. [pgier] [JBAS-6238] Remove jbossmq module from pom. _______________________________________________ builds mailing list builds(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/builds

I would look at the upgrade to aop 2.0.0.SP1 from 2.0.0.GA. So far the tests are running for me (ant tests-clustering-all-stacks).

Scott Stark wrote: > but as soon as the second set of tests started running they are > failing. I'm rerunning with aop 2.0.0.GA > > Scott Stark wrote: >> I would look at the upgrade to aop 2.0.0.SP1 from 2.0.0.GA. So far >> the tests are running for me (ant tests-clustering-all-stacks). >> Kabir did mention that some AS aop tests failed with the upgrade, on hudson, even though they passed locally. It is interesting because 2.0.0.SP1 had just privileged blocks introduced with no change in functionality.

Can you try with the latest trunk? I just did some rollbacks. Kabir Khan wrote: > On 25 Nov 2008, at 06:45, Anil Saldhana wrote: >> Scott Stark wrote: >>> but as soon as the second set of tests started running they are >>> failing. I'm rerunning with aop 2.0.0.GA >>> >>> Scott Stark wrote: >>>> I would look at the upgrade to aop 2.0.0.SP1 from 2.0.0.GA. So >>>> far the tests are running for me (ant tests-clustering-all- >>>> stacks). >>>> >> Kabir did mention that some AS aop tests failed with the upgrade, >> on hudson, even though they passed locally. >> >> It is interesting because 2.0.0.SP1 had just privileged blocks >> introduced with no change in functionality. > I am trying to figure out what happened. Locally AS trunk fails for > the following aop tests > org > .jboss > .test > .aop.test.ScopedExtenderBaseLoadedScopedNoParentDelegationTestCase > org.jboss.test.aop.test.ScopedUnitTestCase > with both AOP 2.0.0.GA and AOP 2.0.0.SP1. Going back it passes with > both for AS trunk r81441 and fails for AS trunk r814551. Looking at > the commits between there, I cannot see anything obvious, but I'll > keep investigating > _______________________________________________ > jboss-development mailing list > jboss-development(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-development _______________________________________________ jboss-development mailing list jboss-development(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-development

It must be my change to reflect 2.0.1.GA, as I see some failures in MC/Kernel. :-( I'm looking into it now. Kabir Khan wrote: > Am updating now. In the meantime, r81442 passed for me and r81446 > failed > On 25 Nov 2008, at 11:19, Dimitris Andreadis wrote: >> Can you try with the latest trunk? I just did some rollbacks. >> >> Kabir Khan wrote: >>> On 25 Nov 2008, at 06:45, Anil Saldhana wrote: >>>> Scott Stark wrote: >>>>> but as soon as the second set of tests started running they are >>>>> failing. I'm rerunning with aop 2.0.0.GA >>>>> >>>>> Scott Stark wrote: >>>>>> I would look at the upgrade to aop 2.0.0.SP1 from 2.0.0.GA. So >>>>>> far the tests are running for me (ant tests-clustering-all- >>>>>> stacks). >>>>>> >>>> Kabir did mention that some AS aop tests failed with the >>>> upgrade, on hudson, even though they passed locally. >>>> >>>> It is interesting because 2.0.0.SP1 had just privileged blocks >>>> introduced with no change in functionality. >>> I am trying to figure out what happened. Locally AS trunk fails >>> for the following aop tests >>> org >>> .jboss >>> .test >>> .aop >>> .test.ScopedExtenderBaseLoadedScopedNoParentDelegationTestCase >>> org.jboss.test.aop.test.ScopedUnitTestCase >>> with both AOP 2.0.0.GA and AOP 2.0.0.SP1. Going back it passes >>> with both for AS trunk r81441 and fails for AS trunk r814551. >>> Looking at the commits between there, I cannot see anything >>> obvious, but I'll keep investigating >>> _______________________________________________ >>> jboss-development mailing list >>> jboss-development(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/jboss-development >> _______________________________________________ >> jboss-development mailing list >> jboss-development(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jboss-development > _______________________________________________ > jboss-development mailing list > jboss-development(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-development _______________________________________________ jboss-development mailing list jboss-development(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-development

Yes, I found that r81443 which is the upgrade to reflect 2.0.1.GA gives the org.jboss.test.aop.test failures mentioned. The aop tests run fine in current as trunk with both aop 2.0.0.GA and 2.0.0.SP1. Dimitris, can I reapply AOP 2.0.0.SP1? On 25 Nov 2008, at 12:04, Ales Justin wrote: > It must be my change to reflect 2.0.1.GA, > as I see some failures in MC/Kernel. :-( > > I'm looking into it now. > > Kabir Khan wrote: >> Am updating now. In the meantime, r81442 passed for me and r81446 failed >> On 25 Nov 2008, at 11:19, Dimitris Andreadis wrote: >>> Can you try with the latest trunk? I just did some rollbacks. >>> >>> Kabir Khan wrote: >>>> On 25 Nov 2008, at 06:45, Anil Saldhana wrote: >>>>> Scott Stark wrote: >>>>>> but as soon as the second set of tests started running they are >>>>>> failing. I'm rerunning with aop 2.0.0.GA >>>>>> >>>>>> Scott Stark wrote: >>>>>>> I would look at the upgrade to aop 2.0.0.SP1 from 2.0.0.GA. So >>>>>>> far the tests are running for me (ant tests-clustering-all-stacks). >>>>>>> >>>>> Kabir did mention that some AS aop tests failed with the upgrade, >>>>> on hudson, even though they passed locally. >>>>> >>>>> It is interesting because 2.0.0.SP1 had just privileged blocks >>>>> introduced with no change in functionality. >>>> I am trying to figure out what happened. Locally AS trunk fails for >>>> the following aop tests >>>> org.jboss.test.aop.test.ScopedExtenderBaseLoadedScopedNoParentDelegationTestCase >>>> org.jboss.test.aop.test.ScopedUnitTestCase >>>> with both AOP 2.0.0.GA and AOP 2.0.0.SP1. Going back it passes with >>>> both for AS trunk r81441 and fails for AS trunk r814551. Looking at >>>> the commits between there, I cannot see anything obvious, but I'll >>>> keep investigating >>>> _______________________________________________ >>>> jboss-development mailing list >>>> jboss-development(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/jboss-development >>> _______________________________________________ >>> jboss-development mailing list >>> jboss-development(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/jboss-development >> _______________________________________________ >> jboss-development mailing list >> jboss-development(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jboss-development > _______________________________________________ > jboss-development mailing list > jboss-development(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-development _______________________________________________ jboss-development mailing list jboss-development(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-development

The AOP tests in trunk run fine with the latest jboss-reflect snapshot mentioned in Ales's forum thread On 25 Nov 2008, at 12:22, Ales Justin wrote: > http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4192071#... > > Kabir Khan wrote: >> Yes, I found that r81443 which is the upgrade to reflect 2.0.1.GA >> gives the org.jboss.test.aop.test failures mentioned. >> The aop tests run fine in current as trunk with both aop 2.0.0.GA and >> 2.0.0.SP1. >> Dimitris, can I reapply AOP 2.0.0.SP1? >> On 25 Nov 2008, at 12:04, Ales Justin wrote: >>> It must be my change to reflect 2.0.1.GA, >>> as I see some failures in MC/Kernel. :-( >>> >>> I'm looking into it now. >>> >>> Kabir Khan wrote: >>>> Am updating now. In the meantime, r81442 passed for me and r81446 >>>> failed >>>> On 25 Nov 2008, at 11:19, Dimitris Andreadis wrote: >>>>> Can you try with the latest trunk? I just did some rollbacks. >>>>> >>>>> Kabir Khan wrote: >>>>>> On 25 Nov 2008, at 06:45, Anil Saldhana wrote: >>>>>>> Scott Stark wrote: >>>>>>>> but as soon as the second set of tests started running they are >>>>>>>> failing. I'm rerunning with aop 2.0.0.GA >>>>>>>> >>>>>>>> Scott Stark wrote: >>>>>>>>> I would look at the upgrade to aop 2.0.0.SP1 from 2.0.0.GA. So >>>>>>>>> far the tests are running for me (ant >>>>>>>>> tests-clustering-all-stacks). >>>>>>>>> >>>>>>> Kabir did mention that some AS aop tests failed with the >>>>>>> upgrade, on hudson, even though they passed locally. >>>>>>> >>>>>>> It is interesting because 2.0.0.SP1 had just privileged blocks >>>>>>> introduced with no change in functionality. >>>>>> I am trying to figure out what happened. Locally AS trunk fails >>>>>> for the following aop tests >>>>>> org.jboss.test.aop.test.ScopedExtenderBaseLoadedScopedNoParentDelegationTestCase >>>>>> org.jboss.test.aop.test.ScopedUnitTestCase >>>>>> with both AOP 2.0.0.GA and AOP 2.0.0.SP1. Going back it passes >>>>>> with both for AS trunk r81441 and fails for AS trunk r814551. >>>>>> Looking at the commits between there, I cannot see anything >>>>>> obvious, but I'll keep investigating >>>>>> _______________________________________________ >>>>>> jboss-development mailing list >>>>>> jboss-development(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/jboss-development >>>>> _______________________________________________ >>>>> jboss-development mailing list >>>>> jboss-development(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/jboss-development >>>> _______________________________________________ >>>> jboss-development mailing list >>>> jboss-development(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/jboss-development >>> _______________________________________________ >>> jboss-development mailing list >>> jboss-development(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/jboss-development >> _______________________________________________ >> jboss-development mailing list >> jboss-development(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jboss-development > _______________________________________________ > jboss-development mailing list > jboss-development(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-development _______________________________________________ jboss-development mailing list jboss-development(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-development

but as soon as the second set of tests started running they are failing. I'm rerunning with aop 2.0.0.GA Scott Stark wrote: > I would look at the upgrade to aop 2.0.0.SP1 from 2.0.0.GA. So far > the tests are running for me (ant tests-clustering-all-stacks). > _______________________________________________ jboss-development mailing list jboss-development(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-development