I'm seeing intermittent failures in the AS clustering testsuite tests of replicating aspectized web session attributes (aka FIELD granularity session replication.) These are tests where classes are prepared with aopc for use with PojoCache, and then inserted into PojoCache via a web session. Problem is one of stale (i.e. undeployed) classloaders being used by ClassProxyFactory to create proxies. Specifically: 1) Test class A deploys http-scoped-field.war, classloader BaseClassLoader@17eb527{vfsfile:/home/bes/dev/jboss/trunk/testsuite/output/lib/http-scoped-field.war} is created. 2) During test, PojoCache uses ClassProxyFactory to create a proxy of java.util.ArrayList. TCCL in effect is the "http-scoped-field.war" classloader, so that is the classloader used to define the proxy class. 3) Test class A finishes, http-scoped-field.war undeploys; all refs to "http-scoped-field.war" classloader should be released. 4) Test class B deploys different war http-field.war". 5) Same as step 2, during test PojoCache uses ClassProxyFactory to create a proxy of java.util.ArrayList. Now TCCL in effect is the "http-field.war" classloader, Step 5 fails, as ClassProxyFactory finds the cached proxy class from step 2 and tries to create an instance of it: | 2008-04-24 16:19:42,799 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/http-field].[jsp]] (http-besdev%2F192.168.1.145-8080-2) Servlet.service() for servlet jsp threw exception | java.lang.IllegalStateException: BaseClassLoader@17eb527{vfsfile:/home/bes/dev/jboss/trunk/testsuite/output/lib/http-scoped-field.war} classLoader is not connected to a domain (probably undeployed?) for class sun.reflect.ConstructorAccessorImpl | at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:723) | at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:372) | at java.lang.ClassLoader.loadClass(ClassLoader.java:299) | at java.lang.ClassLoader.loadClass(ClassLoader.java:251) | at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319) | at sun.misc.Unsafe.defineClass(Native Method) | at sun.reflect.ClassDefiner.defineClass(ClassDefiner.java:45) | at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:381) | at java.security.AccessController.doPrivileged(Native Method) | at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:377) | at sun.reflect.MethodAccessorGenerator.generateConstructor(MethodAccessorGenerator.java:76) | at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:30) | at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) | at java.lang.reflect.Constructor.newInstance(Constructor.java:494) | at java.lang.Class.newInstance0(Class.java:350) | at java.lang.Class.newInstance(Class.java:303) | at org.jboss.aop.proxy.ClassProxyFactory.newInstance(ClassProxyFactory.java:103) | at org.jboss.aop.proxy.ClassProxyFactory.newInstance(ClassProxyFactory.java:71) | at org.jboss.aop.proxy.ClassProxyFactory.newInstance(ClassProxyFactory.java:66) | at org.jboss.cache.pojo.collection.CollectionInterceptorUtil.createProxy(CollectionInterceptorUtil.java:50) | at org.jboss.cache.pojo.collection.CollectionInterceptorUtil.createListProxy(CollectionInterceptorUtil.java:97) | at org.jboss.cache.pojo.impl.CollectionClassHandler.get(CollectionClassHandler.java:63) | at org.jboss.cache.pojo.impl.PojoCacheDelegate.getObjectInternal(PojoCacheDelegate.java:354) | at org.jboss.cache.pojo.impl.PojoCacheDelegate.getObject(PojoCacheDelegate.java:102) | at org.jboss.cache.pojo.impl.PojoCacheImpl.getObject(PojoCacheImpl.java:209) | at org.jboss.cache.pojo.impl.PojoCacheImpl.org$jboss$cache$pojo$impl$PojoCacheImpl$detach$aop(PojoCacheImpl.java:151) | at org.jboss.cache.pojo.impl.PojoCacheImpl$PojoCacheImplAdvisor.detach_N_6302035201148273652(PojoCacheImpl$PojoCacheImplAdvisor.java) | at org.jboss.cache.pojo.impl.PojoCacheImpl.detach(PojoCacheImpl.java) | at org.jboss.cache.pojo.impl.AdvisedPojoHandler.remove(AdvisedPojoHandler.java:182) | at org.jboss.cache.pojo.impl.PojoCacheDelegate.removeObject(PojoCacheDelegate.java:276) | at org.jboss.cache.pojo.impl.PojoCacheImpl.removeObject(PojoCacheImpl.java:171) | at org.jboss.cache.pojo.impl.PojoCacheImpl.org$jboss$cache$pojo$impl$PojoCacheImpl$detach$aop(PojoCacheImpl.java:154) | at org.jboss.cache.pojo.impl.PojoCacheImpl$PojoCacheImplAdvisor.detach_N_6302035201148273652(PojoCacheImpl$PojoCacheImplAdvisor.java) | at org.jboss.cache.pojo.impl.PojoCacheImpl.detach(PojoCacheImpl.java) | at org.jboss.cache.pojo.impl.PojoCacheImpl.detach(PojoCacheImpl.java:165) | at org.jboss.cache.pojo.impl.PojoCacheImpl.detach(PojoCacheImpl.java:143) | at org.jboss.web.tomcat.service.session.FieldBasedJBossCacheService.removePojo(FieldBasedJBossCacheService.java:197) | at org.jboss.web.tomcat.service.session.FieldBasedClusteredSession.removeJBossInternalAttribute(FieldBasedClusteredSession.java:285) | at org.jboss.web.tomcat.service.session.JBossCacheClusteredSession.removeAttributeInternal(JBossCacheClusteredSession.java:146) | at org.jboss.web.tomcat.service.session.ClusteredSession.removeAttributeInternal(ClusteredSession.java:1348) | at org.jboss.web.tomcat.service.session.ClusteredSession.removeAttributeInternal(ClusteredSession.java:1325) | at org.apache.catalina.session.StandardSession.removeAttribute(StandardSession.java:1207) | at org.apache.catalina.session.StandardSession.removeAttribute(StandardSession.java:1179) | at org.apache.catalina.session.StandardSessionFacade.removeAttribute(StandardSessionFacade.java:140) | at org.apache.jsp.removeAttribute_jsp._jspService(removeAttribute_jsp.java:60) | at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) | at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) | at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:369) | at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:337) | at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:266) | at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) | at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) | at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) | at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96) | at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) | at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) | at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235) | at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175) | at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:189) | at org.jboss.web.tomcat.service.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:89) | at org.jboss.web.tomcat.service.session.BatchReplicationClusteredSessionValve.invoke(BatchReplicationClusteredSessionValve.java:102) | at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433) | at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:90) | at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:96) | at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) | at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) | at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157) | at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) | at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:310) | at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844) | at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:601) | at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447) | at java.lang.Thread.run(Thread.java:595) | I've been searching high and low to try and figure out why the old proxy class is being used, since AOP is using a weak ref to cache it. I don't completely understand this (see below) but here's one thing I've seen: MethodHashesByClass.methodHashesByClass + WeakHashMap.table ++ WeakHashMap$Entry.value +++ HashMap.table ++++ HashMap$Entry.value +++++ MethodPersistentReference.referencedObject ++++++ SoftReference.referent +++++++ java.lang.reflect.Method.clazz ++++++++ class Person.classloader +++++++++ "http-scoped-field.war" classloader classes ++++++++++ proxy to ArrayList A chain from the proxy class running through a SoftReference. Basically, holding a SoftReference to a Method effectively converts the ClassProxyFactory.proxyCache's weak ref to the proxy class into a SoftReference. This would help explain the intermittent nature of these failures; if the SoftReference happens to get released, the proxy class is gone and the test passes. Now, the caveat: while trying to debug this I've developed some tests that load and instantiate AOP-prepared (via aopc) classes in a war, then undeploy the war, fill memory to force all soft references to be cleared, and see then test if the classloader was collected. These are variants on the tests discussed at http://wiki.jboss.org/auth/wiki/ClassloaderLeakUnitTestCase . These tests show the classloader not being released, but an analysis of the heap shows all refs to the classloader going through weak references -- i.e. the classloader should have been collected. If I run the same test with the same classes, except I haven't run aopc first to prepare the classes, the classloader is released. Been tearing my hair out trying to understand this; using different profiling tools to examing heap, etc. So, there's still some mystery here. :( View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4146873#... Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&a...