[ https://jira.jboss.org/browse/JBAS-8665?page=com.atlassian.jira.plugin.sy... ] Shelly McGowan reassigned JBAS-8665: ------------------------------------ Assignee: Scott Marlow -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/JBAS-8665?page=com.atlassian.jira.plugin.sy... ] Scott Marlow commented on JBAS-8665: ------------------------------------ For the classloader-leak-in-ear.ear/classloader-leak-simple-ejb.war leak failure, one suspect is com.arjuna.ats.internal.arjuna.recovery.Listener (see attachment). -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/JBAS-8665?page=com.atlassian.jira.plugin.sy... ] Scott Marlow updated JBAS-8665: ------------------------------- Attachment: Screenshot-Eclipse Memory Analyzer -1.png -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/JBAS-8665?page=com.atlassian.jira.plugin.sy... ] Scott Marlow commented on JBAS-8665: ------------------------------------ Call stack that shows origin of the leaked classloader (that goes with screenshot from November 29 4:32pm) is at http://pastebin.com/L1bQgqTL AS output from same leak (note that we correlate on the "mycount=1122" value that also shows up in screendump): 16:10:54,845 INFO [org.jboss.test.classloader.leak.clstore.ClassLoaderStore] Filling 96.0% of free memory. Free memory=360174976 Total Memory=486604800 Max Memory=486604800, target ref = org.jboss.classloader.spi.base.BaseClassLoader@76f9d876BaseClassLoader@76f9d876{vfs:///home/smarlow/work/as6/testsuite/output/lib/classloader-leak-in-ear.ear/classloader-leak-simple-ejb.war/}mycount=1122 -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/JBAS-8665?page=com.atlassian.jira.plugin.sy... ] Scott Marlow edited comment on JBAS-8665 at 11/30/10 9:32 AM: -------------------------------------------------------------- Next, I would like to understand what is going on internally in jbossjts (4.13.1.Final) that leads to this problem (and what triggers it). I'm attempting to build jbossjts via "ant jbossjts" command in root folder of 4.13.1.Final source. If that works, I'll look at instrumenting code responsible for the below reference. com.arjuna.ats.internal.arjuna.recovery.Listener -> java.security.AccessControlContext -> java.security.ProtectionDomain[] -> java.security.ProtectionDomain -> org.apache.jasper.servlet.JasperLoader -> org.jboss.web.tomcat.service.WebCtxLoader$ENCLoader -> org.jboss.classloader.spi.base.BaseClassLoader We might also want to investigate whether we have additional leaks via com.arjuna.ats.internal.arjuna.recovery.Listener.contextClassLoader. I'll attach JBOSSJTS_Listener_contextClassLoader.png which shows what I saw there. However, note that is not causing the unit test failure. was (Author: smarlow(a)redhat.com): Next, I would like to understand what is going on internally in jbossjts (4.13.1.Final) that leads to this problem (and what triggers it). -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/JBAS-8665?page=com.atlassian.jira.plugin.sy... ] Scott Marlow commented on JBAS-8665: ------------------------------------ The leaked com.arjuna.ats.internal.arjuna.recovery.Listener appears to originate from this call stack: http://pastebin.com/fQW3W9jh -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/JBAS-8665?page=com.atlassian.jira.plugin.sy... ] Scott Marlow updated JBAS-8665: ------------------------------- Attachment: debugJTS.patch -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/JBAS-8665?page=com.atlassian.jira.plugin.sy... ] Scott Marlow updated JBAS-8665: ------------------------------- Attachment: eagerInitJTS.patch -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/JBAS-8665?page=com.atlassian.jira.plugin.sy... ] Scott Marlow edited comment on JBAS-8665 at 11/30/10 4:24 PM: -------------------------------------------------------------- I tried to patch the JBossTS to eager initialize sooner (see attached eagerInitJT.patch). Also attached is debugJTS.patch for enabling the debug prints in JBossTS. Running with the eagerInitJT.patch seemed to result in a NCDFE: org.omg.CosTransactions.PropagationContext http://pastebin.com/3MrQAvrU It seems that we need the com.arjuna.ats.internal.arjuna.recovery.Listener thread to be created earlier (so that we don't create from the web thread which leaks the web thread inheritedAccessControlContext). I wonder if we could make a transaction-jboss-beans.xml change to get more eager initialization of JBossTS... was (Author: smarlow(a)redhat.com): I tried to patch the JBossTS to eager initialize sooner (see attached eagerInitJT.patch). Also attached is debugJTS.patch for enabling the debug prints in JBossTS. Running with the eagerInitJT.patch seemed to result in a NCDFE: org.omg.CosTransactions.PropagationContext http://pastebin.com/3MrQAvrU It seems that we need the com.arjuna.ats.internal.arjuna.recovery.Listener thread to be created earlier. I wonder if we could make a transaction-jboss-beans.xml change to get more eager initialization JBossTS... -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/JBAS-8665?page=com.atlassian.jira.plugin.sy... ] Scott Marlow commented on JBAS-8665: ------------------------------------ The transaction-jboss-beans.xml change worked. For the next CL leak, I'm trying a jboss-switchboard-mc-impl.jar change from Jaikiran... -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/JBAS-8665?page=com.atlassian.jira.plugin.sy... ] Scott Marlow reassigned JBAS-8665: ---------------------------------- Assignee: jaikiran pai (was: Scott Marlow) reassigning for the switchboard fix which I verified as working on my machine. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/JBAS-8665?page=com.atlassian.jira.plugin.sy... ] Scott Marlow reassigned JBAS-8665: ---------------------------------- Assignee: Scott Marlow (was: jaikiran pai) -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/JBAS-8665?page=com.atlassian.jira.plugin.sy... ] Scott Marlow reopened JBAS-8665: -------------------------------- http://hudson.qa.jboss.com/hudson/view/JBoss%20AS/job/JBoss-AS-6.0.x-clas... shows a classloader leak failure. I'll attach a picture of the references (see a lot of weak references) -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/JBAS-8665?page=com.atlassian.jira.plugin.sy... ] Scott Marlow commented on JBAS-8665: ------------------------------------ Just for fun, I hacked the test to loop until the leak is gone (just in case we hadn't finished clearing weak references yet). I'm not sure if we did but at least, I know that trying a bit longer doesn't help. I wonder what has changed that would cause only the org.jboss.test.classloader.leak.test.IsolatedEjb3EarClassloaderLeakTestCase test to fail. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/JBAS-8665?page=com.atlassian.jira.plugin.sy... ] Scott Marlow commented on JBAS-8665: ------------------------------------ I'm trying locally to revert back to <version.org.jboss.ejb3.depchain>1.0.0-alpha-27</version.org.jboss.ejb3.depchain>. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/JBAS-8665?page=com.atlassian.jira.plugin.sy... ] Scott Marlow commented on JBAS-8665: ------------------------------------ A list of the (weak) references to the leaked classloader are here http://pastebin.com/zhDbmbK5 -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/JBAS-8665?page=com.atlassian.jira.plugin.sy... ] Scott Marlow commented on JBAS-8665: ------------------------------------ The following call stacks correspond to the threads identified in the previous comment (inactive threads with thread local or TCCL set to ejb3 async stuff). Thread call stack for "pool-2-thread-2" http://pastebin.com/YZTy3h4T Thread call stack for "ajp-172.19.1.2-8009-Acceptor-0" http://pastebin.com/n4Rs9rnF The "pool-2-thread-2" issue is a good suspect to follow up on. Need to know why <Java Local> org.jboss.ha.jndi.DetachedHANamingService$AcceptHandler is not cleared (which references org.jboss.ejb3.async.deployer.AsyncExecutorServiceDeployer and org.jboss.ejb3.async.impl.interceptor.FutureSerializingInterceptor). I don't expect the classloader leak test to be using the AJP thread, so it probably doesn't matter for the classloader leak tests. The AJP thread was probably created by the boot loader and inherited the boot loader thread TCCL. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/JBAS-8665?page=com.atlassian.jira.plugin.sy... ] Scott Marlow resolved JBAS-8665. -------------------------------- Resolution: Done Separating regression into JBAS-8715 and resolving this issue which was already fixed. I had hoped the new regression would be a quick/easy fix to fold into this jira, but... -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira