[infinispan-dev] AS clustered classloader leak hunt and web thread local storage keeping reference to a org.infinispan.context.impl.LocalTxInvocationContext ...

[Scott Marlow]

I created ISPN-772 for this issue and have a working patch that cures
the TLS leak.  

I changed commit & rollback methods to call icc.suspend(). icc.suspend
is currently the only way to clear the invocation context thread local
storage. With the current patch, the AS clustered class loader leak test
failures are fixed.

One alternative would be to register a synchronization callback to
handle calling icc.suspend after the transaction ends. It looks like
InvocationContextContainerImpl has a reference to the TM but that might
be null (InvocationContextContainerImpl.getRunningTx() tests tm for
null).  If anyone knows more, let me know...

If we do use a sync callback, possible code sites to adapt would be:

InvocationContextContainerImpl.createInvocationContext()
InvocationContextContainerImpl.createTxInvocationContext()

The other creation methods might be more difficult.  For the remote case
(createRemoteTxInvocationContext), would have to look into the best way
to handle.

For the local createNonTxInvocationContext, would have to look into the
best way to handle.  

It would be good to avoid leaking TLS objects for the more difficult
cases but I also don't want to break stuff with my fix.  What do you
think?

Thanks,
Scott

On Mon, 2010-11-15 at 14:11 -0500, Scott Marlow wrote:
> I'm seeing the following:
> 
> 1.  InvocationContextContainerImpl.createInvocationContext creates a new
> LocalTxInvocationContext (lets call it #3011) and sets thread local
> (icTL) to it.
> 
> 2.  We InvocationContextContainerImpl.suspend() LocalTxInvocationContext
> #3011 which clears the thread local reference.
> 
> 3.  We InvocationContextContainerImpl.resume() LocalTxInvocationContext
> #3011 which sets the thread local reference.
> 
> 4.  We InvocationContextContainerImpl.suspend() LocalTxInvocationContext
> #3011 which clears the thread local reference.
> 
> 5.  We InvocationContextContainerImpl.resume() LocalTxInvocationContext
> #3011 which sets the thread local reference.
> 
> I don't see another suspend call, so LocalTxInvocationContext #3011 is
> still referenced by the thread local InvocationContextContainerImpl.icTL
> 
> I'll create a ISPN jira for this. :)
> 
> 
> Scott
> 
> On Mon, 2010-11-15 at 13:12 -0500, Scott Marlow wrote:
> > I'm getting a little closer, should be closer sooner :)
> > 
> > The LocalTxInvocationContext that got left in thread local storage was
> > instantiated from here http://pastie.org/1300157
> > 
> > That should bring me closer to the cause.  
> > 
> > On Mon, 2010-11-15 at 10:29 -0500, Scott Marlow wrote:
> > > Hi,
> > > 
> > > I'm trying to chase down some memory leaks for
> > > https://jira.jboss.org/browse/JBAS-8613 (turned into a catch all jira
> > > for a few different AS clustering failures).
> > > 
> > > I captured a picture of a classloader leak that is attached to
> > > JBAS-8613.  
> > > 
> > > The http thread threadLocals has a
> > > org.infinispan.context.impl.LocalTxInvocationContext that is keeping the
> > > classloader in memory (via what it references which is shown in the
> > > abovce link or more directly
> > > https://jira.jboss.org/secure/attachment/12338702/Eclipse+Memory
> > > +Analyzer+Tool.png)
> > > 
> > > I'm looking at a different problem at the moment but would like to fix
> > > this leak soon.
> > > 
> > > Why would org.infinispan.context.impl.LocalTxInvocationContext still be
> > > referenced in thread local storage?  Any suggestions on specific classes
> > > that should be looked at?
> > > 
> > > Thanks,  
> > > Scott
> > > 
> > > _______________________________________________
> > > infinispan-dev mailing list
> > > infinispan-dev at lists.jboss.org
> > > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > 
> > 
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev