[jboss-jira] [JBoss JIRA] (WFLY-6686) JGroups ForkChannel can hold references to Hibernate SessionFactoryImpl which cause memory leak

[Paul Ferraro (JIRA)]

    [ https://issues.jboss.org/browse/WFLY-6686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13249842#comment-13249842 ] 

Paul Ferraro commented on WFLY-6686:
------------------------------------

We can workaround this in WildFly while we wait for a proper upstream fix.

> JGroups ForkChannel can hold references to Hibernate SessionFactoryImpl which cause memory leak
> -----------------------------------------------------------------------------------------------
>
>                 Key: WFLY-6686
>                 URL: https://issues.jboss.org/browse/WFLY-6686
>             Project: WildFly
>          Issue Type: Bug
>          Components: Clustering
>         Environment: Using WF 10 build #2291 (Jun 6, 2016 3:40:44 PM)
> https://ci.jboss.org/hudson/job/WildFly-latest-master/2291/changes
>            Reporter: Mathieu Lachance
>            Assignee: Paul Ferraro
>
> We are using Hibernate second level cache through JPA configured as such:
> {code}
>     <properties>
>         <property name="hibernate.cache.use_query_cache" value="true"/>
>         <property name="hibernate.cache.use_second_level_cache" value="true"/>
>         <property name="hibernate.cache.region.factory_class" value="org.jboss.as.jpa.hibernate5.infinispan.SharedInfinispanRegionFactory"/>
>     </properties>
> {code}
> After heap dump inspection, it seems that the JGroups ForkChannel identified by "hibernate" can hold a listener that hold the SessionFactoryImpl which then hold the whole application classloader.
> When undeploying the application, this can lead to classloader leak.
> I took an heap dump of such scenario and analysed it using eclipse memory analyzer (MAT) and here is the result:
> {code}
> Class Name                                                                                                  | Ref. Objects | Shallow Heap | Ref. Shallow Heap | Retained Heap
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> channel org.jgroups.JChannel @ 0xc0eac6a0                                                                   |            1 |          112 |               136 |           448
> '- channel_listeners java.util.concurrent.CopyOnWriteArraySet @ 0xc0ecd260                                  |            1 |           16 |               136 |           112
>    '- al java.util.concurrent.CopyOnWriteArrayList @ 0xc0ecd270                                             |            1 |           24 |               136 |            96
>       '- array java.lang.Object[2] @ 0xc0ecd2b8                                                             |            1 |           24 |               136 |            24
>          '- [0] org.jgroups.fork.ForkChannel @ 0xc103d250                                                   |            1 |          120 |               136 |         1 328
>             '- channel_listeners java.util.concurrent.CopyOnWriteArraySet @ 0xc103d350                      |            1 |           16 |               136 |           968
>                '- al java.util.concurrent.CopyOnWriteArrayList @ 0xc103d360                                 |            1 |           24 |               136 |           952
>                   '- array java.lang.Object[1] @ 0xc103d3a8                                                 |            1 |           24 |               136 |           880
>                      '- [0] org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher @ 0xc103d1f0|            1 |           96 |               136 |           856
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> {code}
> To remove that leak I ugily patched JGroups ForkChannel close method as such:
> {code}
>     /** Closes the fork-channel, essentially setting its state to CLOSED. Note that - contrary to a regular channel -
>      * a closed fork-channel can be connected again: this means re-attaching the fork-channel to the main-channel*/
>     @Override
>     public void close() {
>         ((ForkProtocolStack)prot_stack).remove(fork_channel_id);
>         if(state == State.CLOSED)
>             return;
>         disconnect();                     // leave group if connected
>         prot_stack.destroy();
>         state=State.CLOSED;
>         notifyChannelClosed(this);
> 	this.clearChannelListeners(); // <-- this is the line I added
>     }
> {code}
> With that change in place, the memory leak is gone. I highly doubt though this is an acceptable fix. Though it does confirm my theory.
> I doubt that JGroups is really the culprit -- I'm more in the thinking that the "thing managing" JGroups is the culprit.
> Since I'm not an expert around that field I've opened the issue against the Wildfly project. Feel free to move it to the proper project.
> If you need any other informations let me know.



--
This message was sent by Atlassian JIRA
(v6.4.11#64026)