8:15 a.m.
Paul Ferraro created WFLY-6693:
----------------------------------
Summary: JGroups ForkChannel can hold references to Hibernate
SessionFactoryImpl which cause memory leak
Key: WFLY-6693
URL: https://issues.jboss.org/browse/WFLY-6693
Project: WildFly
Issue Type: Bug
Components: Clustering
Environment: Using WF 10 build #2291 (Jun 6, 2016 3:40:44 PM)
https://ci.jboss.org/hudson/job/WildFly-latest-master/2291/changes
Reporter: Paul Ferraro
Assignee: Paul Ferraro
Priority: Blocker
Fix For: 10.1.0.Final
We are using Hibernate second level cache through JPA configured as such:
{code}
<properties>
<property name="hibernate.cache.use_query_cache"
value="true"/>
<property name="hibernate.cache.use_second_level_cache"
value="true"/>
<property name="hibernate.cache.region.factory_class"
value="org.jboss.as.jpa.hibernate5.infinispan.SharedInfinispanRegionFactory"/>
</properties>
{code}
After heap dump inspection, it seems that the JGroups ForkChannel identified by
"hibernate" can hold a listener that hold the SessionFactoryImpl which then hold
the whole application classloader.
When undeploying the application, this can lead to classloader leak.
I took an heap dump of such scenario and analysed it using eclipse memory analyzer (MAT)
and here is the result:
{code}
Class Name
| Ref. Objects | Shallow Heap | Ref. Shallow Heap | Retained Heap
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
channel org.jgroups.JChannel @ 0xc0eac6a0
| 1 | 112 | 136 | 448
'- channel_listeners java.util.concurrent.CopyOnWriteArraySet @ 0xc0ecd260
| 1 | 16 | 136 | 112
'- al java.util.concurrent.CopyOnWriteArrayList @ 0xc0ecd270
| 1 | 24 | 136 | 96
'- array java.lang.Object[2] @ 0xc0ecd2b8
| 1 | 24 | 136 | 24
'- [0] org.jgroups.fork.ForkChannel @ 0xc103d250
| 1 | 120 | 136 | 1 328
'- channel_listeners java.util.concurrent.CopyOnWriteArraySet @ 0xc103d350
| 1 | 16 | 136 | 968
'- al java.util.concurrent.CopyOnWriteArrayList @ 0xc103d360
| 1 | 24 | 136 | 952
'- array java.lang.Object[1] @ 0xc103d3a8
| 1 | 24 | 136 | 880
'- [0]
org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher @ 0xc103d1f0|
1 | 96 | 136 | 856
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
{code}
To remove that leak I ugily patched JGroups ForkChannel close method as such:
{code}
/** Closes the fork-channel, essentially setting its state to CLOSED. Note that -
contrary to a regular channel -
* a closed fork-channel can be connected again: this means re-attaching the
fork-channel to the main-channel*/
@Override
public void close() {
((ForkProtocolStack)prot_stack).remove(fork_channel_id);
if(state == State.CLOSED)
return;
disconnect(); // leave group if connected
prot_stack.destroy();
state=State.CLOSED;
notifyChannelClosed(this);
this.clearChannelListeners(); // <-- this is the line I added
}
{code}
With that change in place, the memory leak is gone. I highly doubt though this is an
acceptable fix. Though it does confirm my theory.
I doubt that JGroups is really the culprit -- I'm more in the thinking that the
"thing managing" JGroups is the culprit.
Since I'm not an expert around that field I've opened the issue against the
Wildfly project. Feel free to move it to the proper project.
If you need any other informations let me know.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)