]
Harald Wellmann commented on WFLY-6376:
---------------------------------------
I'm not aware of any region factory, I'm simply using the vanilla
{{standalone.xml}} configuration, and my {{persistence.xml}} contains the following:
{code:xml}
<shared-cache-mode>ALL</shared-cache-mode>
<properties>
<property name="hibernate.hbm2ddl.auto"
value="create-drop" />
<property name="hibernate.cache.use_second_level_cache"
value="true" />
</properties>
{code}
Regarding the application class loader, the problem is that
{{LazyInitializingScheduledExecutorService}} does not actually create an executor when it
is instantiated by WildFly. The executor is lazily created when the first task is
submitted on deployment of my application. At that point, the TCCL of the thread
submitting the task is set to the application classloader, and any newly started worker
thread seems to inherit this TCCL.
To reproduce the issue, simply take {{wildfly-quickstart/hibernate5}} and modify
{{persistence.xml}} as above. Deploy and undeploy the quickstart and then take a heap dump
and look at any threads named {{pool-x-thread-y}} and inspect their
{{contextClassLoader}}.
Class loader leak due to Infinispan ExpirationManagerImpl
---------------------------------------------------------
Key: WFLY-6376
URL:
https://issues.jboss.org/browse/WFLY-6376
Project: WildFly
Issue Type: Bug
Components: Clustering, JPA / Hibernate
Affects Versions: 10.0.0.Final
Reporter: Harald Wellmann
Assignee: Paul Ferraro
Priority: Critical
h3. Scenario
Given a WAR containing a persistence unit with second level cache and query cache
enabled, I'm consistently hitting a Metaspace OutOfMemoryError after redeploying the
unchanged application a couple of times.
Analyzing the situation with Eclipse Memory Analyzer, I found one cause to be WFLY-6348,
but even after applying that fix locally, I'm still having a Classloader leak which I
can trace to a thread used by Infinispan referencing an obsolete web app classloader as
context classloader.
Just to rule out that this might be related to WFLY-6283, WFLY-6285, I repeated my
experiments with a local build of WildFly master
(2f11a59aee0dbdd52b65c5c684eafa83c3f418da), with Hibernate locally upgraded to 5.0.9.
I'm still getting a classloader leak with that build.
h3. Analysis
{{org.infinispan.expiration.impl.ExpirationManagerImpl}} uses a
{{LazyInitializingScheduledExecutorService}}. Due to lazy initialization, the
{{ExecutorService}} and the underlying thread pool is not created until my web app is
deployed. Thus, when the {{ExecutorService}} is created, the context class loader is set
to the web app class loader, and this appears to propagate to the threads of the executor
thread pool.
When the application is undeployed, {{ExpirationManagerImpl.stop()}} gets invoked to
cancel any running expiration task. However, the {{ExecutorService}} is not shut down, the
threads remain alive and still keep a reference to the now obsolete context classloader.
h3. Remarks
I'm not sure if this analysis is correct, at least I hope there's a clue for
WildFly and Infinispan experts to identify the real cause.
By the way, it would be helpful if all threads created by Infinispan had meaningful
names, rather than default names like {{pool-5-thread-1}}.