]
Sorin Potra commented on WFLY-13259:
------------------------------------
[~smarlow] [~gbadner]
I can try to disable the L2 cache on this particular flow as you suggested. However we are
dealing with a large JEE application. It exposes around 150 services (EJBs) with thousands
of operations (public methods). Identifying all the operations that exhibit this
"import" pattern is a huge undertaking. We might have operations that do not
cause OOM but still require more memory due to this behavior. Out clients might have to
use large heap sizes to avoid issues in Production. The fact that, each time a new entity
is persisted, the Hibernate session is leaked into the pending-puts cache for 60s seems
not normal to me. If we have multiple transactions going on in parallel, each creating a
new entity among other things, can lead to a huge number of Hibernate sessions hanging
around until they are evicted from pending-puts cache.
So we might have to stick with our workaround which basically removes the leaked
pending-puts entry after each entity is persisted by calling
PutFromLoadValidator.acquirePutFromLoadLock(). I noticed that the fix attempted tried to
avoid the creation of the pending-put entry. But how about trying to remove the entry in
case of new entities? This might be a silly suggestion as I don't have the knowledge
to really asses if it is feasible. But the workaround that I did seems to be working (see
JPAUtil.removePendingPutAfterCreate() in my test app) although it looks like a terrible
hack as it uses non public APIs.
Memory leak in Hibernate pending-puts cache when L2 cache is enabled
--------------------------------------------------------------------
Key: WFLY-13259
URL:
https://issues.redhat.com/browse/WFLY-13259
Project: WildFly
Issue Type: Enhancement
Components: JPA / Hibernate
Affects Versions: 18.0.1.Final, 19.0.0.Final
Reporter: Sorin Potra
Assignee: Scott Marlow
Priority: Optional
Attachments: PathToGCRoots_strong_refs.PNG, afterOOM.hprof.zip,
beforeOOM.hprof.zip, pending-puts-leak.PNG, simple-hibernate-war-client.zip,
simple-hibernate-war-client.zip.2020-03-25, simple-hibernate-war.war,
simple-hibernate-war.war.2020-03-25, simple-hibernate-war.zip,
simple-hibernate-war.zip.2020-03-25
Under certain conditions, described below, WildFly / Hibernate can leak memory into the
pending-puts cache eventually causing an OutOfMemoryError. Attached you can find a web
application and a standalone client that can be used to reproduce the problem. The web app
defines two entities: a Parent and a Child. There is a bidirectional one-to-many
relationship between the Parent and the Child. JPA L2 cache is enabled (Infinispan is the
cache provider).
Repeatedly executing a transaction that creates a new Child and adds it to the list of
children in the Parent will cause the memory usage to increase steadily until OOM is
encountered. If the execution of these transactions is stopped before reaching OOM, the
memory will be reclaimed after a few minutes of inactivity.
Attached you can find the following:
- simple-hibernate-war.war - the web app that can be deployed in WildFly to reproduce the
issue.
- simple-hibernate-war.zip - the source code for the above web app. The servlet that is
invoked by the client to create and persist a new Child is
com.microfocus.sa.web.AddChildServlet
- simple-hibernate-war-client.zip - the standalone client that can be used to invoke the
AddChildServlet. After unzipping the archive, the client can be run with the following
command from the client folder:
java -cp bin com.microfocus.sa.client.AddChildClient
If you need to run the client multiple times, you have to restart WildFly in between the
runs, to start from a fresh state (the web app uses the h2 in memory databasewhich is
reset at each restart).
- pending-puts-leak.PNG - a screeshot from Memory Analyzer showing a leaked SessionImpl
instance