[jboss-jira] [JBoss JIRA] (WFLY-13259) Memory leak in Hibernate pending-puts cache when L2 cache is enabled

[Sorin Potra (Jira)]

    [ https://issues.redhat.com/browse/WFLY-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14007084#comment-14007084 ] 

Sorin Potra commented on WFLY-13259:
------------------------------------

Before raising this bug I investigated a bit some heap dumps and did some debugging and found the following (please note that I am not an expert in Hibernate / Infinispan so my conclusions might not be very accurate):

- when a new entity is persisted, Hibernate tries to determine if the entity is new or detached. For this it performs a find in the L2 cache and the database. For new entities (our case) the find will add an entry in the pending-puts cache by calling PutFromLoadValidator.registerPendingPut() before reaching to the DB.  For new entities the find will not yield any results from the DB so PutFromLoadValidator.acquirePutFromLoadLock() is not called which basically leaks the pending-puts entry in the cache. This pending-puts entry contains a reference to org.hibernate.internal.SessionImpl which can be quite a large object for large transactions (transactions that load a lot of entities in their PersistenceContext).
- normally the entry in the pending-puts cache expires in 60s so it will be removed from the cache by the reaping thread. However this is still a leak as we need to keep data in memory for longer than required by the application.
- In our case the situation is even worse as the next transaction will load the collection of Files (Children) again and this will cause the load of all its child entities, including the previously added File, which in turn will access the pending-puts entry, effectively resetting its "lastUsed" field and preventing it from expire. 
- The above repeats for each subsequent transaction. This raises another potential issue. From my observations, it seems that at the end of each transaction, the collection entry in the L2 cache is invalidated. This is correct as the collection has been modified by the addition of the newly created File entity. However it also seems that all File entities in the L2 cache are also invalidated which will cause them to be reloaded from the DB during the next transaction. This does not seem to be correct as the existing File entities are not modified in the current transaction. This also causes a performance hit as the L2 cache is basically bypassed for these entities. So this invalidation of the L2 cache entries ensures that the leaked entries in the pending-puts are never removed with each new transaction adding a new such entry in the pending-puts cache.

As a workaround I implemented some logic to manually remove the leaked pending-puts entry after each entity persistence. So basically I'm performing the following steps:

- get a reference to the pending-puts cache by looking up the hibernate CacheContainer in the JNDI and then getting the pending-puts cache
- Get the PendingPutMap entry from the cache, for the key of the entity that has just been persisted
- Using reflection, get a reference to the PutFromLoadValidator instance for the PendingPutMap
- Call acquirePutFromLoadLock() on the above PutFromLoadValidator which will remove the leaked entry from the pending-puts cache

You can find the code in the source code archive attached to this issue (See com.microfocus.sa.persistence.JPAUtil.removePendingPutAfterCreate()).

With this method called after each entity persistence, the problem does not reproduce anymore. However this has the big disadvantage that it uses Hibernate / Infinispan internal classes so it might become invalid in future WildFly releases. 

And then there is the performance impact of continuously loading the File entities from the DB instead of L2 cache. But the performance issue is less severe as our importer runs as a background process (although we might be affected by this on other application flows too).

Hope this makes sense, please let me know in case you need more details.

> Memory leak in Hibernate pending-puts cache when L2 cache is enabled
> --------------------------------------------------------------------
>
>                 Key: WFLY-13259
>                 URL: https://issues.redhat.com/browse/WFLY-13259
>             Project: WildFly
>          Issue Type: Bug
>          Components: JPA / Hibernate
>    Affects Versions: 18.0.1.Final, 19.0.0.Final
>            Reporter: Sorin Potra
>            Assignee: Scott Marlow
>            Priority: Critical
>         Attachments: PathToGCRoots_strong_refs.PNG, afterOOM.hprof.zip, beforeOOM.hprof.zip, pending-puts-leak.PNG, simple-hibernate-war-client.zip, simple-hibernate-war.war, simple-hibernate-war.zip
>
>
> Under certain conditions, described below, WildFly / Hibernate can leak memory into the pending-puts cache eventually causing an OutOfMemoryError. Attached you can find a web application and a standalone client that can be used to reproduce the problem. The web app defines two entities: a Parent and a Child. There is a bidirectional one-to-many relationship between the Parent and the Child. JPA L2 cache is enabled (Infinispan is the cache provider).
> Repeatedly executing a transaction that creates a new Child and adds it to the list of children in the Parent will cause the memory usage to increase steadily until OOM is encountered. If the execution of these transactions is stopped before reaching OOM, the memory will be reclaimed after a few minutes of inactivity.
> Attached you can find the following:
> - simple-hibernate-war.war - the web app that can be deployed in WildFly to reproduce the issue.
> - simple-hibernate-war.zip - the source code for the above web app. The servlet that is invoked by the client to create and persist a new Child is com.microfocus.sa.web.AddChildServlet
> - simple-hibernate-war-client.zip - the standalone client that can be used to invoke the AddChildServlet. After unzipping the archive, the client can be run with the following command from the client folder:
>   
>   java -cp bin com.microfocus.sa.client.AddChildClient
>   
> If you need to run the client multiple times, you have to restart WildFly in between the runs, to start from a fresh state (the web app uses the h2 in memory databasewhich is reset at each restart).
> - pending-puts-leak.PNG - a screeshot from Memory Analyzer showing a leaked SessionImpl instance



--
This message was sent by Atlassian Jira
(v7.13.8#713008)