[jboss-jira] [JBoss JIRA] (WFLY-13259) Memory leak in Hibernate pending-puts cache when L2 cache is enabled

[Radim Vansa (Jira)]

    [ https://issues.redhat.com/browse/WFLY-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14010011#comment-14010011 ] 

Radim Vansa commented on WFLY-13259:
------------------------------------

[~smarlow]I believe that simply setting TRACE level on PutFromLoadValidator gives you all the details.

I think Sorin's analysis is correct; the pending put is not removed from the cache if the entity is not found. IIRC there was no call from Hibernate's side that would tell the cache that it is not going to put anything to the cache, so we're relying on the expiration. With current codebase it might be possible to register a synchronization to remove the entry when the transaction completes/when the session is closed; I can't see if that could break anything or if it would just have performance impact (allocations, synchronized entry access...).

If the same key is repeatedly accessed, every time we check if there's more than 10 'pending puts' (hardcoded GC_THRESHOLD) and in that case we try to remove all records that are older than cache's {{expiration.maxIdle}}. That's where the 60 seconds come from, you can lower the value, but there's no option to let the transaction clean up immediately.

I don't think that this would be specific to 5.3; there were very little changes with regards to the invalidation mode.

You can also try if {{NONSTRICT_READ_WRITE}} access strategy would work for you; this is enabled for non-clustered caches, too. This has the downside of short period during transaction commit when the DB is already committed but cache would return stale values.

> Memory leak in Hibernate pending-puts cache when L2 cache is enabled
> --------------------------------------------------------------------
>
>                 Key: WFLY-13259
>                 URL: https://issues.redhat.com/browse/WFLY-13259
>             Project: WildFly
>          Issue Type: Bug
>          Components: JPA / Hibernate
>    Affects Versions: 18.0.1.Final, 19.0.0.Final
>            Reporter: Sorin Potra
>            Assignee: Scott Marlow
>            Priority: Critical
>         Attachments: PathToGCRoots_strong_refs.PNG, afterOOM.hprof.zip, beforeOOM.hprof.zip, pending-puts-leak.PNG, simple-hibernate-war-client.zip, simple-hibernate-war-client.zip.2020-03-25, simple-hibernate-war.war, simple-hibernate-war.war.2020-03-25, simple-hibernate-war.zip, simple-hibernate-war.zip.2020-03-25
>
>
> Under certain conditions, described below, WildFly / Hibernate can leak memory into the pending-puts cache eventually causing an OutOfMemoryError. Attached you can find a web application and a standalone client that can be used to reproduce the problem. The web app defines two entities: a Parent and a Child. There is a bidirectional one-to-many relationship between the Parent and the Child. JPA L2 cache is enabled (Infinispan is the cache provider).
> Repeatedly executing a transaction that creates a new Child and adds it to the list of children in the Parent will cause the memory usage to increase steadily until OOM is encountered. If the execution of these transactions is stopped before reaching OOM, the memory will be reclaimed after a few minutes of inactivity.
> Attached you can find the following:
> - simple-hibernate-war.war - the web app that can be deployed in WildFly to reproduce the issue.
> - simple-hibernate-war.zip - the source code for the above web app. The servlet that is invoked by the client to create and persist a new Child is com.microfocus.sa.web.AddChildServlet
> - simple-hibernate-war-client.zip - the standalone client that can be used to invoke the AddChildServlet. After unzipping the archive, the client can be run with the following command from the client folder:
>   
>   java -cp bin com.microfocus.sa.client.AddChildClient
>   
> If you need to run the client multiple times, you have to restart WildFly in between the runs, to start from a fresh state (the web app uses the h2 in memory databasewhich is reset at each restart).
> - pending-puts-leak.PNG - a screeshot from Memory Analyzer showing a leaked SessionImpl instance



--
This message was sent by Atlassian Jira
(v7.13.8#713008)