[jboss-jira] [JBoss JIRA] (WFLY-13259) Memory leak in Hibernate pending-puts cache when L2 cache is enabled

[Scott Marlow (Jira)]

    [ https://issues.redhat.com/browse/WFLY-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14005979#comment-14005979 ] 

Scott Marlow commented on WFLY-13259:
-------------------------------------

Attached is beforeOOM.hprof.zip which was captured from before the OOM occurred (ran WildFly with min/max heap size of 64mb to keep the heap dump zip size under 50mb).

Also attached is afterOOM.hprof.zip which was captured after getting an excessive number of GCs in a short time:
{quote}
10:25:44,688 WARN  [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (default task-2) SQL Error: 90108, SQLState: 90108
10:25:44,856 ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (default task-2) Out of memory.; SQL statement:
select children0_.PARENT_ID as PARENT_I3_0_0_, children0_.CHILD_ID as CHILD_ID1_0_0_, children0_.CHILD_ID as CHILD_ID1_0_1_, children0_.CHILD_NAME as CHILD_NA2_0_1_, children0_.PARENT_ID as PARENT_I3_0_1_ from CHILD children0_ where children0_.PARENT_ID=? [90108-197]
10:25:44,934 WARN  [org.jboss.modules.define] (default task-2) Failed to define class org.hibernate.internal.util.JdbcExceptionHelper in Module "org.hibernate" version 5.3.15.Final from local module loader @5f2108b5 (finder: local module finder @31a5c39e (roots: /home/smarlow/work/wildfly/dist/target/wildfly-20.0.0.Beta1-SNAPSHOT/modules,/home/smarlow/work/wildfly/dist/target/wildfly-20.0.0.Beta1-SNAPSHOT/modules/system/layers/base)): java.lang.OutOfMemoryError: GC overhead limit exceeded

10:25:45,106 ERROR [io.undertow.request] (default task-2) UT005023: Exception handling request to /simple-hibernate-war/addChild: java.lang.OutOfMemoryError: GC overhead limit exceeded

10:25:45,411 ERROR [org.jboss.as.txn] (default task-2) WFLYTX0003: APPLICATION ERROR: transaction still active in request with status 0
10:25:46,368 ERROR [io.undertow.request] (default task-2) UT005071: Undertow request failed HttpServerExchange{ GET /simple-hibernate-war/addChild}: java.lang.OutOfMemoryError: GC overhead limit exceeded

10:25:47,204 WARN  [org.jboss.modules.define] (default task-2) Failed to define class io.undertow.server.HttpServerExchange$ReadDispatchChannel in Module "io.undertow.core" version 2.0.30.Final from local module loader @5f2108b5 (finder: local module finder @31a5c39e (roots: /home/smarlow/work/wildfly/dist/target/wildfly-20.0.0.Beta1-SNAPSHOT/modules,/home/smarlow/work/wildfly/dist/target/wildfly-20.0.0.Beta1-SNAPSHOT/modules/system/layers/base)): java.lang.OutOfMemoryError: GC overhead limit exceeded

10:25:47,543 ERROR [org.jboss.threads.errors] (default task-2) Thread Thread[default task-2,5,main] threw an uncaught exception: java.lang.OutOfMemoryError: GC overhead limit exceeded
{quote}.

[~spotra] please do answer the questions above.

Scott

> Memory leak in Hibernate pending-puts cache when L2 cache is enabled
> --------------------------------------------------------------------
>
>                 Key: WFLY-13259
>                 URL: https://issues.redhat.com/browse/WFLY-13259
>             Project: WildFly
>          Issue Type: Bug
>          Components: JPA / Hibernate
>    Affects Versions: 18.0.1.Final, 19.0.0.Final
>            Reporter: Sorin Potra
>            Assignee: Scott Marlow
>            Priority: Critical
>         Attachments: PathToGCRoots_strong_refs.PNG, afterOOM.hprof.zip, beforeOOM.hprof.zip, pending-puts-leak.PNG, simple-hibernate-war-client.zip, simple-hibernate-war.war, simple-hibernate-war.zip
>
>
> Under certain conditions, described below, WildFly / Hibernate can leak memory into the pending-puts cache eventually causing an OutOfMemoryError. Attached you can find a web application and a standalone client that can be used to reproduce the problem. The web app defines two entities: a Parent and a Child. There is a bidirectional one-to-many relationship between the Parent and the Child. JPA L2 cache is enabled (Infinispan is the cache provider).
> Repeatedly executing a transaction that creates a new Child and adds it to the list of children in the Parent will cause the memory usage to increase steadily until OOM is encountered. If the execution of these transactions is stopped before reaching OOM, the memory will be reclaimed after a few minutes of inactivity.
> Attached you can find the following:
> - simple-hibernate-war.war - the web app that can be deployed in WildFly to reproduce the issue.
> - simple-hibernate-war.zip - the source code for the above web app. The servlet that is invoked by the client to create and persist a new Child is com.microfocus.sa.web.AddChildServlet
> - simple-hibernate-war-client.zip - the standalone client that can be used to invoke the AddChildServlet. After unzipping the archive, the client can be run with the following command from the client folder:
>   
>   java -cp bin com.microfocus.sa.client.AddChildClient
>   
> If you need to run the client multiple times, you have to restart WildFly in between the runs, to start from a fresh state (the web app uses the h2 in memory databasewhich is reset at each restart).
> - pending-puts-leak.PNG - a screeshot from Memory Analyzer showing a leaked SessionImpl instance



--
This message was sent by Atlassian Jira
(v7.13.8#713008)