[jboss-jira] [JBoss JIRA] (WFLY-13259) Memory leak in Hibernate pending-puts cache when L2 cache is enabled

[ https://issues.redhat.com/browse/WFLY-13259?page=com.atlassian.jira.plugi... ] Sorin Potra commented on WFLY-13259: ------------------------------------ Before raising this bug I investigated a bit some heap dumps and did some debugging and found the following (please note that I am not an expert in Hibernate / Infinispan so my conclusions might not be very accurate): - when a new entity is persisted, Hibernate tries to determine if the entity is new or detached. For this it performs a find in the L2 cache and the database. For new entities (our case) the find will add an entry in the pending-puts cache by calling PutFromLoadValidator.registerPendingPut() before reaching to the DB. For new entities the find will not yield any results from the DB so PutFromLoadValidator.acquirePutFromLoadLock() is not called which basically leaks the pending-puts entry in the cache. This pending-puts entry contains a reference to org.hibernate.internal.SessionImpl which can be quite a large object for large transactions (transactions that load a lot of entities in their PersistenceContext). - normally the entry in the pending-puts cache expires in 60s so it will be removed from the cache by the reaping thread. However this is still a leak as we need to keep data in memory for longer than required by the application. - In our case the situation is even worse as the next transaction will load the collection of Files (Children) again and this will cause the load of all its child entities, including the previously added File, which in turn will access the pending-puts entry, effectively resetting its "lastUsed" field and preventing it from expire. - The above repeats for each subsequent transaction. This raises another potential issue. From my observations, it seems that at the end of each transaction, the collection entry in the L2 cache is invalidated. This is correct as the collection has been modified by the addition of the newly created File entity. However it also seems that all File entities in the L2 cache are also invalidated which will cause them to be reloaded from the DB during the next transaction. This does not seem to be correct as the existing File entities are not modified in the current transaction. This also causes a performance hit as the L2 cache is basically bypassed for these entities. So this invalidation of the L2 cache entries ensures that the leaked entries in the pending-puts are never removed with each new transaction adding a new such entry in the pending-puts cache. As a workaround I implemented some logic to manually remove the leaked pending-puts entry after each entity persistence. So basically I'm performing the following steps: - get a reference to the pending-puts cache by looking up the hibernate CacheContainer in the JNDI and then getting the pending-puts cache - Get the PendingPutMap entry from the cache, for the key of the entity that has just been persisted - Using reflection, get a reference to the PutFromLoadValidator instance for the PendingPutMap - Call acquirePutFromLoadLock() on the above PutFromLoadValidator which will remove the leaked entry from the pending-puts cache You can find the code in the source code archive attached to this issue (See com.microfocus.sa.persistence.JPAUtil.removePendingPutAfterCreate()). With this method called after each entity persistence, the problem does not reproduce anymore. However this has the big disadvantage that it uses Hibernate / Infinispan internal classes so it might become invalid in future WildFly releases. And then there is the performance impact of continuously loading the File entities from the DB instead of L2 cache. But the performance issue is less severe as our importer runs as a background process (although we might be affected by this on other application flows too). Hope this makes sense, please let me know in case you need more details. -- This message was sent by Atlassian Jira (v7.13.8#713008)