Hi,

As requested in the forum (post below) I have created this issue.

In a JTA multi-threaded environment we are working on a project where we are processing large amounts of data. But we ran into a problem where it seems that our software is leaking memory. We analyzed the heapdumps and determined that it was Envers’s AuditProcessManager that was keeping data from being garbage collected.

Perhaps this issue should be posted in the Jboss/Narayana forums. I have been debugging and testing and determined that the cause lies in the fact that the transaction times out and the Transaction Reaper aborts the transaction at a point where Hibernate is not getting a chance to cleanup.

The issue seems to be related and/or originated in the following issue/fix: HHH-7910 Closed

I have tried to write it down as simple as possible. The following is happening (Note that the lock I am talking about is the _syncLock in TwoPhaseCoordinator):
Thread: work
Thread: flush. Envers registers beforeCompletion and afterCompletion on Session’s actionQueue (afterCompletion will do the cleanup)
Thread: doCommit
Thread: beforeCompletion -> lock -> Envers does its work
Reaper: abort transaction
Reaper: afterCompletion -> wait for lock
Thread: finds out transaction is rolled back and runs delayedAfterCompletion
Thread: ends beforeCompletion -> unlock
Reaper: gets lock, calls afterCompletion and finds out it is not the original Thread, thus sets delayedAfterCompletion.
Thread: afterCompletion -> wait for lock
Reaper: finishes afterCompletion -> unlock
Thread: gets lock and finds out afterCompletion has already been called (_afterCalled in TwoPhaseCoordinator) -> nothing to do and unlock

Now Envers still has a reference to the audit-work in the Map in AuditProcessManager it is supposed to cleanup during afterCompletion, but it will never be called.

This issues has been found using an older version of Hibernate and Narayana, but I have tested this in a small example application with the latest version of Hibernate (5.2.15.Final) and the latest version of Narayana (5.8.0.Final) and I can reproduce the issue.

I have created a github-repository where I have pushed my test application: https://github.com/ronaldmik/memoryleak

Any help would be appreciated. If this should (also) be posted in the Jboss/Narayana forums, let me know.

Thanks in advance!

Forum post: https://discourse.hibernate.org/t/possible-memory-leak-in-envers-due-to-narayana-transaction-reaper/463