[jboss-as7-dev] JTA Synchronization.afterCompletion callback can occur in a background thread but JPA EntityManager must be accessed in single-threaded manner...

[Scott Marlow]

On 02/22/2013 03:56 PM, Jason Greene wrote:
>
> On Feb 22, 2013, at 2:39 PM, Scott Marlow <smarlow at redhat.com> wrote:
>
>>
>> On 02/22/2013 02:53 PM, Jason Greene wrote:
>>> I had a discussion with Steve about this. Not sure if you guys talked about my suggestion.
>>
>> I didn't know that but Steve did mention previously that he wanted to get your input.
>>
>>>
>>> My suggestion was to simply detect if afterCompletion is running on the thread that created it. If the threads match proceed normally. If, however, they do not match then DO NOTHING (well at first):)
>>>
>>> Then have the EntityManager, immediately before certain actions such as a flush, detect if the Transaction has been rolled back (using Transaction.getStatus()). If it has been rolled back then at that point of time trigger an exception to the calling application code, with a message that the "Transaction manager has decided to revert the transaction, either due to a timeout or a resource failure. Check your log for details".
>>
>> Yes, that sounds very familiar! :)
>>
>> For HHH-7910, we can instrument enough code to do that and cover some other cases (replacing possible in-flight exception with an exception like your mentioning in the EntityManager method tail).  We also need to clear the EntityManager in each method start and tail (after detecting that this case previously occurred).
>>
>> For AS7-6586, when the background thread runs the Synchronization.afterCompletion(), the application (transaction origin) thread may still be active or it may of completed its use of the transaction scoped EntityManager.  When the Synchronization.afterCompletion is invoked, the AS JPA container code is supposed to call EntityManager.close(), which shouldn't be called while another thread is (possibly) in a EntityManager invocation (more likely in the tx timeout case).
>
> Why? There should deb no need for it to close (why I said it should do nothing), because the entity manager is going to rollback anyway (it will either get an exception or monitor the TX status), and so it can self close.

I'm closing the entity manager because its a JPA spec (7.9.1) 
requirement for the container to close it:

"After the JTA transaction has completed (either by transaction commit 
or rollback), the container closes the entity manager calling 
EntityManager.close.
"

If I don't close it, the GC might be able to clean it up for us but that 
depends on whether there are any strong references to it.  The 
transaction will get GCd, so the transaction reference to the 
EntityManager will disappear.  This has potential to create a memory leak.

>
>>
>> I started this thread discussion more to discuss the AS7-6586 case and ask if anyone knows of any other similar problems with other AS code + Synchronization.before/afterCompletion.
>>
>> If we were to solve AS7-6586, at a higher level (via a container level queue), we might want to consider solving HHH-7910 the same way (I think).
>
> What I was saying in one of my other emails is that I don't see how this can work. The application thread won't release until the state of the TX is completed. So if you have a queue after the TX has rolled back, its not very useful.
>
>> If AS7-6586 is the only case we need to solve, we could do something custom (e.g. have every EntityManager call get an exclusive lock or something faster).
>
> There should be no need for any additional locks. A simple volatile read is all thats need to determine this (which should be what Transaction.getStatus() already does).
>
>
>>
>>>
>>> On Feb 22, 2013, at 8:45 AM, Scott Marlow <smarlow at redhat.com> wrote:
>>>
>>>> Related jira issues are HHH-7910 + AS7-6586.  HHH-7910 has been around a
>>>> bit longer and contains comments about the current plan to address the
>>>> issue in Hibernate.
>>>>
>>>> As mentioned in HHH-7910, the JTA specification allows
>>>> Sychronization.afterCompletion callbacks to occur in a different thread
>>>> than the application thread that is using the transaction (typically for
>>>> transaction timeout or when a transaction is propagated to a remote thread).
>>>>
>>>> Since the JPA EntityManager is by design, not thread safe, invoking the
>>>> EntityManager.clear() or EntityManager.close() methods from a background
>>>> thread while the application thread may be in the middle of an
>>>> EntityManager invocation is not the best situation (IMO).
>>>>
>>>> One improvement that we talked to the JBossTM/TS team about, is adding a
>>>> TM policy that arranges for the Synchronization.afterCompletion to
>>>> always run in the application thread (some of the IRC discussion is
>>>> attached to HHH-7910).  The TM team doesn't think that they could do
>>>> that for both the TX timeout and tx propagating to a remote thread
>>>> (JBossTS) uses.
>>>>
>>>> One alternative solution, might be to create a top level container level
>>>> queue that the background thread Synchronization.afterCompletion defers
>>>> processing to.  As soon as the application thread returns control to the
>>>> top level, the queue is processed in FIFO order.
>>>>
>>>> *Does AS have any other uses of Synchronization.afterCompletion or
>>>> Synchronization.beforeCompletion that expect to run in the application
>>>> thread*?
>>>>
>>>>  From the JPA specification about thread unsafety:
>>>>
>>>> "
>>>> An entity manager must not be shared among multiple concurrently
>>>> executing threads, as the entity manager and persistence context are not
>>>> required to be threadsafe. Entity managers must only be accessed in a
>>>> single-threaded manner.
>>>> "
>>>> _______________________________________________
>>>> jboss-as7-dev mailing list
>>>> jboss-as7-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>
>>> --
>>> Jason T. Greene
>>> JBoss AS Lead / EAP Platform Architect
>>> JBoss, a division of Red Hat
>>>
>>
>
> --
> Jason T. Greene
> JBoss AS Lead / EAP Platform Architect
> JBoss, a division of Red Hat
>