One alternative solution, might be to create a top level container level queue that the background thread Synchronization.afterCompletion defers processing to. As soon as the application thread returns control to the top level, the queue is processed in FIFO order.

I had a discussion with Steve about this. Not sure if you guys talked about my suggestion. My suggestion was to simply detect if afterCompletion is running on the thread that created it. If the threads match proceed normally. If, however, they do not match then DO NOTHING (well at first):) Then have the EntityManager, immediately before certain actions such as a flush, detect if the Transaction has been rolled back (using Transaction.getStatus()). If it has been rolled back then at that point of time trigger an exception to the calling application code, with a message that the "Transaction manager has decided to revert the transaction, either due to a timeout or a resource failure. Check your log for details". On Feb 22, 2013, at 8:45 AM, Scott Marlow <smarlow(a)redhat.com&gt; wrote: > Related jira issues are HHH-7910 + AS7-6586. HHH-7910 has been around a > bit longer and contains comments about the current plan to address the > issue in Hibernate. > > As mentioned in HHH-7910, the JTA specification allows > Sychronization.afterCompletion callbacks to occur in a different thread > than the application thread that is using the transaction (typically for > transaction timeout or when a transaction is propagated to a remote thread). > > Since the JPA EntityManager is by design, not thread safe, invoking the > EntityManager.clear() or EntityManager.close() methods from a background > thread while the application thread may be in the middle of an > EntityManager invocation is not the best situation (IMO). > > One improvement that we talked to the JBossTM/TS team about, is adding a > TM policy that arranges for the Synchronization.afterCompletion to > always run in the application thread (some of the IRC discussion is > attached to HHH-7910). The TM team doesn't think that they could do > that for both the TX timeout and tx propagating to a remote thread > (JBossTS) uses. > > One alternative solution, might be to create a top level container level > queue that the background thread Synchronization.afterCompletion defers > processing to. As soon as the application thread returns control to the > top level, the queue is processed in FIFO order. > > *Does AS have any other uses of Synchronization.afterCompletion or > Synchronization.beforeCompletion that expect to run in the application > thread*? > > From the JPA specification about thread unsafety: > > " > An entity manager must not be shared among multiple concurrently > executing threads, as the entity manager and persistence context are not > required to be threadsafe. Entity managers must only be accessed in a > single-threaded manner. > " > _______________________________________________ > jboss-as7-dev mailing list > jboss-as7-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev -- Jason T. Greene JBoss AS Lead / EAP Platform Architect JBoss, a division of Red Hat _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

To be perfectly honest, I don't think the TM is the best place to be tracking timeouts - the front end should always do stuff like that when possible (imo). But that ship has sailed, as they say; I think this proposal sounds workable, if a bit tedious to implement. On 02/22/2013 02:11 PM, Steve Ebersole wrote: > Thats sort of what we are planning on doing. The rub is every library > that interacts with JTA having to do this as well. Thats more what I > think Scott was getting at. > > For clarity your suggestion (method entry checks) is part of our > solution. We will also do method exit checks. Ugh. Lot of work to > handle something the TM is doing > > On Feb 22, 2013 1:53 PM, "Jason Greene" <jason.greene(a)redhat.com > <mailto:jason.greene@redhat.com>> wrote: > > I had a discussion with Steve about this. Not sure if you guys > talked about my suggestion. > > My suggestion was to simply detect if afterCompletion is running on > the thread that created it. If the threads match proceed normally. > If, however, they do not match then DO NOTHING (well at first):) > > Then have the EntityManager, immediately before certain actions such > as a flush, detect if the Transaction has been rolled back (using > Transaction.getStatus()). If it has been rolled back then at that > point of time trigger an exception to the calling application code, > with a message that the "Transaction manager has decided to revert > the transaction, either due to a timeout or a resource failure. > Check your log for details". > > On Feb 22, 2013, at 8:45 AM, Scott Marlow <smarlow(a)redhat.com > <mailto:smarlow@redhat.com>> wrote: > >> Related jira issues are HHH-7910 + AS7-6586. HHH-7910 has been > around a >> bit longer and contains comments about the current plan to > address the >> issue in Hibernate. >> >> As mentioned in HHH-7910, the JTA specification allows >> Sychronization.afterCompletion callbacks to occur in a different > thread >> than the application thread that is using the transaction > (typically for >> transaction timeout or when a transaction is propagated to a > remote thread). >> >> Since the JPA EntityManager is by design, not thread safe, > invoking the >> EntityManager.clear() or EntityManager.close() methods from a > background >> thread while the application thread may be in the middle of an >> EntityManager invocation is not the best situation (IMO). >> >> One improvement that we talked to the JBossTM/TS team about, is > adding a >> TM policy that arranges for the Synchronization.afterCompletion to >> always run in the application thread (some of the IRC discussion is >> attached to HHH-7910). The TM team doesn't think that they could do >> that for both the TX timeout and tx propagating to a remote thread >> (JBossTS) uses. >> >> One alternative solution, might be to create a top level > container level >> queue that the background thread Synchronization.afterCompletion > defers >> processing to. As soon as the application thread returns control > to the >> top level, the queue is processed in FIFO order. >> >> *Does AS have any other uses of Synchronization.afterCompletion or >> Synchronization.beforeCompletion that expect to run in the > application >> thread*? >> >> From the JPA specification about thread unsafety: >> >> " >> An entity manager must not be shared among multiple concurrently >> executing threads, as the entity manager and persistence context > are not >> required to be threadsafe. Entity managers must only be accessed in a >> single-threaded manner. >> " >> _______________________________________________ >> jboss-as7-dev mailing list >> jboss-as7-dev(a)lists.jboss.org <mailto:jboss-as7-dev@lists.jboss.org> >> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev > > -- > Jason T. Greene > JBoss AS Lead / EAP Platform Architect > JBoss, a division of Red Hat > > > _______________________________________________ > jboss-as7-dev mailing list > jboss-as7-dev(a)lists.jboss.org <mailto:jboss-as7-dev@lists.jboss.org> > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev > > > > _______________________________________________ > jboss-as7-dev mailing list > jboss-as7-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev > -- - DML _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

The only other systems I know of that use syncs (off the top of my head - yes being lazy here) are clustering and EJB timers. Both of those control resources that are accessed from multiple threads. On Feb 22, 2013, at 2:11 PM, Steve Ebersole <steve(a)hibernate.org&gt; wrote: > Thats sort of what we are planning on doing. The rub is every library that interacts with JTA having to do this as well. Thats more what I think Scott was getting at. > > For clarity your suggestion (method entry checks) is part of our solution. We will also do method exit checks. Ugh. Lot of work to handle something the TM is doing > > On Feb 22, 2013 1:53 PM, "Jason Greene" <jason.greene(a)redhat.com&gt; wrote: > I had a discussion with Steve about this. Not sure if you guys talked about my suggestion. > > My suggestion was to simply detect if afterCompletion is running on the thread that created it. If the threads match proceed normally. If, however, they do not match then DO NOTHING (well at first):) > > Then have the EntityManager, immediately before certain actions such as a flush, detect if the Transaction has been rolled back (using Transaction.getStatus()). If it has been rolled back then at that point of time trigger an exception to the calling application code, with a message that the "Transaction manager has decided to revert the transaction, either due to a timeout or a resource failure. Check your log for details". > > On Feb 22, 2013, at 8:45 AM, Scott Marlow <smarlow(a)redhat.com&gt; wrote: > >> Related jira issues are HHH-7910 + AS7-6586. HHH-7910 has been around a >> bit longer and contains comments about the current plan to address the >> issue in Hibernate. >> >> As mentioned in HHH-7910, the JTA specification allows >> Sychronization.afterCompletion callbacks to occur in a different thread >> than the application thread that is using the transaction (typically for >> transaction timeout or when a transaction is propagated to a remote thread). >> >> Since the JPA EntityManager is by design, not thread safe, invoking the >> EntityManager.clear() or EntityManager.close() methods from a background >> thread while the application thread may be in the middle of an >> EntityManager invocation is not the best situation (IMO). >> >> One improvement that we talked to the JBossTM/TS team about, is adding a >> TM policy that arranges for the Synchronization.afterCompletion to >> always run in the application thread (some of the IRC discussion is >> attached to HHH-7910). The TM team doesn't think that they could do >> that for both the TX timeout and tx propagating to a remote thread >> (JBossTS) uses. >> >> One alternative solution, might be to create a top level container level >> queue that the background thread Synchronization.afterCompletion defers >> processing to. As soon as the application thread returns control to the >> top level, the queue is processed in FIFO order. >> >> *Does AS have any other uses of Synchronization.afterCompletion or >> Synchronization.beforeCompletion that expect to run in the application >> thread*? >> >> From the JPA specification about thread unsafety: >> >> " >> An entity manager must not be shared among multiple concurrently >> executing threads, as the entity manager and persistence context are not >> required to be threadsafe. Entity managers must only be accessed in a >> single-threaded manner. >> " >> _______________________________________________ >> jboss-as7-dev mailing list >> jboss-as7-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev > > -- > Jason T. Greene > JBoss AS Lead / EAP Platform Architect > JBoss, a division of Red Hat > > > _______________________________________________ > jboss-as7-dev mailing list > jboss-as7-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev -- Jason T. Greene JBoss AS Lead / EAP Platform Architect JBoss, a division of Red Hat

I had a discussion with Steve about this. Not sure if you guys talked about my suggestion.

My suggestion was to simply detect if afterCompletion is running on the thread that created it. If the threads match proceed normally. If, however, they do not match then DO NOTHING (well at first):) Then have the EntityManager, immediately before certain actions such as a flush, detect if the Transaction has been rolled back (using Transaction.getStatus()). If it has been rolled back then at that point of time trigger an exception to the calling application code, with a message that the "Transaction manager has decided to revert the transaction, either due to a timeout or a resource failure. Check your log for details".

On Feb 22, 2013, at 8:45 AM, Scott Marlow <smarlow(a)redhat.com&gt; wrote: > Related jira issues are HHH-7910 + AS7-6586. HHH-7910 has been around a > bit longer and contains comments about the current plan to address the > issue in Hibernate. > > As mentioned in HHH-7910, the JTA specification allows > Sychronization.afterCompletion callbacks to occur in a different thread > than the application thread that is using the transaction (typically for > transaction timeout or when a transaction is propagated to a remote thread). > > Since the JPA EntityManager is by design, not thread safe, invoking the > EntityManager.clear() or EntityManager.close() methods from a background > thread while the application thread may be in the middle of an > EntityManager invocation is not the best situation (IMO). > > One improvement that we talked to the JBossTM/TS team about, is adding a > TM policy that arranges for the Synchronization.afterCompletion to > always run in the application thread (some of the IRC discussion is > attached to HHH-7910). The TM team doesn't think that they could do > that for both the TX timeout and tx propagating to a remote thread > (JBossTS) uses. > > One alternative solution, might be to create a top level container level > queue that the background thread Synchronization.afterCompletion defers > processing to. As soon as the application thread returns control to the > top level, the queue is processed in FIFO order. > > *Does AS have any other uses of Synchronization.afterCompletion or > Synchronization.beforeCompletion that expect to run in the application > thread*? > > From the JPA specification about thread unsafety: > > " > An entity manager must not be shared among multiple concurrently > executing threads, as the entity manager and persistence context are not > required to be threadsafe. Entity managers must only be accessed in a > single-threaded manner. > " > _______________________________________________ > jboss-as7-dev mailing list > jboss-as7-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev -- Jason T. Greene JBoss AS Lead / EAP Platform Architect JBoss, a division of Red Hat

On Feb 22, 2013, at 2:39 PM, Scott Marlow <smarlow(a)redhat.com&gt; wrote: > > On 02/22/2013 02:53 PM, Jason Greene wrote: >> I had a discussion with Steve about this. Not sure if you guys talked about my suggestion. > > I didn't know that but Steve did mention previously that he wanted to get your input. > >> >> My suggestion was to simply detect if afterCompletion is running on the thread that created it. If the threads match proceed normally. If, however, they do not match then DO NOTHING (well at first):) >> >> Then have the EntityManager, immediately before certain actions such as a flush, detect if the Transaction has been rolled back (using Transaction.getStatus()). If it has been rolled back then at that point of time trigger an exception to the calling application code, with a message that the "Transaction manager has decided to revert the transaction, either due to a timeout or a resource failure. Check your log for details". > > Yes, that sounds very familiar! :) > > For HHH-7910, we can instrument enough code to do that and cover some other cases (replacing possible in-flight exception with an exception like your mentioning in the EntityManager method tail). We also need to clear the EntityManager in each method start and tail (after detecting that this case previously occurred). > > For AS7-6586, when the background thread runs the Synchronization.afterCompletion(), the application (transaction origin) thread may still be active or it may of completed its use of the transaction scoped EntityManager. When the Synchronization.afterCompletion is invoked, the AS JPA container code is supposed to call EntityManager.close(), which shouldn't be called while another thread is (possibly) in a EntityManager invocation (more likely in the tx timeout case). Why? There should deb no need for it to close (why I said it should do nothing), because the entity manager is going to rollback anyway (it will either get an exception or monitor the TX status), and so it can self close.

> > I started this thread discussion more to discuss the AS7-6586 case and ask if anyone knows of any other similar problems with other AS code + Synchronization.before/afterCompletion. > > If we were to solve AS7-6586, at a higher level (via a container level queue), we might want to consider solving HHH-7910 the same way (I think). What I was saying in one of my other emails is that I don't see how this can work. The application thread won't release until the state of the TX is completed. So if you have a queue after the TX has rolled back, its not very useful. > If AS7-6586 is the only case we need to solve, we could do something custom (e.g. have every EntityManager call get an exclusive lock or something faster). There should be no need for any additional locks. A simple volatile read is all thats need to determine this (which should be what Transaction.getStatus() already does). > >> >> On Feb 22, 2013, at 8:45 AM, Scott Marlow <smarlow(a)redhat.com&gt; wrote: >> >>> Related jira issues are HHH-7910 + AS7-6586. HHH-7910 has been around a >>> bit longer and contains comments about the current plan to address the >>> issue in Hibernate. >>> >>> As mentioned in HHH-7910, the JTA specification allows >>> Sychronization.afterCompletion callbacks to occur in a different thread >>> than the application thread that is using the transaction (typically for >>> transaction timeout or when a transaction is propagated to a remote thread). >>> >>> Since the JPA EntityManager is by design, not thread safe, invoking the >>> EntityManager.clear() or EntityManager.close() methods from a background >>> thread while the application thread may be in the middle of an >>> EntityManager invocation is not the best situation (IMO). >>> >>> One improvement that we talked to the JBossTM/TS team about, is adding a >>> TM policy that arranges for the Synchronization.afterCompletion to >>> always run in the application thread (some of the IRC discussion is >>> attached to HHH-7910). The TM team doesn't think that they could do >>> that for both the TX timeout and tx propagating to a remote thread >>> (JBossTS) uses. >>> >>> One alternative solution, might be to create a top level container level >>> queue that the background thread Synchronization.afterCompletion defers >>> processing to. As soon as the application thread returns control to the >>> top level, the queue is processed in FIFO order. >>> >>> *Does AS have any other uses of Synchronization.afterCompletion or >>> Synchronization.beforeCompletion that expect to run in the application >>> thread*? >>> >>> From the JPA specification about thread unsafety: >>> >>> " >>> An entity manager must not be shared among multiple concurrently >>> executing threads, as the entity manager and persistence context are not >>> required to be threadsafe. Entity managers must only be accessed in a >>> single-threaded manner. >>> " >>> _______________________________________________ >>> jboss-as7-dev mailing list >>> jboss-as7-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >> >> -- >> Jason T. Greene >> JBoss AS Lead / EAP Platform Architect >> JBoss, a division of Red Hat >> > -- Jason T. Greene JBoss AS Lead / EAP Platform Architect JBoss, a division of Red Hat

On Feb 22, 2013, at 3:13 PM, Scott Marlow <smarlow(a)redhat.com&gt; wrote: > > I'm closing the entity manager because its a JPA spec (7.9.1) requirement for the container to close it: > > "After the JTA transaction has completed (either by transaction commit or rollback), the container closes the entity manager calling EntityManager.close. > " We don't have to interpret it literally. When the transaction rolls back (either via exception or getStatus) take the exact same action that EntityManager.close would take.

> > If I don't close it, the GC might be able to clean it up for us but that depends on whether there are any strong references to it. The transaction will get GCd, so the transaction reference to the EntityManager will disappear. This has potential to create a memory leak. No need for GC. We *know* the outcome of the transaction, and we control the implementation of the entity manager, therefore we can close it in the application thread.

-- Jason T. Greene JBoss AS Lead / EAP Platform Architect JBoss, a division of Red Hat

One improvement that we talked to the JBossTM/TS team about, is adding a TM policy that arranges for the Synchronization.afterCompletion to always run in the application thread (some of the IRC discussion is attached to HHH-7910). The TM team doesn't think that they could do that for both the TX timeout and tx propagating to a remote thread (JBossTS) uses.

One alternative solution, might be to create a top level container level queue that the background thread Synchronization.afterCompletion defers processing to. As soon as the application thread returns control to the top level, the queue is processed in FIFO order.

On 03/15/2013 04:07 PM, Scott Marlow wrote: > [1] contains the updated JPA 2.1 specification requirements for when the > JTA transaction times out from a background thread. > > AS7-6586 [2] will address the [1] requirements for AS8 and will be a > candidate for back-porting as needed. For the container side, knowing > if we are in the "Transaction Reaper Worker N" thread would be helpful > (by checking thread name or via a JBossTM spi). I don't like the idea > of checking the thread name but also don't have the > (isCurrentThreadTheReaper()) spi in hand currently. I created JBTM-1556 for adding a JBossTM/JBossTS way to know if the current thread is the reaper. > > The Hibernate team already has committed the provider side fix [3]. > > [1] > http://java.net/projects/jpa-spec/lists/jsr338-experts/archive/2013-03/me... > > [2] AS7-6586 - "JPA afterCompletion sync may be called from > non-application thread, resolve concurrency concern" > > [3] HHH-7910 - "Transaction timeout can cause non-threadsafe session > access by reaper thread" > > Scott > _______________________________________________ > jboss-as7-dev mailing list > jboss-as7-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev > _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev