On Sat, Dec 3, 2016 at 3:40 AM, Flavia Rainone <frainone(a)redhat.com&gt; wrote: > Hi, > > I'm creating this thread to discuss the remaining details of graceful > shutdown for ejb transactions. > > This is more or less what I've done so far: > > https://github.com/fl4via/wildfly/commit/7017146522af9a979a8a8e0c92039e6a... > > While discussing this in the hip chat yesterday, Stuart mentioned that maybe > we could have the transactions subsystem responsible for keeping track of > how many active transactions we have, instead of putting that code in > EjbRemoteTransactionsRepository. > > Stuart, does that include having the suspend callback being done at > transactions subsystem as well? I'm thinking maybe not, because there are > two points in the ejb subsystem we need to know if transactions suspension > is over: > No, that still has to be handled at an EJB subsystem level. Conceptually this is similar to what was done for the XTS subsytem, so it should probably use a similar design. Ideally while the server is in the running state the only graceful related code that is run is the control point request tracking, however this may not be possible. One other thing that came up on our hipchat discussion yesterday is TX level graceful shutdown actually has some significant drawbacks, as you cannot send out the module unavailability message until all the transactions have been closed. This means that while we are waiting for transactions to complete the node will still be part of a cluster, and clients will send it requests that will be immediately rejected. Stuart > - at EjbSuspendInterceptor if it is over, no request is allowed, if it is > not over, we need to check if current invocation contains a reference to an > active transaction > > - at some point, we need to let control point notify that the ejb module is > not longer available to ejb client after transaction suspension is over, > i.e., we need to do that when suspend has been requested and there are no > remaining active transactions available. > > On the other hand, it is hard to draw the line between what should be in the > transactions subsystem and what shouldn't. If the callback is done at > transactions subsystem, we need a way of having ejb3 notified that it is > done. If it is not done at transactions subsystem, ejb3 has to be notified > of the active transactions going to zero, which seems a lot of overhead, so > from this point of view maybe the callback should be in the transactions > system after all. > > Stuart and Gytis, any thoughts? > > > -- > Flavia Rainone > Principal Software Engineer > JBoss EAP/WildFly Team > M: (+55) 11 981-225-466 > > Red Hat. > Better technology. > Faster innovation. > Powered by community collaboration.

I'm just wondering if we are making this graceful shutdown more complicated than necessary. Why wouldn't we just cancel and force a rollback on any active transactions when shutting down? Having experienced what a graceful shutdown can look like with a different architecture (BEA Tuxedo), I can tell you that it can take a very long time for the server to get to the point of shutting down, and appear to be hung by the administrator, depending on what was going on at the time the command was entered.

We used to get administrators killing Tuxedo while it was "gracefully" shutting down, and messing lots of stuff up. Andy On Mon, Dec 5, 2016 at 12:34 PM, Flavia Rainone <frainone(a)redhat.com&gt; wrote: > I think we can keep open transaction tracking only inside transactions > subsystem while we are not shutting down, and then we can enroll for > notification of open active transactions only on suspend if needed... IMHO > that's as clean as we can get regarding shutdown code when the server is in > running state. > > I would go with some sort of ActiveTransactionListener, that would be > notified of no more active transactions only if the listener is set? > > Something along the lines below at ejb side: > > ServerActivityCallback callback = null; > > public void suspend(ServerActivityCallback callback) { > if ( transactionSubsystem.getActiveTransactions() > 0) { > transactionSubsystem.setActiveTransactionsListener(this); > } > else { > callback.done(); // done suspending > } > } > > // listener method > public void noMoreActiveTransactions() { > callback.done(); // done suspending > // then we let control point notify clients that this node is no > longer available > ... > } > > At transactions side: > ActiveTransactionListener listener = null; > > private void incrementTxnCount() { > ... > } > > private void decrementTxnCount() { > if (txnCountUpdater.decrementAndGet() == 0 && listener != null) > listener.noMoreActiveTransactions(); > } > > public int getActiveTransactions() { > return txnCountUpdater.get(); > > } > > > > > > > > On 04-12-2016 20:39, Stuart Douglas wrote: > > On Sat, Dec 3, 2016 at 3:40 AM, Flavia Rainone <frainone(a)redhat.com&gt; <frainone(a)redhat.com&gt; wrote: > > Hi, > > I'm creating this thread to discuss the remaining details of graceful > shutdown for ejb transactions. > > This is more or less what I've done so far: > https://github.com/fl4via/wildfly/commit/7017146522af9a979a8a8e0c92039e6a... > > While discussing this in the hip chat yesterday, Stuart mentioned that maybe > we could have the transactions subsystem responsible for keeping track of > how many active transactions we have, instead of putting that code in > EjbRemoteTransactionsRepository. > > Stuart, does that include having the suspend callback being done at > transactions subsystem as well? I'm thinking maybe not, because there are > two points in the ejb subsystem we need to know if transactions suspension > is over: > > > No, that still has to be handled at an EJB subsystem level. > Conceptually this is similar to what was done for the XTS subsytem, so > it should probably use a similar design. Ideally while the server is > in the running state the only graceful related code that is run is the > control point request tracking, however this may not be possible. > > One other thing that came up on our hipchat discussion yesterday is TX > level graceful shutdown actually has some significant drawbacks, as > you cannot send out the module unavailability message until all the > transactions have been closed. This means that while we are waiting > for transactions to complete the node will still be part of a cluster, > and clients will send it requests that will be immediately rejected. > > Stuart > > > - at EjbSuspendInterceptor if it is over, no request is allowed, if it is > not over, we need to check if current invocation contains a reference to an > active transaction > > - at some point, we need to let control point notify that the ejb module is > not longer available to ejb client after transaction suspension is over, > i.e., we need to do that when suspend has been requested and there are no > remaining active transactions available. > > On the other hand, it is hard to draw the line between what should be in the > transactions subsystem and what shouldn't. If the callback is done at > transactions subsystem, we need a way of having ejb3 notified that it is > done. If it is not done at transactions subsystem, ejb3 has to be notified > of the active transactions going to zero, which seems a lot of overhead, so > from this point of view maybe the callback should be in the transactions > system after all. > > Stuart and Gytis, any thoughts? > > > -- > Flavia Rainone > Principal Software Engineer > JBoss EAP/WildFly Team > M: (+55) 11 981-225-466 > > Red Hat. > Better technology. > Faster innovation. > Powered by community collaboration. > > > -- > Flavia Rainone > Principal Software Engineer > JBoss EAP/WildFly Team > M: (+55) 11 981-225-466 > > Red Hat. > Better technology. > Faster innovation. > Powered by community collaboration. > > > _______________________________________________ > wildfly-dev mailing list > wildfly-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/wildfly-dev > -- Andrig (Andy) T. Miller Global Platform Director, Middleware Red Hat, Inc. _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev

On Dec 6, 2016, at 8:50 AM, Andrig Miller <anmiller(a)redhat.com&gt; wrote: On Mon, Dec 5, 2016 at 9:54 PM, Stuart Douglas <stuart.w.douglas(a)gmail.com <mailto:stuart.w.douglas@gmail.com>> wrote: On Tue, Dec 6, 2016 at 10:12 AM, Andrig Miller <anmiller(a)redhat.com <mailto:anmiller@redhat.com>> wrote: I'm just wondering if we are making this graceful shutdown more complicated than necessary. Why wouldn't we just cancel and force a rollback on any active transactions when shutting down? Having experienced what a graceful shutdown can look like with a different architecture (BEA Tuxedo), I can tell you that it can take a very long time for the server to get to the point of shutting down, and appear to be hung by the administrator, depending on what was going on at the time the command was entered. I think at the very least this has to be optional, so we can still have the old non transnational behavior (i.e. wait for requests to finish rather than transactions). ​My personal opinion is that it should be the default behavior as well. Most of the time, when our administrators would try to gracefully shutdown Tuxedo, then ended up killing it with a kill -9, because it took too long. Of course, that caused all kinds of consistency problems and transaction recovery on the next startup. Generally, it was a mess. Andy ​ Stuart We used to get administrators killing Tuxedo while it was "gracefully" shutting down, and messing lots of stuff up. Andy On Mon, Dec 5, 2016 at 12:34 PM, Flavia Rainone <frainone(a)redhat.com <mailto:frainone@redhat.com>> wrote: I think we can keep open transaction tracking only inside transactions subsystem while we are not shutting down, and then we can enroll for notification of open active transactions only on suspend if needed... IMHO that's as clean as we can get regarding shutdown code when the server is in running state. I would go with some sort of ActiveTransactionListener, that would be notified of no more active transactions only if the listener is set? Something along the lines below at ejb side: ServerActivityCallback callback = null; public void suspend(ServerActivityCallback callback) { if ( transactionSubsystem.getActiveTransactions() > 0) { transactionSubsystem.setActiveTransactionsListener(this); } else { callback.done(); // done suspending } } // listener method public void noMoreActiveTransactions() { callback.done(); // done suspending // then we let control point notify clients that this node is no longer available ... } At transactions side: ActiveTransactionListener listener = null; private void incrementTxnCount() { ... } private void decrementTxnCount() { if (txnCountUpdater.decrementAndGet() == 0 && listener != null) listener.noMoreActiveTransactions(); } public int getActiveTransactions() { return txnCountUpdater.get(); } On 04-12-2016 20:39, Stuart Douglas wrote: > On Sat, Dec 3, 2016 at 3:40 AM, Flavia Rainone <frainone(a)redhat.com&gt; <mailto:frainone@redhat.com> wrote: >> Hi, >> >> I'm creating this thread to discuss the remaining details of graceful >> shutdown for ejb transactions. >> >> This is more or less what I've done so far: >> >> https://github.com/fl4via/wildfly/commit/7017146522af9a979a8a8e0c92039e6a... <https://github.com/fl4via/wildfly/commit/7017146522af9a979a8a8e0c92039e6a... >> >> While discussing this in the hip chat yesterday, Stuart mentioned that maybe >> we could have the transactions subsystem responsible for keeping track of >> how many active transactions we have, instead of putting that code in >> EjbRemoteTransactionsRepository. >> >> Stuart, does that include having the suspend callback being done at >> transactions subsystem as well? I'm thinking maybe not, because there are >> two points in the ejb subsystem we need to know if transactions suspension >> is over: >> > No, that still has to be handled at an EJB subsystem level. > Conceptually this is similar to what was done for the XTS subsytem, so > it should probably use a similar design. Ideally while the server is > in the running state the only graceful related code that is run is the > control point request tracking, however this may not be possible. > > One other thing that came up on our hipchat discussion yesterday is TX > level graceful shutdown actually has some significant drawbacks, as > you cannot send out the module unavailability message until all the > transactions have been closed. This means that while we are waiting > for transactions to complete the node will still be part of a cluster, > and clients will send it requests that will be immediately rejected. > > Stuart > >> - at EjbSuspendInterceptor if it is over, no request is allowed, if it is >> not over, we need to check if current invocation contains a reference to an >> active transaction >> >> - at some point, we need to let control point notify that the ejb module is >> not longer available to ejb client after transaction suspension is over, >> i.e., we need to do that when suspend has been requested and there are no >> remaining active transactions available. >> >> On the other hand, it is hard to draw the line between what should be in the >> transactions subsystem and what shouldn't. If the callback is done at >> transactions subsystem, we need a way of having ejb3 notified that it is >> done. If it is not done at transactions subsystem, ejb3 has to be notified >> of the active transactions going to zero, which seems a lot of overhead, so >> from this point of view maybe the callback should be in the transactions >> system after all. >> >> Stuart and Gytis, any thoughts? >> >> >> -- >> Flavia Rainone >> Principal Software Engineer >> JBoss EAP/WildFly Team >> M: (+55) 11 981-225-466 >> >> Red Hat. >> Better technology. >> Faster innovation. >> Powered by community collaboration. -- Flavia Rainone Principal Software Engineer JBoss EAP/WildFly Team M: (+55) 11 981-225-466 Red Hat. Better technology. Faster innovation. Powered by community collaboration. _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org> https://lists.jboss.org/mailman/listinfo/wildfly-dev <https://lists.jboss.org/mailman/listinfo/wildfly-dev> -- Andrig (Andy) T. Miller Global Platform Director, Middleware Red Hat, Inc. _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org> https://lists.jboss.org/mailman/listinfo/wildfly-dev <https://lists.jboss.org/mailman/listinfo/wildfly-dev> -- Andrig (Andy) T. Miller Global Platform Director, Middleware Red Hat, Inc. _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org> https://lists.jboss.org/mailman/listinfo/wildfly-dev <https://lists.jboss.org/mailman/listinfo/wildfly-dev>

Definitely +1 for prompting in logs what is being happening and why shutdown takes longer period of time. Also the graceful shutdown adds timeout which can be used to limit maximum number of seconds for finishing shutdown gracefully, after that time it should stop waiting for remaining transactions/requests to finish. Note current default behaviour is to stop the server without using the graceful shutdown (timeout needs to be specified for actually using the graceful shutdown logic), for details see description of shutdown command timeout value. Cheers. Radim On 12/07/2016 02:49 PM, Jim Tyrrell wrote: > Team, > > How might we process the shutting down with a transaction log > message/s saying it is shutting down or saying something like: > > Started shutting down > 100 pending transactions > 10 shut down 90 pending > 20 shut down 80 pending > > or something like that. > > Feedback to the user in this sounds important to ensure people don’t > kill -9 the process. > > Jim Tyrrell > Principal JBoss Solutions Architect > Public Sector Middleware Advocate > 720-839-2251 mobile > >> On Dec 6, 2016, at 8:50 AM, Andrig Miller <anmiller(a)redhat.com >> <mailto:anmiller@redhat.com>> wrote: >> >> >> >> On Mon, Dec 5, 2016 at 9:54 PM, Stuart >> Douglas&lt;stuart.w.douglas(a)gmail.com >> <mailto:stuart.w.douglas@gmail.com>>wrote: >> >> >> >> On Tue, Dec 6, 2016 at 10:12 AM, Andrig >> Miller&lt;anmiller(a)redhat.com <mailto:anmiller@redhat.com>>wrote: >> >> I'm just wondering if we are making this graceful shutdown >> more complicated than necessary. >> >> Why wouldn't we just cancel and force a rollback on any >> active transactions when shutting down? Having experienced >> what a graceful shutdown can look like with a different >> architecture (BEA Tuxedo), I can tell you that it can take a >> very long time for the server to get to the point of >> shutting down, and appear to be hung by the administrator, >> depending on what was going on at the time the command was >> entered. >> >> >> >> I think at the very least this has to be optional, so we can >> still have the old non transnational behavior (i.e. wait for >> requests to finish rather than transactions). >> >> >> ​My personal opinion is that it should be the default behavior as >> well. Most of the time, when our administrators would try to >> gracefully shutdown Tuxedo, then ended up killing it with a kill -9, >> because it took too long. Of course, that caused all kinds of >> consistency problems and transaction recovery on the next startup. >> Generally, it was a mess. >> >> Andy >> ​ >> >> >> Stuart >> >> >> We used to get administrators killing Tuxedo while it was >> "gracefully" shutting down, and messing lots of stuff up. >> >> Andy >> >> On Mon, Dec 5, 2016 at 12:34 PM, Flavia >> Rainone&lt;frainone(a)redhat.com <mailto:frainone@redhat.com>>wrote: >> >> I think we can keep open transaction tracking only >> inside transactions subsystem while we are not shutting >> down, and then we can enroll for notification of open >> active transactions only on suspend if needed... IMHO >> that's as clean as we can get regarding shutdown code >> when the server is in running state. >> >> I would go with some sort of ActiveTransactionListener, >> that would be notified of no more active transactions >> only if the listener is set? >> >> Something along the lines below at ejb side: >> >> ServerActivityCallback callback = null; >> >> public void suspend(ServerActivityCallback callback) { >> if ( transactionSubsystem.getActiveTransactions() > 0) { >> transactionSubsystem.setActiveTransactionsListener(this); >> } >> else { >> callback.done(); // done suspending >> } >> } >> >> // listener method >> public void noMoreActiveTransactions() { >> callback.done(); // done suspending >> // then we let control point notify clients that this >> node is no longer available >> ... >> } >> >> At transactions side: >> ActiveTransactionListener listener = null; >> >> private void incrementTxnCount() { >> ... >> } >> >> private void decrementTxnCount() { >> if (txnCountUpdater.decrementAndGet() == 0 && listener >> != null) >> listener.noMoreActiveTransactions(); >> } >> >> public int getActiveTransactions() { >> return txnCountUpdater.get(); >> >> } >> >> >> >> >> >> >> >> On 04-12-2016 20:39, Stuart Douglas wrote: >>> On Sat, Dec 3, 2016 at 3:40 AM, Flavia Rainone&lt;frainone(a)redhat.com&gt; <mailto:frainone@redhat.com> wrote: >>>> Hi, >>>> >>>> I'm creating this thread to discuss the remaining details of graceful >>>> shutdown for ejb transactions. >>>> >>>> This is more or less what I've done so far: >>>> >>>> https://github.com/fl4via/wildfly/commit/7017146522af9a979a8a8e0c92039e6a... >>>> <https://github.com/fl4via/wildfly/commit/7017146522af9a979a8a8e0c92039e6a... >>>> >>>> While discussing this in the hip chat yesterday, Stuart mentioned that maybe >>>> we could have the transactions subsystem responsible for keeping track of >>>> how many active transactions we have, instead of putting that code in >>>> EjbRemoteTransactionsRepository. >>>> >>>> Stuart, does that include having the suspend callback being done at >>>> transactions subsystem as well? I'm thinking maybe not, because there are >>>> two points in the ejb subsystem we need to know if transactions suspension >>>> is over: >>>> >>> No, that still has to be handled at an EJB subsystem level. >>> Conceptually this is similar to what was done for the XTS subsytem, so >>> it should probably use a similar design. Ideally while the server is >>> in the running state the only graceful related code that is run is the >>> control point request tracking, however this may not be possible. >>> >>> One other thing that came up on our hipchat discussion yesterday is TX >>> level graceful shutdown actually has some significant drawbacks, as >>> you cannot send out the module unavailability message until all the >>> transactions have been closed. This means that while we are waiting >>> for transactions to complete the node will still be part of a cluster, >>> and clients will send it requests that will be immediately rejected. >>> >>> Stuart >>> >>>> - at EjbSuspendInterceptor if it is over, no request is allowed, if it is >>>> not over, we need to check if current invocation contains a reference to an >>>> active transaction >>>> >>>> - at some point, we need to let control point notify that the ejb module is >>>> not longer available to ejb client after transaction suspension is over, >>>> i.e., we need to do that when suspend has been requested and there are no >>>> remaining active transactions available. >>>> >>>> On the other hand, it is hard to draw the line between what should be in the >>>> transactions subsystem and what shouldn't. If the callback is done at >>>> transactions subsystem, we need a way of having ejb3 notified that it is >>>> done. If it is not done at transactions subsystem, ejb3 has to be notified >>>> of the active transactions going to zero, which seems a lot of overhead, so >>>> from this point of view maybe the callback should be in the transactions >>>> system after all. >>>> >>>> Stuart and Gytis, any thoughts? >>>> >>>> >>>> -- >>>> Flavia Rainone >>>> Principal Software Engineer >>>> JBoss EAP/WildFly Team >>>> M: (+55) 11 981-225-466 >>>> >>>> Red Hat. >>>> Better technology. >>>> Faster innovation. >>>> Powered by community collaboration. >> -- >> Flavia Rainone >> Principal Software Engineer >> JBoss EAP/WildFly Team >> M: (+55) 11 981-225-466 >> >> Red Hat. >> Better technology. >> Faster innovation. >> Powered by community collaboration. >> >> _______________________________________________ >> wildfly-dev mailing list wildfly-dev(a)lists.jboss.org >> <mailto:wildfly-dev@lists.jboss.org> >> https://lists.jboss.org/mailman/listinfo/wildfly-dev >> <https://lists.jboss.org/mailman/listinfo/wildfly-dev> >> >> -- >> Andrig (Andy) T. Miller >> Global Platform Director, Middleware >> Red Hat, Inc. >> _______________________________________________ wildfly-dev >> mailing list wildfly-dev(a)lists.jboss.org >> <mailto:wildfly-dev@lists.jboss.org> >> https://lists.jboss.org/mailman/listinfo/wildfly-dev >> <https://lists.jboss.org/mailman/listinfo/wildfly-dev> >> >> -- >> Andrig (Andy) T. Miller >> Global Platform Director, Middleware >> Red Hat, Inc. >> _______________________________________________ wildfly-dev mailing >> list wildfly-dev(a)lists.jboss.org >> <mailto:wildfly-dev@lists.jboss.org> >> https://lists.jboss.org/mailman/listinfo/wildfly-dev > > _______________________________________________ > wildfly-dev mailing list > wildfly-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/wildfly-dev _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev

(EJB, CDI, or UserTransaction