[wildfly-dev] EJB Transactions Graceful Shutdown
Flavia Rainone
frainone at redhat.com
Wed Dec 7 13:32:05 EST 2016
+1, I like the idea of logging the number of active pending transactions.
On 07-12-2016 12:16, Radim Hatlapatka wrote:
>
> Definitely +1 for prompting in logs what is being happening and why
> shutdown takes longer period of time.
>
> Also the graceful shutdown adds timeout which can be used to limit
> maximum number of seconds for finishing shutdown gracefully, after
> that time it should stop waiting for remaining transactions/requests
> to finish.
>
> Note current default behaviour is to stop the server without using the
> graceful shutdown (timeout needs to be specified for actually using
> the graceful shutdown logic), for details see description of shutdown
> command timeout value.
>
> Cheers.
>
> Radim
>
>
> On 12/07/2016 02:49 PM, Jim Tyrrell wrote:
>> Team,
>>
>> How might we process the shutting down with a transaction log
>> message/s saying it is shutting down or saying something like:
>>
>> Started shutting down
>> 100 pending transactions
>> 10 shut down 90 pending
>> 20 shut down 80 pending
>>
>> or something like that.
>>
>> Feedback to the user in this sounds important to ensure people don’t
>> kill -9 the process.
>>
>> Jim Tyrrell
>> Principal JBoss Solutions Architect
>> Public Sector Middleware Advocate
>> 720-839-2251 mobile
>>
>>> On Dec 6, 2016, at 8:50 AM, Andrig Miller <anmiller at redhat.com
>>> <mailto:anmiller at redhat.com>> wrote:
>>>
>>>
>>>
>>> On Mon, Dec 5, 2016 at 9:54 PM, Stuart
>>> Douglas<stuart.w.douglas at gmail.com
>>> <mailto:stuart.w.douglas at gmail.com>>wrote:
>>>
>>>
>>>
>>> On Tue, Dec 6, 2016 at 10:12 AM, Andrig
>>> Miller<anmiller at redhat.com <mailto:anmiller at redhat.com>>wrote:
>>>
>>> I'm just wondering if we are making this graceful shutdown
>>> more complicated than necessary.
>>>
>>> Why wouldn't we just cancel and force a rollback on any
>>> active transactions when shutting down? Having experienced
>>> what a graceful shutdown can look like with a different
>>> architecture (BEA Tuxedo), I can tell you that it can take a
>>> very long time for the server to get to the point of
>>> shutting down, and appear to be hung by the administrator,
>>> depending on what was going on at the time the command was
>>> entered.
>>>
>>>
>>>
>>> I think at the very least this has to be optional, so we can
>>> still have the old non transnational behavior (i.e. wait for
>>> requests to finish rather than transactions).
>>>
>>>
>>> My personal opinion is that it should be the default behavior as
>>> well. Most of the time, when our administrators would try to
>>> gracefully shutdown Tuxedo, then ended up killing it with a kill -9,
>>> because it took too long. Of course, that caused all kinds of
>>> consistency problems and transaction recovery on the next startup.
>>> Generally, it was a mess.
>>>
>>> Andy
>>>
>>>
>>>
>>> Stuart
>>>
>>>
>>> We used to get administrators killing Tuxedo while it was
>>> "gracefully" shutting down, and messing lots of stuff up.
>>>
>>> Andy
>>>
>>> On Mon, Dec 5, 2016 at 12:34 PM, Flavia
>>> Rainone<frainone at redhat.com <mailto:frainone at redhat.com>>wrote:
>>>
>>> I think we can keep open transaction tracking only
>>> inside transactions subsystem while we are not shutting
>>> down, and then we can enroll for notification of open
>>> active transactions only on suspend if needed... IMHO
>>> that's as clean as we can get regarding shutdown code
>>> when the server is in running state.
>>>
>>> I would go with some sort of ActiveTransactionListener,
>>> that would be notified of no more active transactions
>>> only if the listener is set?
>>>
>>> Something along the lines below at ejb side:
>>>
>>> ServerActivityCallback callback = null;
>>>
>>> public void suspend(ServerActivityCallback callback) {
>>> if ( transactionSubsystem.getActiveTransactions() > 0) {
>>> transactionSubsystem.setActiveTransactionsListener(this);
>>> }
>>> else {
>>> callback.done(); // done suspending
>>> }
>>> }
>>>
>>> // listener method
>>> public void noMoreActiveTransactions() {
>>> callback.done(); // done suspending
>>> // then we let control point notify clients that this
>>> node is no longer available
>>> ...
>>> }
>>>
>>> At transactions side:
>>> ActiveTransactionListener listener = null;
>>>
>>> private void incrementTxnCount() {
>>> ...
>>> }
>>>
>>> private void decrementTxnCount() {
>>> if (txnCountUpdater.decrementAndGet() == 0 && listener
>>> != null)
>>> listener.noMoreActiveTransactions();
>>> }
>>>
>>> public int getActiveTransactions() {
>>> return txnCountUpdater.get();
>>>
>>> }
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On 04-12-2016 20:39, Stuart Douglas wrote:
>>>> On Sat, Dec 3, 2016 at 3:40 AM, Flavia Rainone<frainone at redhat.com> <mailto:frainone at redhat.com> wrote:
>>>>> Hi,
>>>>>
>>>>> I'm creating this thread to discuss the remaining details of graceful
>>>>> shutdown for ejb transactions.
>>>>>
>>>>> This is more or less what I've done so far:
>>>>>
>>>>> https://github.com/fl4via/wildfly/commit/7017146522af9a979a8a8e0c92039e6a5fb18760
>>>>> <https://github.com/fl4via/wildfly/commit/7017146522af9a979a8a8e0c92039e6a5fb18760>
>>>>>
>>>>> While discussing this in the hip chat yesterday, Stuart mentioned that maybe
>>>>> we could have the transactions subsystem responsible for keeping track of
>>>>> how many active transactions we have, instead of putting that code in
>>>>> EjbRemoteTransactionsRepository.
>>>>>
>>>>> Stuart, does that include having the suspend callback being done at
>>>>> transactions subsystem as well? I'm thinking maybe not, because there are
>>>>> two points in the ejb subsystem we need to know if transactions suspension
>>>>> is over:
>>>>>
>>>> No, that still has to be handled at an EJB subsystem level.
>>>> Conceptually this is similar to what was done for the XTS subsytem, so
>>>> it should probably use a similar design. Ideally while the server is
>>>> in the running state the only graceful related code that is run is the
>>>> control point request tracking, however this may not be possible.
>>>>
>>>> One other thing that came up on our hipchat discussion yesterday is TX
>>>> level graceful shutdown actually has some significant drawbacks, as
>>>> you cannot send out the module unavailability message until all the
>>>> transactions have been closed. This means that while we are waiting
>>>> for transactions to complete the node will still be part of a cluster,
>>>> and clients will send it requests that will be immediately rejected.
>>>>
>>>> Stuart
>>>>
>>>>> - at EjbSuspendInterceptor if it is over, no request is allowed, if it is
>>>>> not over, we need to check if current invocation contains a reference to an
>>>>> active transaction
>>>>>
>>>>> - at some point, we need to let control point notify that the ejb module is
>>>>> not longer available to ejb client after transaction suspension is over,
>>>>> i.e., we need to do that when suspend has been requested and there are no
>>>>> remaining active transactions available.
>>>>>
>>>>> On the other hand, it is hard to draw the line between what should be in the
>>>>> transactions subsystem and what shouldn't. If the callback is done at
>>>>> transactions subsystem, we need a way of having ejb3 notified that it is
>>>>> done. If it is not done at transactions subsystem, ejb3 has to be notified
>>>>> of the active transactions going to zero, which seems a lot of overhead, so
>>>>> from this point of view maybe the callback should be in the transactions
>>>>> system after all.
>>>>>
>>>>> Stuart and Gytis, any thoughts?
>>>>>
>>>>>
>>>>> --
>>>>> Flavia Rainone
>>>>> Principal Software Engineer
>>>>> JBoss EAP/WildFly Team
>>>>> M: (+55) 11 981-225-466
>>>>>
>>>>> Red Hat.
>>>>> Better technology.
>>>>> Faster innovation.
>>>>> Powered by community collaboration.
>>> --
>>> Flavia Rainone
>>> Principal Software Engineer
>>> JBoss EAP/WildFly Team
>>> M: (+55) 11 981-225-466
>>>
>>> Red Hat.
>>> Better technology.
>>> Faster innovation.
>>> Powered by community collaboration.
>>>
>>> _______________________________________________
>>> wildfly-dev mailing list wildfly-dev at lists.jboss.org
>>> <mailto:wildfly-dev at lists.jboss.org>
>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>> <https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>>>
>>> --
>>> Andrig (Andy) T. Miller
>>> Global Platform Director, Middleware
>>> Red Hat, Inc.
>>> _______________________________________________ wildfly-dev
>>> mailing list wildfly-dev at lists.jboss.org
>>> <mailto:wildfly-dev at lists.jboss.org>
>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>> <https://lists.jboss.org/mailman/listinfo/wildfly-dev>
>>>
>>> --
>>> Andrig (Andy) T. Miller
>>> Global Platform Director, Middleware
>>> Red Hat, Inc.
>>> _______________________________________________ wildfly-dev mailing
>>> list wildfly-dev at lists.jboss.org
>>> <mailto:wildfly-dev at lists.jboss.org>
>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>
>> _______________________________________________
>> wildfly-dev mailing list
>> wildfly-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
--
Flavia Rainone
Principal Software Engineer
JBoss EAP/WildFly Team
M: (+55) 11 981-225-466
Red Hat.
Better technology.
Faster innovation.
Powered by community collaboration.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/wildfly-dev/attachments/20161207/cbdc4f83/attachment-0001.html
More information about the wildfly-dev
mailing list