Wanted to document some thoughts (mostly David's) on overall handling of graceful shutdown in AS 7.
The first core requirement is users need to be able to ask the server to shut down gracefully. Shutting down gracefully means allowing ongoing work, including long-running tasks, to complete before shutting down or, if possible, shifting the long running work to another server and cleanly telling the client of the change. The second core requirement is allowing the user to provide an overall timeout for how long to allow ongoing work to complete before accepting possible disruption to that work and pushing forward with the shutdown.
The individual services within the AS are the ones who understand what the ongoing tasks are, how to monitor them for completeness, how to push them to other servers and how to terminate them "gracelessly" if needed.
The overall AS knows how long it's been since graceful shutdown was requested and thus whether a timeout has been exceeded.
The gist of our thoughts on this is the normal behaviour of any AS subsystem should be to shut down gracefully. That is, the stop() method of the services that comprise the subsystem should understand what the ongoing tasks are, monitor them for completeness, try if possible to push them to other servers, and only shut down when ongoing work is complete. The stop() method should allow this to take as long as necessary.
However, the services should also provide a hook to allow the overall AS to signal that the shutdown timeout has passed and that shutdown needs to now proceed without regard for disruption to ongoing work. That signal would come on a separate thread from the one that invoked the service's stop() method. When that signal is received the thread executing stop() should be informed of that (e.g. via setting flags that stop() periodically checks, interrupting the thread running stop(), etc) and then the stop() execution should proceed down a code path that promptly stops the service.
The hook provided by the AS would consist of a service that any service that supports graceful shutdown could have injected:
public interface GracefulShutdownService() {
void registerGracefulShutdownTerminator(GracefulShutdownTerminator terminator);
}
The service that supports graceful shutdown would register a callback with GracefulShutdownService that the AS would use to signal that the "graceful" aspect of shutdown no longer applies:
public interface GracefulShutdownTerminator() {
void terminateGracefulShutdown();
}
I think it might be better though to just simplify this, skip GracefulShutdownTerminator and have services register a Runnable with GracefulShutdownService.
public interface GracefulShutdownService() {
void registerGracefulShutdownTerminator(Runnable terminator);
}
All this is just about the overall control of the shutdown. The complexity is in the subsystem's graceful shutdown handling. That's where tricky stuff needs to happen, e.g.
1) Identifying existing long running work that extends beyond a single request
2) Preventing external clients creating new long running work that extends beyond a single request
3) Allowing internal clients to continue creating new long running tasks (i.e. trust the internal client to properly do 2) and assume any new work it submits is needed to allow some overall existing work to gracefully complete)
4) Moving long running tasks to other servers or letting them complete
5) Leaving external connectors open until 4) is done but stopping them as soon as possible once externally generated long running work is handled
6) Final shutdown