<div dir="ltr"><div>I think A is likely the simplest option and would "revert" to the behavior users are likely looking for. However I do think B is an interesting idea to be able to opt subsystems unto graceful startup.</div><div><br></div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jul 29, 2019 at 11:16 AM Brian Stansberry <<a href="mailto:brian.stansberry@redhat.com">brian.stansberry@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="font-family:"trebuchet ms",sans-serif">tl;dr question is how to disable 'graceful startup'. Skip the background if you know what that means. :)</div><div style="font-family:"trebuchet ms",sans-serif"><br></div><div style="font-family:"trebuchet ms",sans-serif">Background</div><div style="font-family:"trebuchet ms",sans-serif"><br></div><div style="font-family:"trebuchet ms",sans-serif"><br></div><div style="font-family:"trebuchet ms",sans-serif">Back in 2016 when we added the feature to allow a server to be started in 'suspended' state[1], that work also included a fix for the longstanding bug whereby during server start endpoints would be started and accepting external requests before all the services (e.g. from deployments) would be started. The result would be requests could reach the still-starting server and would fail, e.g. HTTP requests might get a 404 or some variety of 500.</div><div style="font-family:"trebuchet ms",sans-serif"><br></div><div style="font-family:"trebuchet ms",sans-serif">I refer to this bug fix as 'graceful startup'.</div><div style="font-family:"trebuchet ms",sans-serif"><br></div><div style="font-family:"trebuchet ms",sans-serif">Since the fix was introduced we've gotten quite a number of requests to be able to turn off that bug fix, e.g. WFCORE-4291.[2] The scenario is users deploy two apps, where app A during start makes an *external* request to app B and won't complete start until that request is handled. And, the users deploy both A and B in the same server. The server won't allow the external request during boot, so A won't complete start and thus the overall server start hangs until timeout.</div><div style="font-family:"trebuchet ms",sans-serif"><br></div><div style="font-family:"trebuchet ms",sans-serif">I consider this kind of deployment pattern to be a bit of an anti-pattern, but we've gotten enough request to allow it that I'm looking into how to satisfy it. Also, at least for HTTP requests, mod_cluster can be used to prevent external requests reaching a server before things are ready, so if the 'internal' requests were not sent through the LB there's at least one 'error free' use case for this.</div><div style="font-family:"trebuchet ms",sans-serif"><br></div><div style="font-family:"trebuchet ms",sans-serif"><br></div><div style="font-family:"trebuchet ms",sans-serif">The Question</div><div style="font-family:"trebuchet ms",sans-serif"><br></div><div style="font-family:"trebuchet ms",sans-serif">Question is whether to </div><div style="font-family:"trebuchet ms",sans-serif"><br></div><div style="font-family:"trebuchet ms",sans-serif">a) have an overall config switch to disable graceful startup across the board (e.g. a new value for the --start-mode cmd line param passed to standalone.sh)</div><div style="font-family:"trebuchet ms",sans-serif"><br></div><div style="font-family:"trebuchet ms",sans-serif">b) have a subsystem specific setting in the undertow subsystem that configures undertow to allow requests in during boot.</div><div style="font-family:"trebuchet ms",sans-serif"><br></div><div style="font-family:"trebuchet ms",sans-serif">Pros of a)</div><div style="font-family:"trebuchet ms",sans-serif"><br></div><div style="font-family:"trebuchet ms",sans-serif">* Other request patterns are also handled. For example, if our app A was making a remote EJB call to app B, then an undertow only setting won't handle it. If we start adding multiple per-subsystem flags it gets ugly.</div><div style="font-family:"trebuchet ms",sans-serif">* Requests to web applications may still fail, as there are other aspects of the server that are rejecting certain calls until 'graceful startup' is complete. For example ee-concurrency rejects adding scheduled tasks (although that is somewhat a bug[3]), and the XTS integration looks to be designed to reject certain requests.[4] There may be others. If we have make web requests an exceptional pattern, going forward we have to account for that pattern in everything.</div><div style="font-family:"trebuchet ms",sans-serif">* The undertow subsystem itself has two different mechanisms for rejecting requests, with three different call patterns, all of which would need to be adapted.</div><div style="font-family:"trebuchet ms",sans-serif"><br></div><div style="font-family:"trebuchet ms",sans-serif">Pros of b)</div><div style="font-family:"trebuchet ms",sans-serif"><br></div><div style="font-family:"trebuchet ms",sans-serif">* It limits the change to the HTTP use case, the one where we know mod_cluster can be used to prevent external requests.</div><div style="font-family:"trebuchet ms",sans-serif">* I'm not sure about the batch subsystem; i.e. whether it is ok to have batch jobs starting before server start is complete. If the relevant services all have MSC dependencies on everthing they need it should be ok. If not there needs to be some adaptation listen for when the server is fully started, which seems doable.</div></div></blockquote><div><br></div><div>Batch jobs require some other component to start them. For example an EJB, servlet, etc. The one exception would be on a reload where the subsystem itself may restart jobs that were previously running before the reload.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><font face="trebuchet ms, sans-serif">* There may be code that is using this 'graceful startup' as a way not to prevent end user activity, but to prevent premature internal server activity. I think </font>RecoverySuspendController may be an example of this; i.e. preventing start of the tx recovery thread until the server is started. But for this kind of thing there are other, better solutions.</div><div><br></div><div><br></div><div>Right now my preference is a), a global switch. If we're doing this I'm not inclined to limit it to HTTP only as I expect we'll just have to revisit it later. And I think I know how to deal with the more technical pros of the http-only approach.</div><div><br></div><div>WDYT?</div>
<div style="font-family:"trebuchet ms",sans-serif"><br></div><div><div style="font-family:"trebuchet ms",sans-serif"></div><div style="font-family:"trebuchet ms",sans-serif">[1] <a href="https://issues.jboss.org/browse/WFCORE-1829" style="font-family:Arial,Helvetica,sans-serif" target="_blank">https://issues.jboss.org/browse/WFCORE-1829</a></div><div style="font-family:"trebuchet ms",sans-serif">[2] <a href="https://issues.jboss.org/browse/WFCORE-4291" target="_blank">https://issues.jboss.org/browse/WFCORE-4291</a></div><div style="font-family:"trebuchet ms",sans-serif">[3] <a href="https://issues.jboss.org/browse/WFLY-12329" target="_blank">https://issues.jboss.org/browse/WFLY-12329</a></div><div style="font-family:"trebuchet ms",sans-serif">[4] <a href="https://github.com/wildfly/wildfly/pull/9419" style="font-family:Arial,Helvetica,sans-serif" target="_blank">https://github.com/wildfly/wildfly/pull/9419</a></div><div style="font-family:"trebuchet ms",sans-serif"><br></div><div style="font-family:"trebuchet ms",sans-serif"><br></div><br></div><span class="gmail_default" style="font-family:"trebuchet ms",sans-serif">Best regards,</span><div><span class="gmail_default" style="font-family:"trebuchet ms",sans-serif">Brian</span></div><div><span class="gmail_default" style="font-family:"trebuchet ms",sans-serif"><br></span></div></div>
_______________________________________________<br>
wildfly-dev mailing list<br>
<a href="mailto:wildfly-dev@lists.jboss.org" target="_blank">wildfly-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/wildfly-dev" rel="noreferrer" target="_blank">https://lists.jboss.org/mailman/listinfo/wildfly-dev</a></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>James R. Perkins</div><div>JBoss by Red Hat</div></div></div></div></div></div>