[undertow-dev] http/2 server starts dying after rapid refresh
Stuart Douglas
sdouglas at redhat.com
Wed Jun 3 03:36:28 EDT 2015
https://github.com/undertow-io/undertow/commit/4cd914008fc7fddca97af5300523b96bba81d895
Is the one.
Stuart
----- Original Message -----
> From: "Michael Hixson" <michael.hixson at gmail.com>
> To: undertow-dev at lists.jboss.org
> Sent: Wednesday, 3 June, 2015 1:13:24 AM
> Subject: Re: [undertow-dev] http/2 server starts dying after rapid refresh
>
> Was this Jira issue 461? I'm seeing promising results with this
> commit, while if I revert the commit I run into the issue again.
> https://github.com/undertow-io/undertow/commit/8f8e1658c5aface7f65a351e985d40c386a3f99b
>
> If so, thanks!
>
> -Michael
>
> On Fri, May 29, 2015 at 10:26 PM, Michael Hixson
> <michael.hixson at gmail.com> wrote:
> > Also here is a jmap -histo from a little later:
> > https://gist.github.com/michaelhixson/de6ede25fdb79a4f25b4
> >
> > (apologies in advance if gmail sends this email twice like my last one)
> >
> > -Michael
> >
> > On Fri, May 29, 2015 at 10:04 PM, Michael Hixson
> > <michael.hixson at gmail.com> wrote:
> >> Hi Jason,
> >>
> >> I believe the close() call is coming from me declaring the writer in a
> >> try-with-resources statement.
> >>
> >> Here's a jstack from just now, where my CPU is sitting at 96%:
> >> https://gist.github.com/michaelhixson/8a6f2ebf4b167fb1b67e
> >>
> >> -Michael
> >>
> >> On Fri, May 29, 2015 at 9:28 PM, Jason T. Greene
> >> <jason.greene at redhat.com> wrote:
> >>> Hi Michael,
> >>>
> >>> That stack trace seems to show a close() call to the writer in your
> >>> handler, but I don't see that call in the code?
> >>>
> >>> Note that getting I/o exceptions during write and close calls is normal
> >>> after a client abnormally terminates. You typically just drop the
> >>> exception (perhaps trace logging).
> >>>
> >>> Could you get a jstack output under the scenario where you see high CPU
> >>> load?
> >>>
> >>>
> >>>> On May 28, 2015, at 6:19 PM, Michael Hixson <michael.hixson at gmail.com>
> >>>> wrote:
> >>>>
> >>>> Hello,
> >>>>
> >>>> I'm trying out HTTP/2 with undertow-core in my application, and I see
> >>>> that I'm able to get the server into a bad state pretty reliably by
> >>>> refreshing quickly in my browser. It stops responding to requests and
> >>>> the CPU usage of the Java process starts to climb towards 100%.
> >>>>
> >>>> I was able to replicate this in the HTTP/2 example in the latest from
> >>>> the master branch, by changing the main HttpHandler. This is what I
> >>>> put in Http2Server#main(String[]):
> >>>>
> >>>> Undertow server = Undertow.builder()
> >>>> .setServerOption(UndertowOptions.ENABLE_HTTP2, true)
> >>>> .setServerOption(UndertowOptions.ENABLE_SPDY, true)
> >>>> .addHttpListener(8080, bindAddress)
> >>>> .addHttpsListener(8443, bindAddress, sslContext)
> >>>> .setHandler(new HttpHandler() {
> >>>> @Override
> >>>> public void handleRequest(HttpServerExchange exchange) {
> >>>> if (exchange.isInIoThread()) {
> >>>> exchange.dispatch(this);
> >>>> return;
> >>>> }
> >>>> exchange.startBlocking();
> >>>> exchange.getResponseHeaders().put(
> >>>> new HttpString("Content-Type"), "text/html");
> >>>> try (BufferedWriter writer = new BufferedWriter(
> >>>> new OutputStreamWriter(
> >>>> exchange.getOutputStream(),
> >>>> StandardCharsets.UTF_8))) {
> >>>> writer.append("<!DOCTYPE html><html>\n");
> >>>> for (int i = 0; i < 100; i++) {
> >>>> writer.append(
> >>>> "<link rel=\"stylesheet\" href=\"/" + i +
> >>>> ".css\">\n");
> >>>> }
> >>>> writer.append("hello");
> >>>> } catch (IOException e) {
> >>>> e.printStackTrace();
> >>>> throw new RuntimeException(e);
> >>>> }
> >>>> }
> >>>> }).build();
> >>>>
> >>>> Then I go to https://localhost:8443/ in a browser and hold down F5 for
> >>>> a bit. I start seeing errors like this:
> >>>>
> >>>> java.nio.channels.ClosedChannelException
> >>>> at
> >>>> io.undertow.server.protocol.framed.AbstractFramedStreamSinkChannel.write(AbstractFramedStreamSinkChannel.java:396)
> >>>> at org.xnio.channels.Channels.writeFinalBasic(Channels.java:961)
> >>>> at
> >>>> io.undertow.server.protocol.framed.AbstractFramedStreamSinkChannel.writeFinal(AbstractFramedStreamSinkChannel.java:420)
> >>>> at
> >>>> org.xnio.conduits.StreamSinkChannelWrappingConduit.writeFinal(StreamSinkChannelWrappingConduit.java:66)
> >>>> at
> >>>> org.xnio.conduits.ConduitStreamSinkChannel.writeFinal(ConduitStreamSinkChannel.java:104)
> >>>> at
> >>>> io.undertow.channels.DetachableStreamSinkChannel.writeFinal(DetachableStreamSinkChannel.java:195)
> >>>> at
> >>>> io.undertow.server.HttpServerExchange$WriteDispatchChannel.writeFinal(HttpServerExchange.java:1882)
> >>>> at
> >>>> io.undertow.io.UndertowOutputStream.writeBufferBlocking(UndertowOutputStream.java:288)
> >>>> at
> >>>> io.undertow.io.UndertowOutputStream.close(UndertowOutputStream.java:331)
> >>>> at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:320)
> >>>> at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:149)
> >>>> at java.io.OutputStreamWriter.close(OutputStreamWriter.java:233)
> >>>> at java.io.BufferedWriter.close(BufferedWriter.java:266)
> >>>> at
> >>>> io.undertow.examples.http2.Http2Server$1.handleRequest(Http2Server.java:94)
> >>>> at
> >>>> io.undertow.server.Connectors.executeRootHandler(Connectors.java:199)
> >>>> at
> >>>> io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:774)
> >>>> at
> >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >>>> at
> >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>> at java.lang.Thread.run(Thread.java:745)
> >>>>
> >>>> In case it helps, I saw a different error in my real (not undertow
> >>>> example) application when I started seeing the same effects visibly
> >>>> (site down and CPU climbing to 100%). Maybe it is the same root cause
> >>>> but just being caught/logged differently:
> >>>>
> >>>> ERROR [2015-05-28 14:23:59,804] org.xnio.nio: XNIO000011: Task
> >>>> io.undertow.server.protocol.framed.AbstractFramedChannel$2 at 308b5f1f
> >>>> failed with an exception
> >>>> java.lang.IllegalStateException: null
> >>>> at
> >>>> io.undertow.server.protocol.framed.AbstractFramedStreamSinkChannel.getBuffer(AbstractFramedStreamSinkChannel.java:523)
> >>>> ~[undertow-core-1.3.0.Beta1-SNAPSHOT.jar:1.3.0.Beta1-SNAPSHOT]
> >>>> at
> >>>> io.undertow.server.protocol.framed.AbstractFramedChannel.flushSenders(AbstractFramedChannel.java:496)
> >>>> ~[undertow-core-1.3.0.Beta1-SNAPSHOT.jar:1.3.0.Beta1-SNAPSHOT]
> >>>> at
> >>>> io.undertow.server.protocol.framed.AbstractFramedChannel$2.run(AbstractFramedChannel.java:581)
> >>>> ~[undertow-core-1.3.0.Beta1-SNAPSHOT.jar:1.3.0.Beta1-SNAPSHOT]
> >>>> at org.xnio.nio.WorkerThread.safeRun(WorkerThread.java:560)
> >>>> ~[xnio-nio-3.3.1.Final.jar:3.3.1.Final]
> >>>> at org.xnio.nio.WorkerThread.run(WorkerThread.java:462)
> >>>> ~[xnio-nio-3.3.1.Final.jar:3.3.1.Final]
> >>>>
> >>>> I see the same behavior in my real application with both undertow-core
> >>>> 1.2.6.Final and 1.3.0.Beta1-SNAPSHOT (the latter installed locally
> >>>> from source). I only tried the Http2Server example in the latest
> >>>> version though.
> >>>>
> >>>> -Michael
> >>>> _______________________________________________
> >>>> undertow-dev mailing list
> >>>> undertow-dev at lists.jboss.org
> >>>> https://lists.jboss.org/mailman/listinfo/undertow-dev
> _______________________________________________
> undertow-dev mailing list
> undertow-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/undertow-dev
>
More information about the undertow-dev
mailing list