[undertow-dev] occasional 502 from Apache HTTP Proxy
Edgar Espina
espina.edgar at gmail.com
Tue Jan 20 10:03:10 EST 2015
seems to be some sort of timeout, because everything work as expected, but
then after waiting for a while and got again the 502.
also, I did set the content-length for static resources and call
.endExchange after a 304 response, set a backlog of 1000 and 10000 too. But
it didn't help.
On Mon, Jan 19, 2015 at 9:57 AM, Tomaž Cerar <tomaz.cerar at gmail.com> wrote:
> Maybe related to
> http://stackoverflow.com/questions/169453/bad-gateway-502-error-with-apache-mod-proxy-and-tomcat
>
> or
> http://qnalist.com/questions/4502641/users-httpd-mod-proxy-ignores-incompleteness-of-chunked-coding-response-from-backend
>
> On Mon, Jan 19, 2015 at 1:33 PM, Edgar Espina <espina.edgar at gmail.com>
> wrote:
>
>> Sure, it is 2.2.15.
>>
>>
>>
>> On Mon, Jan 19, 2015 at 9:29 AM, Tomaž Cerar <tomaz.cerar at gmail.com>
>> wrote:
>>
>>> Edgar,
>>>
>>> what is the version of Apache and mod_proxy module you are using.
>>> maybe it will be easier to reproduce with exact version you have..
>>>
>>> --
>>> tomaz
>>>
>>> On Mon, Jan 19, 2015 at 1:15 PM, Edgar Espina <espina.edgar at gmail.com>
>>> wrote:
>>>
>>>> The source code can be found here:
>>>> https://github.com/jooby-project/jooby
>>>>
>>>> Undertow related classes are here:
>>>> https://github.com/jooby-project/jooby/tree/master/jooby/src/main/java/org/jooby/internal/undertow
>>>> .
>>>> Server is built here:
>>>> https://github.com/jooby-project/jooby/blob/master/jooby/src/main/java/org/jooby/internal/undertow/UndertowServer.java
>>>> Response is sent here:
>>>> https://github.com/jooby-project/jooby/blob/master/jooby/src/main/java/org/jooby/internal/undertow/UndertowResponse.java#L336-382
>>>>
>>>> My microweb-framework was built on top of Jetty, so for now I used the
>>>> blocking API and follow more or less what we usually do with Servlets
>>>> (acquire an outstream).
>>>>
>>>> I will review what I'm doing and try to figure it out what is going on
>>>> too.
>>>>
>>>> Thanks for your help, Stuart.
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Jan 19, 2015 at 2:12 AM, Stuart Douglas <sdouglas at redhat.com>
>>>> wrote:
>>>>
>>>>> My best guess as to what is happening is that Undertow closing a
>>>>> connection after a request is done for some reason, and the apache attempts
>>>>> to re-use this connection without realising that it is dead.
>>>>>
>>>>> In general this should not happen, Undertow should only forcibly close
>>>>> a connection if it knows that it is broken (e.g. a content length is set
>>>>> and the full amount of content is not written). For a normal graceful close
>>>>> Undertow should be sending Connection:close headers.
>>>>>
>>>>> I am going to investigate some more, and see if I can figure out what
>>>>> is going on. It seems unlikely but is there any chance your code forcibly
>>>>> closes the ServerConnection (HttpServerExchange.getConnection()) because
>>>>> that could potentially cause this issue. Setting the exchange to
>>>>> non-persistent after headers have been sent could also cause it.
>>>>>
>>>>> Stuart
>>>>>
>>>>> ----- Original Message -----
>>>>> > From: "Edgar Espina" <espina.edgar at gmail.com>
>>>>> > To: "Stuart Douglas" <sdouglas at redhat.com>
>>>>> > Cc: undertow-dev at lists.jboss.org
>>>>> > Sent: Monday, 19 January, 2015 2:53:06 PM
>>>>> > Subject: Re: [undertow-dev] occasional 502 from Apache HTTP Proxy
>>>>> >
>>>>> > Done, but makes no difference. Still got the 502 random errors :S
>>>>> >
>>>>> > On Sun, Jan 18, 2015 at 11:13 PM, Stuart Douglas <
>>>>> sdouglas at redhat.com>
>>>>> > wrote:
>>>>> >
>>>>> > > I had a play around with apache locally, and I could reproduce
>>>>> this in
>>>>> > > some circumstances, and it looks like we are not setting a high
>>>>> enough
>>>>> > > backlog by default.
>>>>> > >
>>>>> > > Can you try adding:
>>>>> > >
>>>>> > > undertow.setSocketOption(Options.BACKLOG, 1000)
>>>>> > >
>>>>> > > To your Undertow builder? I am going to increase this in our
>>>>> default
>>>>> > > config upstream.
>>>>> > >
>>>>> > > Hopefully this is the issue that you are running into.
>>>>> > >
>>>>> > > Stuart
>>>>> > >
>>>>> > >
>>>>> > > ----- Original Message -----
>>>>> > > > From: "Edgar Espina" <espina.edgar at gmail.com>
>>>>> > > > To: "Stuart Douglas" <sdouglas at redhat.com>
>>>>> > > > Cc: undertow-dev at lists.jboss.org
>>>>> > > > Sent: Monday, 19 January, 2015 12:10:16 PM
>>>>> > > > Subject: Re: [undertow-dev] occasional 502 from Apache HTTP Proxy
>>>>> > > >
>>>>> > > > It is a default undertow instance with a HTTP listener, just set
>>>>> work
>>>>> > > > threads to 200. Is there a default idle timeout? I can' tell
>>>>> from javadoc
>>>>> > > > and looking at the code.
>>>>> > > >
>>>>> > > >
>>>>> > > > On Sun, Jan 18, 2015 at 9:59 PM, Stuart Douglas <
>>>>> sdouglas at redhat.com>
>>>>> > > wrote:
>>>>> > > >
>>>>> > > > >
>>>>> > > > >
>>>>> > > > > Edgar Espina wrote:
>>>>> > > > >
>>>>> > > > >> I'm not, channel does it.
>>>>> > > > >>
>>>>> > > > >> Problem is present on both. For example, home page has a
>>>>> > > content-length
>>>>> > > > >> header but jquery.js use chunked.
>>>>> > > > >>
>>>>> > > > >
>>>>> > > > > In general if you are serving static resources you are better
>>>>> off
>>>>> > > setting
>>>>> > > > > the content length (slightly more efficient, and the browser
>>>>> can
>>>>> > > display
>>>>> > > > > progress on downloads).
>>>>> > > > >
>>>>> > > > >
>>>>> > > > >> Do I have to call .endExchange? or closing the output stream
>>>>> is
>>>>> > > enough?
>>>>> > > > >>
>>>>> > > > >
>>>>> > > > > Closing the output stream is enough. endExchange is
>>>>> automatically
>>>>> > > called
>>>>> > > > > once the call stack returns anyway, unless you have dispatched
>>>>> the
>>>>> > > exchange
>>>>> > > > > or started async IO.
>>>>> > > > >
>>>>> > > > > The 'connection reset by peer' error in the apache log while
>>>>> reading
>>>>> > > the
>>>>> > > > > status like kinda indicates that the request does not even get
>>>>> to this
>>>>> > > > > point anyway, and the underlying TCP connection is probably
>>>>> being torn
>>>>> > > down
>>>>> > > > > somehow.
>>>>> > > > >
>>>>> > > > > Do you have any kind of timeouts set? If you have an idle
>>>>> timeout set
>>>>> > > on
>>>>> > > > > the listener there is a race where Undertow can close the
>>>>> channel due
>>>>> > > to
>>>>> > > > > inactivity just as the front end starts to send a request.
>>>>> > > > >
>>>>> > > > > Stuart
>>>>> > > > >
>>>>> > > > >
>>>>> > > > >> Thanks
>>>>> > > > >>
>>>>> > > > >> On Sun, Jan 18, 2015 at 9:48 PM, Stuart Douglas <
>>>>> sdouglas at redhat.com
>>>>> > > > >> <mailto:sdouglas at redhat.com>> wrote:
>>>>> > > > >>
>>>>> > > > >> Another question, are you setting a content length on the
>>>>> > > responses?
>>>>> > > > >> If not the channel will automatically set one if the
>>>>> response fits
>>>>> > > > >> inside a buffer, otherwise chunked encoding will be used.
>>>>> > > > >>
>>>>> > > > >> It might be helpful to know if this only happens on
>>>>> chunked, fixed
>>>>> > > > >> length or both.
>>>>> > > > >>
>>>>> > > > >> Stuart
>>>>> > > > >>
>>>>> > > > >> Edgar Espina wrote:
>>>>> > > > >>
>>>>> > > > >> latest: 1.2.0.Beta8
>>>>> > > > >>
>>>>> > > > >> On Sun, Jan 18, 2015 at 9:42 PM, Stuart Douglas
>>>>> > > > >> <sdouglas at redhat.com <mailto:sdouglas at redhat.com>
>>>>> > > > >> <mailto:sdouglas at redhat.com <mailto:
>>>>> sdouglas at redhat.com>>>
>>>>> > > wrote:
>>>>> > > > >>
>>>>> > > > >> Also what version of Undertow are you using?
>>>>> > > > >>
>>>>> > > > >> Stuart
>>>>> > > > >>
>>>>> > > > >> ----- Original Message -----
>>>>> > > > >> > From: "Edgar Espina" <espina.edgar at gmail.com
>>>>> > > > >> <mailto:espina.edgar at gmail.com>
>>>>> > > > >> <mailto:espina.edgar at gmail.com <mailto:
>>>>> espina.edgar at gmail.com
>>>>> > > > >> >__>>
>>>>> > > > >> > To: "Stuart Douglas" <sdouglas at redhat.com
>>>>> > > > >> <mailto:sdouglas at redhat.com>
>>>>> > > > >> <mailto:sdouglas at redhat.com <mailto:
>>>>> sdouglas at redhat.com>>>
>>>>> > > > >> > Cc: undertow-dev at lists.jboss.org
>>>>> > > > >> <mailto:undertow-dev at lists.jboss.org>
>>>>> > > > >> <mailto:undertow-dev at lists.__jboss.org
>>>>> > > > >> <mailto:undertow-dev at lists.jboss.org>>
>>>>> > > > >> > Sent: Monday, 19 January, 2015 11:13:21 AM
>>>>> > > > >> > Subject: Re: [undertow-dev] occasional 502 from
>>>>> Apache HTTP
>>>>> > > > >> Proxy
>>>>> > > > >> >
>>>>> > > > >> > found this in apache:
>>>>> > > > >> >
>>>>> > > > >> > (104) Connection reset by peer: proxy: error
>>>>> reading status
>>>>> > > > >> line from
>>>>> > > > >> > remote server
>>>>> > > > >> >
>>>>> > > > >> > but nothing in undertow.
>>>>> > > > >> >
>>>>> > > > >> > got 502 on HTTP GET. Sometimes while calling the
>>>>> home page
>>>>> > > / or
>>>>> > > > >> when page
>>>>> > > > >> > loads OK, got 502 on page resources (js, css,
>>>>> images).
>>>>> > > > >> >
>>>>> > > > >> > I call startBlocking and use the outputstream to
>>>>> write the
>>>>> > > > >> response, when
>>>>> > > > >> > everything has been written I call the
>>>>> "outputstream.close"
>>>>> > > > >> method. Do I
>>>>> > > > >> > need to call exchange.endExchange too?
>>>>> > > > >> >
>>>>> > > > >> > Thanks
>>>>> > > > >> >
>>>>> > > > >> > On Sun, Jan 18, 2015 at 8:52 PM, Stuart Douglas
>>>>> > > > >> <sdouglas at redhat.com <mailto:sdouglas at redhat.com>
>>>>> > > > >> <mailto:sdouglas at redhat.com <mailto:
>>>>> sdouglas at redhat.com>>>
>>>>> > > wrote:
>>>>> > > > >> >
>>>>> > > > >> > > Is there any info in the log? Or is there any
>>>>> specific
>>>>> > > type
>>>>> > > > >> of
>>>>> > > > >> request
>>>>> > > > >> > > that causes this?
>>>>> > > > >> > >
>>>>> > > > >> > > Stuart
>>>>> > > > >> > >
>>>>> > > > >> > >
>>>>> > > > >> > > ----- Original Message -----
>>>>> > > > >> > > > From: "Edgar Espina" <espina.edgar at gmail.com
>>>>> > > > >> <mailto:espina.edgar at gmail.com>
>>>>> > > > >> <mailto:espina.edgar at gmail.com <mailto:
>>>>> espina.edgar at gmail.com
>>>>> > > > >> >__>>
>>>>> > > > >> > > > To: undertow-dev at lists.jboss.org
>>>>> > > > >> <mailto:undertow-dev at lists.jboss.org>
>>>>> > > > >> <mailto:undertow-dev at lists.__jboss.org
>>>>> > > > >> <mailto:undertow-dev at lists.jboss.org>>
>>>>> > > > >> > > > Sent: Monday, 19 January, 2015 9:42:19 AM
>>>>> > > > >> > > > Subject: [undertow-dev] occasional 502 from
>>>>> Apache HTTP
>>>>> > > > >> Proxy
>>>>> > > > >> > > >
>>>>> > > > >> > > > Hi,
>>>>> > > > >> > > >
>>>>> > > > >> > > > I've an Undertow application behind apache
>>>>> reverse
>>>>> > > proxy,
>>>>> > > > >> trying to load
>>>>> > > > >> > > a
>>>>> > > > >> > > > page displays error 502 proxy error.
>>>>> > > > >> > > >
>>>>> > > > >> > > > Still couldn't find why so I wonder if any of
>>>>> you find
>>>>> > > a
>>>>> > > > >> similar problem
>>>>> > > > >> > > with
>>>>> > > > >> > > > Undertow and Apache.
>>>>> > > > >> > > >
>>>>> > > > >> > > > Please note this is our first app on top of
>>>>> Undertow,
>>>>> > > > >> existing apps
>>>>> > > > >> > > running
>>>>> > > > >> > > > on Tomcat/Jetty are OK.
>>>>> > > > >> > > >
>>>>> > > > >> > > > Appreciate any help.
>>>>> > > > >> > > >
>>>>> > > > >> > > > Thanks
>>>>> > > > >> > > >
>>>>> > > > >> > > > --
>>>>> > > > >> > > > edgar
>>>>> > > > >> > > >
>>>>> > > > >> > > >
>>>>> _________________________________________________
>>>>> > > > >> > > > undertow-dev mailing list
>>>>> > > > >> > > > undertow-dev at lists.jboss.org
>>>>> > > > >> <mailto:undertow-dev at lists.jboss.org>
>>>>> > > > >> <mailto:undertow-dev at lists.__jboss.org
>>>>> > > > >> <mailto:undertow-dev at lists.jboss.org>>
>>>>> > > > >> > > >
>>>>> > > https://lists.jboss.org/__mailman/listinfo/undertow-dev
>>>>> > > > >> <
>>>>> https://lists.jboss.org/mailman/listinfo/undertow-dev>
>>>>> > > > >> > >
>>>>> > > > >> >
>>>>> > > > >> >
>>>>> > > > >> >
>>>>> > > > >> > --
>>>>> > > > >> > edgar
>>>>> > > > >> >
>>>>> > > > >>
>>>>> > > > >>
>>>>> > > > >>
>>>>> > > > >>
>>>>> > > > >> --
>>>>> > > > >> edgar
>>>>> > > > >>
>>>>> > > > >>
>>>>> > > > >>
>>>>> > > > >>
>>>>> > > > >> --
>>>>> > > > >> edgar
>>>>> > > > >>
>>>>> > > > >
>>>>> > > >
>>>>> > > >
>>>>> > > > --
>>>>> > > > edgar
>>>>> > > >
>>>>> > >
>>>>> >
>>>>> >
>>>>> >
>>>>> > --
>>>>> > edgar
>>>>> >
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> edgar
>>>>
>>>> _______________________________________________
>>>> undertow-dev mailing list
>>>> undertow-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/undertow-dev
>>>>
>>>
>>>
>>
>>
>> --
>> edgar
>>
>
>
--
edgar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/undertow-dev/attachments/20150120/44174afd/attachment-0001.html
More information about the undertow-dev
mailing list