From: "Edgar Espina" <espina.edgar(a)gmail.com>
To: "Tomaž Cerar" <tomaz.cerar(a)gmail.com>
Cc: "Stuart Douglas" <sdouglas(a)redhat.com>, undertow-dev(a)lists.jboss.org
Sent: Wednesday, 21 January, 2015 2:03:10 AM
Subject: Re: [undertow-dev] occasional 502 from Apache HTTP Proxy
seems to be some sort of timeout, because everything work as expected, but
then after waiting for a while and got again the 502.
That is odd, as there is no timeout configured. What does your network config between the
LB and Undertow look like?
Does enabling TCP keep alive (Options.KEEP_ALIVE) help?
Stuart
also, I did set the content-length for static resources and call
.endExchange after a 304 response, set a backlog of 1000 and 10000 too. But
it didn't help.
On Mon, Jan 19, 2015 at 9:57 AM, Tomaž Cerar <tomaz.cerar(a)gmail.com> wrote:
> Maybe related to
>
http://stackoverflow.com/questions/169453/bad-gateway-502-error-with-apac...
>
> or
>
http://qnalist.com/questions/4502641/users-httpd-mod-proxy-ignores-incomp...
>
> On Mon, Jan 19, 2015 at 1:33 PM, Edgar Espina <espina.edgar(a)gmail.com>
> wrote:
>
>> Sure, it is 2.2.15.
>>
>>
>>
>> On Mon, Jan 19, 2015 at 9:29 AM, Tomaž Cerar <tomaz.cerar(a)gmail.com>
>> wrote:
>>
>>> Edgar,
>>>
>>> what is the version of Apache and mod_proxy module you are using.
>>> maybe it will be easier to reproduce with exact version you have..
>>>
>>> --
>>> tomaz
>>>
>>> On Mon, Jan 19, 2015 at 1:15 PM, Edgar Espina
<espina.edgar(a)gmail.com>
>>> wrote:
>>>
>>>> The source code can be found here:
>>>>
https://github.com/jooby-project/jooby
>>>>
>>>> Undertow related classes are here:
>>>>
https://github.com/jooby-project/jooby/tree/master/jooby/src/main/java/or...
>>>> .
>>>> Server is built here:
>>>>
https://github.com/jooby-project/jooby/blob/master/jooby/src/main/java/or...
>>>> Response is sent here:
>>>>
https://github.com/jooby-project/jooby/blob/master/jooby/src/main/java/or...
>>>>
>>>> My microweb-framework was built on top of Jetty, so for now I used the
>>>> blocking API and follow more or less what we usually do with Servlets
>>>> (acquire an outstream).
>>>>
>>>> I will review what I'm doing and try to figure it out what is going
on
>>>> too.
>>>>
>>>> Thanks for your help, Stuart.
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Jan 19, 2015 at 2:12 AM, Stuart Douglas
<sdouglas(a)redhat.com>
>>>> wrote:
>>>>
>>>>> My best guess as to what is happening is that Undertow closing a
>>>>> connection after a request is done for some reason, and the apache
>>>>> attempts
>>>>> to re-use this connection without realising that it is dead.
>>>>>
>>>>> In general this should not happen, Undertow should only forcibly
close
>>>>> a connection if it knows that it is broken (e.g. a content length
is
>>>>> set
>>>>> and the full amount of content is not written). For a normal
graceful
>>>>> close
>>>>> Undertow should be sending Connection:close headers.
>>>>>
>>>>> I am going to investigate some more, and see if I can figure out
what
>>>>> is going on. It seems unlikely but is there any chance your code
>>>>> forcibly
>>>>> closes the ServerConnection (HttpServerExchange.getConnection())
>>>>> because
>>>>> that could potentially cause this issue. Setting the exchange to
>>>>> non-persistent after headers have been sent could also cause it.
>>>>>
>>>>> Stuart
>>>>>
>>>>> ----- Original Message -----
>>>>> > From: "Edgar Espina" <espina.edgar(a)gmail.com>
>>>>> > To: "Stuart Douglas" <sdouglas(a)redhat.com>
>>>>> > Cc: undertow-dev(a)lists.jboss.org
>>>>> > Sent: Monday, 19 January, 2015 2:53:06 PM
>>>>> > Subject: Re: [undertow-dev] occasional 502 from Apache HTTP
Proxy
>>>>> >
>>>>> > Done, but makes no difference. Still got the 502 random errors
:S
>>>>> >
>>>>> > On Sun, Jan 18, 2015 at 11:13 PM, Stuart Douglas <
>>>>> sdouglas(a)redhat.com>
>>>>> > wrote:
>>>>> >
>>>>> > > I had a play around with apache locally, and I could
reproduce
>>>>> this in
>>>>> > > some circumstances, and it looks like we are not setting a
high
>>>>> enough
>>>>> > > backlog by default.
>>>>> > >
>>>>> > > Can you try adding:
>>>>> > >
>>>>> > > undertow.setSocketOption(Options.BACKLOG, 1000)
>>>>> > >
>>>>> > > To your Undertow builder? I am going to increase this in
our
>>>>> default
>>>>> > > config upstream.
>>>>> > >
>>>>> > > Hopefully this is the issue that you are running into.
>>>>> > >
>>>>> > > Stuart
>>>>> > >
>>>>> > >
>>>>> > > ----- Original Message -----
>>>>> > > > From: "Edgar Espina"
<espina.edgar(a)gmail.com>
>>>>> > > > To: "Stuart Douglas"
<sdouglas(a)redhat.com>
>>>>> > > > Cc: undertow-dev(a)lists.jboss.org
>>>>> > > > Sent: Monday, 19 January, 2015 12:10:16 PM
>>>>> > > > Subject: Re: [undertow-dev] occasional 502 from
Apache HTTP Proxy
>>>>> > > >
>>>>> > > > It is a default undertow instance with a HTTP
listener, just set
>>>>> work
>>>>> > > > threads to 200. Is there a default idle timeout? I
can' tell
>>>>> from javadoc
>>>>> > > > and looking at the code.
>>>>> > > >
>>>>> > > >
>>>>> > > > On Sun, Jan 18, 2015 at 9:59 PM, Stuart Douglas <
>>>>> sdouglas(a)redhat.com>
>>>>> > > wrote:
>>>>> > > >
>>>>> > > > >
>>>>> > > > >
>>>>> > > > > Edgar Espina wrote:
>>>>> > > > >
>>>>> > > > >> I'm not, channel does it.
>>>>> > > > >>
>>>>> > > > >> Problem is present on both. For example,
home page has a
>>>>> > > content-length
>>>>> > > > >> header but jquery.js use chunked.
>>>>> > > > >>
>>>>> > > > >
>>>>> > > > > In general if you are serving static resources
you are better
>>>>> off
>>>>> > > setting
>>>>> > > > > the content length (slightly more efficient, and
the browser
>>>>> can
>>>>> > > display
>>>>> > > > > progress on downloads).
>>>>> > > > >
>>>>> > > > >
>>>>> > > > >> Do I have to call .endExchange? or closing
the output stream
>>>>> is
>>>>> > > enough?
>>>>> > > > >>
>>>>> > > > >
>>>>> > > > > Closing the output stream is enough. endExchange
is
>>>>> automatically
>>>>> > > called
>>>>> > > > > once the call stack returns anyway, unless you
have dispatched
>>>>> the
>>>>> > > exchange
>>>>> > > > > or started async IO.
>>>>> > > > >
>>>>> > > > > The 'connection reset by peer' error in
the apache log while
>>>>> reading
>>>>> > > the
>>>>> > > > > status like kinda indicates that the request
does not even get
>>>>> to this
>>>>> > > > > point anyway, and the underlying TCP connection
is probably
>>>>> being torn
>>>>> > > down
>>>>> > > > > somehow.
>>>>> > > > >
>>>>> > > > > Do you have any kind of timeouts set? If you
have an idle
>>>>> timeout set
>>>>> > > on
>>>>> > > > > the listener there is a race where Undertow can
close the
>>>>> channel due
>>>>> > > to
>>>>> > > > > inactivity just as the front end starts to send
a request.
>>>>> > > > >
>>>>> > > > > Stuart
>>>>> > > > >
>>>>> > > > >
>>>>> > > > >> Thanks
>>>>> > > > >>
>>>>> > > > >> On Sun, Jan 18, 2015 at 9:48 PM, Stuart
Douglas <
>>>>> sdouglas(a)redhat.com
>>>>> > > > >> <mailto:sdouglas@redhat.com>>
wrote:
>>>>> > > > >>
>>>>> > > > >> Another question, are you setting a
content length on the
>>>>> > > responses?
>>>>> > > > >> If not the channel will automatically
set one if the
>>>>> response fits
>>>>> > > > >> inside a buffer, otherwise chunked
encoding will be used.
>>>>> > > > >>
>>>>> > > > >> It might be helpful to know if this only
happens on
>>>>> chunked, fixed
>>>>> > > > >> length or both.
>>>>> > > > >>
>>>>> > > > >> Stuart
>>>>> > > > >>
>>>>> > > > >> Edgar Espina wrote:
>>>>> > > > >>
>>>>> > > > >> latest: 1.2.0.Beta8
>>>>> > > > >>
>>>>> > > > >> On Sun, Jan 18, 2015 at 9:42 PM,
Stuart Douglas
>>>>> > > > >> <sdouglas(a)redhat.com
<mailto:sdouglas@redhat.com>
>>>>> > > > >> <mailto:sdouglas@redhat.com
<mailto:
>>>>> sdouglas(a)redhat.com>>>
>>>>> > > wrote:
>>>>> > > > >>
>>>>> > > > >> Also what version of Undertow
are you using?
>>>>> > > > >>
>>>>> > > > >> Stuart
>>>>> > > > >>
>>>>> > > > >> ----- Original Message -----
>>>>> > > > >> > From: "Edgar
Espina" <espina.edgar(a)gmail.com
>>>>> > > > >>
<mailto:espina.edgar@gmail.com>
>>>>> > > > >> <mailto:espina.edgar@gmail.com
<mailto:
>>>>> espina.edgar(a)gmail.com
>>>>> > > > >> >__>>
>>>>> > > > >> > To: "Stuart Douglas"
<sdouglas(a)redhat.com
>>>>> > > > >> <mailto:sdouglas@redhat.com>
>>>>> > > > >> <mailto:sdouglas@redhat.com
<mailto:
>>>>> sdouglas(a)redhat.com>>>
>>>>> > > > >> > Cc:
undertow-dev(a)lists.jboss.org
>>>>> > > > >>
<mailto:undertow-dev@lists.jboss.org>
>>>>> > > > >>
<mailto:undertow-dev@lists.__jboss.org
>>>>> > > > >>
<mailto:undertow-dev@lists.jboss.org>>
>>>>> > > > >> > Sent: Monday, 19 January, 2015
11:13:21 AM
>>>>> > > > >> > Subject: Re: [undertow-dev]
occasional 502 from
>>>>> Apache HTTP
>>>>> > > > >> Proxy
>>>>> > > > >> >
>>>>> > > > >> > found this in apache:
>>>>> > > > >> >
>>>>> > > > >> > (104) Connection reset by
peer: proxy: error
>>>>> reading status
>>>>> > > > >> line from
>>>>> > > > >> > remote server
>>>>> > > > >> >
>>>>> > > > >> > but nothing in undertow.
>>>>> > > > >> >
>>>>> > > > >> > got 502 on HTTP GET. Sometimes
while calling the
>>>>> home page
>>>>> > > / or
>>>>> > > > >> when page
>>>>> > > > >> > loads OK, got 502 on page
resources (js, css,
>>>>> images).
>>>>> > > > >> >
>>>>> > > > >> > I call startBlocking and use
the outputstream to
>>>>> write the
>>>>> > > > >> response, when
>>>>> > > > >> > everything has been written I
call the
>>>>> "outputstream.close"
>>>>> > > > >> method. Do I
>>>>> > > > >> > need to call
exchange.endExchange too?
>>>>> > > > >> >
>>>>> > > > >> > Thanks
>>>>> > > > >> >
>>>>> > > > >> > On Sun, Jan 18, 2015 at 8:52
PM, Stuart Douglas
>>>>> > > > >> <sdouglas(a)redhat.com
<mailto:sdouglas@redhat.com>
>>>>> > > > >> <mailto:sdouglas@redhat.com
<mailto:
>>>>> sdouglas(a)redhat.com>>>
>>>>> > > wrote:
>>>>> > > > >> >
>>>>> > > > >> > > Is there any info in the
log? Or is there any
>>>>> specific
>>>>> > > type
>>>>> > > > >> of
>>>>> > > > >> request
>>>>> > > > >> > > that causes this?
>>>>> > > > >> > >
>>>>> > > > >> > > Stuart
>>>>> > > > >> > >
>>>>> > > > >> > >
>>>>> > > > >> > > ----- Original Message
-----
>>>>> > > > >> > > > From: "Edgar
Espina" <espina.edgar(a)gmail.com
>>>>> > > > >>
<mailto:espina.edgar@gmail.com>
>>>>> > > > >> <mailto:espina.edgar@gmail.com
<mailto:
>>>>> espina.edgar(a)gmail.com
>>>>> > > > >> >__>>
>>>>> > > > >> > > > To:
undertow-dev(a)lists.jboss.org
>>>>> > > > >>
<mailto:undertow-dev@lists.jboss.org>
>>>>> > > > >>
<mailto:undertow-dev@lists.__jboss.org
>>>>> > > > >>
<mailto:undertow-dev@lists.jboss.org>>
>>>>> > > > >> > > > Sent: Monday, 19
January, 2015 9:42:19 AM
>>>>> > > > >> > > > Subject:
[undertow-dev] occasional 502 from
>>>>> Apache HTTP
>>>>> > > > >> Proxy
>>>>> > > > >> > > >
>>>>> > > > >> > > > Hi,
>>>>> > > > >> > > >
>>>>> > > > >> > > > I've an Undertow
application behind apache
>>>>> reverse
>>>>> > > proxy,
>>>>> > > > >> trying to load
>>>>> > > > >> > > a
>>>>> > > > >> > > > page displays error
502 proxy error.
>>>>> > > > >> > > >
>>>>> > > > >> > > > Still couldn't
find why so I wonder if any of
>>>>> you find
>>>>> > > a
>>>>> > > > >> similar problem
>>>>> > > > >> > > with
>>>>> > > > >> > > > Undertow and
Apache.
>>>>> > > > >> > > >
>>>>> > > > >> > > > Please note this is
our first app on top of
>>>>> Undertow,
>>>>> > > > >> existing apps
>>>>> > > > >> > > running
>>>>> > > > >> > > > on Tomcat/Jetty are
OK.
>>>>> > > > >> > > >
>>>>> > > > >> > > > Appreciate any
help.
>>>>> > > > >> > > >
>>>>> > > > >> > > > Thanks
>>>>> > > > >> > > >
>>>>> > > > >> > > > --
>>>>> > > > >> > > > edgar
>>>>> > > > >> > > >
>>>>> > > > >> > > >
>>>>> _________________________________________________
>>>>> > > > >> > > > undertow-dev mailing
list
>>>>> > > > >> > > >
undertow-dev(a)lists.jboss.org
>>>>> > > > >>
<mailto:undertow-dev@lists.jboss.org>
>>>>> > > > >>
<mailto:undertow-dev@lists.__jboss.org
>>>>> > > > >>
<mailto:undertow-dev@lists.jboss.org>>
>>>>> > > > >> > > >
>>>>> > >
https://lists.jboss.org/__mailman/listinfo/undertow-dev
>>>>> > > > >> <
>>>>>
https://lists.jboss.org/mailman/listinfo/undertow-dev>
>>>>> > > > >> > >
>>>>> > > > >> >
>>>>> > > > >> >
>>>>> > > > >> >
>>>>> > > > >> > --
>>>>> > > > >> > edgar
>>>>> > > > >> >
>>>>> > > > >>
>>>>> > > > >>
>>>>> > > > >>
>>>>> > > > >>
>>>>> > > > >> --
>>>>> > > > >> edgar
>>>>> > > > >>
>>>>> > > > >>
>>>>> > > > >>
>>>>> > > > >>
>>>>> > > > >> --
>>>>> > > > >> edgar
>>>>> > > > >>
>>>>> > > > >
>>>>> > > >
>>>>> > > >
>>>>> > > > --
>>>>> > > > edgar
>>>>> > > >
>>>>> > >
>>>>> >
>>>>> >
>>>>> >
>>>>> > --
>>>>> > edgar
>>>>> >
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> edgar
>>>>
>>>> _______________________________________________
>>>> undertow-dev mailing list
>>>> undertow-dev(a)lists.jboss.org
>>>>
https://lists.jboss.org/mailman/listinfo/undertow-dev
>>>>
>>>
>>>
>>
>>
>> --
>> edgar
>>
>
>
--
edgar