[undertow-dev] occasional 502 from Apache HTTP Proxy

Stuart Douglas sdouglas at redhat.com
Tue Jan 20 15:07:35 EST 2015 


----- Original Message -----
> From: "Edgar Espina" <espina.edgar at gmail.com>
> To: "Tomaž Cerar" <tomaz.cerar at gmail.com>
> Cc: "Stuart Douglas" <sdouglas at redhat.com>, undertow-dev at lists.jboss.org
> Sent: Wednesday, 21 January, 2015 2:03:10 AM
> Subject: Re: [undertow-dev] occasional 502 from Apache HTTP Proxy
> 
> seems to be some sort of timeout, because everything work as expected, but
> then after waiting for a while and got again the 502.

That is odd, as there is no timeout configured. What does your network config between the LB and Undertow look like?

Does enabling TCP keep alive (Options.KEEP_ALIVE) help?

Stuart


> 
> also, I did set the content-length for static resources and call
> .endExchange after a 304 response, set a backlog of 1000 and 10000 too. But
> it didn't help.
> 
> 
> 
> On Mon, Jan 19, 2015 at 9:57 AM, Tomaž Cerar <tomaz.cerar at gmail.com> wrote:
> 
> > Maybe related to
> > http://stackoverflow.com/questions/169453/bad-gateway-502-error-with-apache-mod-proxy-and-tomcat
> >
> > or
> > http://qnalist.com/questions/4502641/users-httpd-mod-proxy-ignores-incompleteness-of-chunked-coding-response-from-backend
> >
> > On Mon, Jan 19, 2015 at 1:33 PM, Edgar Espina <espina.edgar at gmail.com>
> > wrote:
> >
> >> Sure, it is 2.2.15.
> >>
> >>
> >>
> >> On Mon, Jan 19, 2015 at 9:29 AM, Tomaž Cerar <tomaz.cerar at gmail.com>
> >> wrote:
> >>
> >>> Edgar,
> >>>
> >>> what is the version of Apache and mod_proxy module you are using.
> >>> maybe it will be easier to reproduce with exact version you have..
> >>>
> >>> --
> >>> tomaz
> >>>
> >>> On Mon, Jan 19, 2015 at 1:15 PM, Edgar Espina <espina.edgar at gmail.com>
> >>> wrote:
> >>>
> >>>> The source code can be found here:
> >>>> https://github.com/jooby-project/jooby
> >>>>
> >>>> Undertow related classes are here:
> >>>> https://github.com/jooby-project/jooby/tree/master/jooby/src/main/java/org/jooby/internal/undertow
> >>>> .
> >>>> Server is built here:
> >>>> https://github.com/jooby-project/jooby/blob/master/jooby/src/main/java/org/jooby/internal/undertow/UndertowServer.java
> >>>> Response is sent here:
> >>>> https://github.com/jooby-project/jooby/blob/master/jooby/src/main/java/org/jooby/internal/undertow/UndertowResponse.java#L336-382
> >>>>
> >>>> My microweb-framework was built on top of Jetty, so for now I used the
> >>>> blocking API and follow more or less what we usually do with Servlets
> >>>> (acquire an outstream).
> >>>>
> >>>> I will review what I'm doing and try to figure it out what is going on
> >>>> too.
> >>>>
> >>>> Thanks for your help, Stuart.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Mon, Jan 19, 2015 at 2:12 AM, Stuart Douglas <sdouglas at redhat.com>
> >>>> wrote:
> >>>>
> >>>>> My best guess as to what is happening is that Undertow closing a
> >>>>> connection after a request is done for some reason, and the apache
> >>>>> attempts
> >>>>> to re-use this connection without realising that it is dead.
> >>>>>
> >>>>> In general this should not happen, Undertow should only forcibly close
> >>>>> a connection if it knows that it is broken (e.g. a content length is
> >>>>> set
> >>>>> and the full amount of content is not written). For a normal graceful
> >>>>> close
> >>>>> Undertow should be sending Connection:close headers.
> >>>>>
> >>>>> I am going to investigate some more, and see if I can figure out what
> >>>>> is going on. It seems unlikely but is there any chance your code
> >>>>> forcibly
> >>>>> closes the ServerConnection (HttpServerExchange.getConnection())
> >>>>> because
> >>>>> that could potentially cause this issue. Setting the exchange to
> >>>>> non-persistent after headers have been sent could also cause it.
> >>>>>
> >>>>> Stuart
> >>>>>
> >>>>> ----- Original Message -----
> >>>>> > From: "Edgar Espina" <espina.edgar at gmail.com>
> >>>>> > To: "Stuart Douglas" <sdouglas at redhat.com>
> >>>>> > Cc: undertow-dev at lists.jboss.org
> >>>>> > Sent: Monday, 19 January, 2015 2:53:06 PM
> >>>>> > Subject: Re: [undertow-dev] occasional 502 from Apache HTTP Proxy
> >>>>> >
> >>>>> > Done, but makes no difference. Still got the 502 random errors :S
> >>>>> >
> >>>>> > On Sun, Jan 18, 2015 at 11:13 PM, Stuart Douglas <
> >>>>> sdouglas at redhat.com>
> >>>>> > wrote:
> >>>>> >
> >>>>> > > I had a play around with apache locally, and I could reproduce
> >>>>> this in
> >>>>> > > some circumstances, and it looks like we are not setting a high
> >>>>> enough
> >>>>> > > backlog by default.
> >>>>> > >
> >>>>> > > Can you try adding:
> >>>>> > >
> >>>>> > > undertow.setSocketOption(Options.BACKLOG, 1000)
> >>>>> > >
> >>>>> > > To your Undertow builder? I am going to increase this in our
> >>>>> default
> >>>>> > > config upstream.
> >>>>> > >
> >>>>> > > Hopefully this is the issue that you are running into.
> >>>>> > >
> >>>>> > > Stuart
> >>>>> > >
> >>>>> > >
> >>>>> > > ----- Original Message -----
> >>>>> > > > From: "Edgar Espina" <espina.edgar at gmail.com>
> >>>>> > > > To: "Stuart Douglas" <sdouglas at redhat.com>
> >>>>> > > > Cc: undertow-dev at lists.jboss.org
> >>>>> > > > Sent: Monday, 19 January, 2015 12:10:16 PM
> >>>>> > > > Subject: Re: [undertow-dev] occasional 502 from Apache HTTP Proxy
> >>>>> > > >
> >>>>> > > > It is a default undertow instance with a HTTP listener, just set
> >>>>> work
> >>>>> > > > threads to 200. Is there a default idle timeout? I can' tell
> >>>>> from javadoc
> >>>>> > > > and looking at the code.
> >>>>> > > >
> >>>>> > > >
> >>>>> > > > On Sun, Jan 18, 2015 at 9:59 PM, Stuart Douglas <
> >>>>> sdouglas at redhat.com>
> >>>>> > > wrote:
> >>>>> > > >
> >>>>> > > > >
> >>>>> > > > >
> >>>>> > > > > Edgar Espina wrote:
> >>>>> > > > >
> >>>>> > > > >> I'm not, channel does it.
> >>>>> > > > >>
> >>>>> > > > >> Problem is present on both. For example, home page has a
> >>>>> > > content-length
> >>>>> > > > >> header but jquery.js use chunked.
> >>>>> > > > >>
> >>>>> > > > >
> >>>>> > > > > In general if you are serving static resources you are better
> >>>>> off
> >>>>> > > setting
> >>>>> > > > > the content length (slightly more efficient, and the browser
> >>>>> can
> >>>>> > > display
> >>>>> > > > > progress on downloads).
> >>>>> > > > >
> >>>>> > > > >
> >>>>> > > > >> Do I have to call .endExchange? or closing the output stream
> >>>>> is
> >>>>> > > enough?
> >>>>> > > > >>
> >>>>> > > > >
> >>>>> > > > > Closing the output stream is enough. endExchange is
> >>>>> automatically
> >>>>> > > called
> >>>>> > > > > once the call stack returns anyway, unless you have dispatched
> >>>>> the
> >>>>> > > exchange
> >>>>> > > > > or started async IO.
> >>>>> > > > >
> >>>>> > > > > The 'connection reset by peer' error in the apache log while
> >>>>> reading
> >>>>> > > the
> >>>>> > > > > status like kinda indicates that the request does not even get
> >>>>> to this
> >>>>> > > > > point anyway, and the underlying TCP connection is probably
> >>>>> being torn
> >>>>> > > down
> >>>>> > > > > somehow.
> >>>>> > > > >
> >>>>> > > > > Do you have any kind of timeouts set? If you have an idle
> >>>>> timeout set
> >>>>> > > on
> >>>>> > > > > the listener there is a race where Undertow can close the
> >>>>> channel due
> >>>>> > > to
> >>>>> > > > > inactivity just as the front end starts to send a request.
> >>>>> > > > >
> >>>>> > > > > Stuart
> >>>>> > > > >
> >>>>> > > > >
> >>>>> > > > >> Thanks
> >>>>> > > > >>
> >>>>> > > > >> On Sun, Jan 18, 2015 at 9:48 PM, Stuart Douglas <
> >>>>> sdouglas at redhat.com
> >>>>> > > > >> <mailto:sdouglas at redhat.com>> wrote:
> >>>>> > > > >>
> >>>>> > > > >>     Another question, are you setting a content length on the
> >>>>> > > responses?
> >>>>> > > > >>     If not the channel will automatically set one if the
> >>>>> response fits
> >>>>> > > > >>     inside a buffer, otherwise chunked encoding will be used.
> >>>>> > > > >>
> >>>>> > > > >>     It might be helpful to know if this only happens on
> >>>>> chunked, fixed
> >>>>> > > > >>     length or both.
> >>>>> > > > >>
> >>>>> > > > >>     Stuart
> >>>>> > > > >>
> >>>>> > > > >>     Edgar Espina wrote:
> >>>>> > > > >>
> >>>>> > > > >>         latest: 1.2.0.Beta8
> >>>>> > > > >>
> >>>>> > > > >>         On Sun, Jan 18, 2015 at 9:42 PM, Stuart Douglas
> >>>>> > > > >>         <sdouglas at redhat.com <mailto:sdouglas at redhat.com>
> >>>>> > > > >>         <mailto:sdouglas at redhat.com <mailto:
> >>>>> sdouglas at redhat.com>>>
> >>>>> > > wrote:
> >>>>> > > > >>
> >>>>> > > > >>              Also what version of Undertow are you using?
> >>>>> > > > >>
> >>>>> > > > >>              Stuart
> >>>>> > > > >>
> >>>>> > > > >>              ----- Original Message -----
> >>>>> > > > >>         >   From: "Edgar Espina" <espina.edgar at gmail.com
> >>>>> > > > >>         <mailto:espina.edgar at gmail.com>
> >>>>> > > > >>         <mailto:espina.edgar at gmail.com <mailto:
> >>>>> espina.edgar at gmail.com
> >>>>> > > > >> >__>>
> >>>>> > > > >>         >  To: "Stuart Douglas" <sdouglas at redhat.com
> >>>>> > > > >>         <mailto:sdouglas at redhat.com>
> >>>>> > > > >>         <mailto:sdouglas at redhat.com <mailto:
> >>>>> sdouglas at redhat.com>>>
> >>>>> > > > >>         >  Cc: undertow-dev at lists.jboss.org
> >>>>> > > > >>         <mailto:undertow-dev at lists.jboss.org>
> >>>>> > > > >>         <mailto:undertow-dev at lists.__jboss.org
> >>>>> > > > >>         <mailto:undertow-dev at lists.jboss.org>>
> >>>>> > > > >>         >  Sent: Monday, 19 January, 2015 11:13:21 AM
> >>>>> > > > >>         >  Subject: Re: [undertow-dev] occasional 502 from
> >>>>> Apache HTTP
> >>>>> > > > >> Proxy
> >>>>> > > > >>         >
> >>>>> > > > >>         >  found this in apache:
> >>>>> > > > >>         >
> >>>>> > > > >>         >  (104) Connection reset by peer: proxy: error
> >>>>> reading status
> >>>>> > > > >>         line from
> >>>>> > > > >>         >  remote server
> >>>>> > > > >>         >
> >>>>> > > > >>         >  but nothing in undertow.
> >>>>> > > > >>         >
> >>>>> > > > >>         >  got 502 on HTTP GET. Sometimes while calling the
> >>>>> home page
> >>>>> > > / or
> >>>>> > > > >>              when page
> >>>>> > > > >>         >  loads OK, got 502 on page resources (js, css,
> >>>>> images).
> >>>>> > > > >>         >
> >>>>> > > > >>         >  I call startBlocking and use the outputstream to
> >>>>> write the
> >>>>> > > > >>              response, when
> >>>>> > > > >>         >  everything has been written I call the
> >>>>> "outputstream.close"
> >>>>> > > > >>              method. Do I
> >>>>> > > > >>         >  need to call exchange.endExchange too?
> >>>>> > > > >>         >
> >>>>> > > > >>         >  Thanks
> >>>>> > > > >>         >
> >>>>> > > > >>         >  On Sun, Jan 18, 2015 at 8:52 PM, Stuart Douglas
> >>>>> > > > >>         <sdouglas at redhat.com <mailto:sdouglas at redhat.com>
> >>>>> > > > >>         <mailto:sdouglas at redhat.com <mailto:
> >>>>> sdouglas at redhat.com>>>
> >>>>> > > wrote:
> >>>>> > > > >>         >
> >>>>> > > > >>         >  > Is there any info in the log? Or is there any
> >>>>> specific
> >>>>> > > type
> >>>>> > > > >> of
> >>>>> > > > >>              request
> >>>>> > > > >>         >  > that causes this?
> >>>>> > > > >>         >  >
> >>>>> > > > >>         >  > Stuart
> >>>>> > > > >>         >  >
> >>>>> > > > >>         >  >
> >>>>> > > > >>         >  > ----- Original Message -----
> >>>>> > > > >>         >  > > From: "Edgar Espina" <espina.edgar at gmail.com
> >>>>> > > > >>         <mailto:espina.edgar at gmail.com>
> >>>>> > > > >>         <mailto:espina.edgar at gmail.com <mailto:
> >>>>> espina.edgar at gmail.com
> >>>>> > > > >> >__>>
> >>>>> > > > >>          > > > To: undertow-dev at lists.jboss.org
> >>>>> > > > >>         <mailto:undertow-dev at lists.jboss.org>
> >>>>> > > > >>         <mailto:undertow-dev at lists.__jboss.org
> >>>>> > > > >>         <mailto:undertow-dev at lists.jboss.org>>
> >>>>> > > > >>          > > > Sent: Monday, 19 January, 2015 9:42:19 AM
> >>>>> > > > >>          > > > Subject: [undertow-dev] occasional 502 from
> >>>>> Apache HTTP
> >>>>> > > > >> Proxy
> >>>>> > > > >>          > > >
> >>>>> > > > >>          > > > Hi,
> >>>>> > > > >>          > > >
> >>>>> > > > >>          > > > I've an Undertow application behind apache
> >>>>> reverse
> >>>>> > > proxy,
> >>>>> > > > >>              trying to load
> >>>>> > > > >>          > > a
> >>>>> > > > >>          > > > page displays error 502 proxy error.
> >>>>> > > > >>          > > >
> >>>>> > > > >>          > > > Still couldn't find why so I wonder if any of
> >>>>> you find
> >>>>> > > a
> >>>>> > > > >>              similar problem
> >>>>> > > > >>          > > with
> >>>>> > > > >>          > > > Undertow and Apache.
> >>>>> > > > >>          > > >
> >>>>> > > > >>          > > > Please note this is our first app on top of
> >>>>> Undertow,
> >>>>> > > > >>              existing apps
> >>>>> > > > >>          > > running
> >>>>> > > > >>          > > > on Tomcat/Jetty are OK.
> >>>>> > > > >>          > > >
> >>>>> > > > >>          > > > Appreciate any help.
> >>>>> > > > >>          > > >
> >>>>> > > > >>          > > > Thanks
> >>>>> > > > >>          > > >
> >>>>> > > > >>          > > > --
> >>>>> > > > >>          > > > edgar
> >>>>> > > > >>          > > >
> >>>>> > > > >>          > > >
> >>>>> _________________________________________________
> >>>>> > > > >>          > > > undertow-dev mailing list
> >>>>> > > > >>          > > > undertow-dev at lists.jboss.org
> >>>>> > > > >>         <mailto:undertow-dev at lists.jboss.org>
> >>>>> > > > >>         <mailto:undertow-dev at lists.__jboss.org
> >>>>> > > > >>         <mailto:undertow-dev at lists.jboss.org>>
> >>>>> > > > >>         >  > >
> >>>>> > > https://lists.jboss.org/__mailman/listinfo/undertow-dev
> >>>>> > > > >>         <
> >>>>> https://lists.jboss.org/mailman/listinfo/undertow-dev>
> >>>>> > > > >>         >  >
> >>>>> > > > >>         >
> >>>>> > > > >>         >
> >>>>> > > > >>         >
> >>>>> > > > >>         >  --
> >>>>> > > > >>         >  edgar
> >>>>> > > > >>         >
> >>>>> > > > >>
> >>>>> > > > >>
> >>>>> > > > >>
> >>>>> > > > >>
> >>>>> > > > >>         --
> >>>>> > > > >>         edgar
> >>>>> > > > >>
> >>>>> > > > >>
> >>>>> > > > >>
> >>>>> > > > >>
> >>>>> > > > >> --
> >>>>> > > > >> edgar
> >>>>> > > > >>
> >>>>> > > > >
> >>>>> > > >
> >>>>> > > >
> >>>>> > > > --
> >>>>> > > > edgar
> >>>>> > > >
> >>>>> > >
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> > --
> >>>>> > edgar
> >>>>> >
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> edgar
> >>>>
> >>>> _______________________________________________
> >>>> undertow-dev mailing list
> >>>> undertow-dev at lists.jboss.org
> >>>> https://lists.jboss.org/mailman/listinfo/undertow-dev
> >>>>
> >>>
> >>>
> >>
> >>
> >> --
> >> edgar
> >>
> >
> >
> 
> 
> --
> edgar
>

More information about the undertow-dev mailing list