[undertow-dev] occasional 502 from Apache HTTP Proxy

Stuart Douglas sdouglas at redhat.com
Thu Jan 22 19:29:21 EST 2015 

I actually just found a bug that could be the root cause of this. The header size check was being performed incorrectly, so in some circumstances a request may be closed after the headers have been sent. 

This is fixed upstream, and will make it into a release shortly. 

Stuart

----- Original Message -----
> From: "Edgar Espina" <espina.edgar at gmail.com>
> To: "Jason Greene" <jason.greene at redhat.com>
> Cc: "Stuart Douglas" <sdouglas at redhat.com>, undertow-dev at lists.jboss.org
> Sent: Friday, 23 January, 2015 3:18:25 AM
> Subject: Re: [undertow-dev] occasional 502 from Apache HTTP Proxy
> 
> we put nginx upfront undertow and everything works as expected.
> 
> so we have: apache -> nginx -> undertow :S
> 
> apache can't be removed it and as I said before existing apps running on
> Tomcat/Jetty work without any issue
> 
> we need to move on with our project (wasted a lot of time on this) but I
> will try capture the traffic with wireshark and shared it.
> 
> still, if you find out something will be helpful too.
> 
> 
> 
> On Wed, Jan 21, 2015 at 4:51 PM, Jason Greene <jason.greene at redhat.com>
> wrote:
> 
> > Would you be willing to capture the traffic with wireshark or (tcpdump
> > using -s 65535) when the failure occurs? It would be helpful in diagnosing
> > the exact reason the connection is lost (tcp, http protocol issue, etc).
> >
> > If you prefer not to post the dump on list, you can email us privately.
> >
> > Thanks!
> >
> >
> > > On Jan 21, 2015, at 1:40 PM, Edgar Espina <espina.edgar at gmail.com>
> > wrote:
> > >
> > > Nope, it doesn't help.
> > >
> > > Stuart, you mentioned that:
> > > "For a normal graceful close Undertow should be sending Connection:close
> > headers."
> > >
> > > Is it mean that on a req/res cycle, Undertow should always set the
> > Connection response header with close or keep-alive?
> > >
> > > Thanks
> > >
> > >
> > >
> > >
> > > On Tue, Jan 20, 2015 at 5:07 PM, Stuart Douglas <sdouglas at redhat.com>
> > wrote:
> > >
> > >
> > > ----- Original Message -----
> > > > From: "Edgar Espina" <espina.edgar at gmail.com>
> > > > To: "Tomaž Cerar" <tomaz.cerar at gmail.com>
> > > > Cc: "Stuart Douglas" <sdouglas at redhat.com>,
> > undertow-dev at lists.jboss.org
> > > > Sent: Wednesday, 21 January, 2015 2:03:10 AM
> > > > Subject: Re: [undertow-dev] occasional 502 from Apache HTTP Proxy
> > > >
> > > > seems to be some sort of timeout, because everything work as expected,
> > but
> > > > then after waiting for a while and got again the 502.
> > >
> > > That is odd, as there is no timeout configured. What does your network
> > config between the LB and Undertow look like?
> > >
> > > Does enabling TCP keep alive (Options.KEEP_ALIVE) help?
> > >
> > > Stuart
> > >
> > >
> > > >
> > > > also, I did set the content-length for static resources and call
> > > > .endExchange after a 304 response, set a backlog of 1000 and 10000
> > too. But
> > > > it didn't help.
> > > >
> > > >
> > > >
> > > > On Mon, Jan 19, 2015 at 9:57 AM, Tomaž Cerar <tomaz.cerar at gmail.com>
> > wrote:
> > > >
> > > > > Maybe related to
> > > > >
> > http://stackoverflow.com/questions/169453/bad-gateway-502-error-with-apache-mod-proxy-and-tomcat
> > > > >
> > > > > or
> > > > >
> > http://qnalist.com/questions/4502641/users-httpd-mod-proxy-ignores-incompleteness-of-chunked-coding-response-from-backend
> > > > >
> > > > > On Mon, Jan 19, 2015 at 1:33 PM, Edgar Espina <
> > espina.edgar at gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Sure, it is 2.2.15.
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Mon, Jan 19, 2015 at 9:29 AM, Tomaž Cerar <tomaz.cerar at gmail.com
> > >
> > > > >> wrote:
> > > > >>
> > > > >>> Edgar,
> > > > >>>
> > > > >>> what is the version of Apache and mod_proxy module you are using.
> > > > >>> maybe it will be easier to reproduce with exact version you have..
> > > > >>>
> > > > >>> --
> > > > >>> tomaz
> > > > >>>
> > > > >>> On Mon, Jan 19, 2015 at 1:15 PM, Edgar Espina <
> > espina.edgar at gmail.com>
> > > > >>> wrote:
> > > > >>>
> > > > >>>> The source code can be found here:
> > > > >>>> https://github.com/jooby-project/jooby
> > > > >>>>
> > > > >>>> Undertow related classes are here:
> > > > >>>>
> > https://github.com/jooby-project/jooby/tree/master/jooby/src/main/java/org/jooby/internal/undertow
> > > > >>>> .
> > > > >>>> Server is built here:
> > > > >>>>
> > https://github.com/jooby-project/jooby/blob/master/jooby/src/main/java/org/jooby/internal/undertow/UndertowServer.java
> > > > >>>> Response is sent here:
> > > > >>>>
> > https://github.com/jooby-project/jooby/blob/master/jooby/src/main/java/org/jooby/internal/undertow/UndertowResponse.java#L336-382
> > > > >>>>
> > > > >>>> My microweb-framework was built on top of Jetty, so for now I
> > used the
> > > > >>>> blocking API and follow more or less what we usually do with
> > Servlets
> > > > >>>> (acquire an outstream).
> > > > >>>>
> > > > >>>> I will review what I'm doing and try to figure it out what is
> > going on
> > > > >>>> too.
> > > > >>>>
> > > > >>>> Thanks for your help, Stuart.
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> On Mon, Jan 19, 2015 at 2:12 AM, Stuart Douglas <
> > sdouglas at redhat.com>
> > > > >>>> wrote:
> > > > >>>>
> > > > >>>>> My best guess as to what is happening is that Undertow closing a
> > > > >>>>> connection after a request is done for some reason, and the
> > apache
> > > > >>>>> attempts
> > > > >>>>> to re-use this connection without realising that it is dead.
> > > > >>>>>
> > > > >>>>> In general this should not happen, Undertow should only forcibly
> > close
> > > > >>>>> a connection if it knows that it is broken (e.g. a content
> > length is
> > > > >>>>> set
> > > > >>>>> and the full amount of content is not written). For a normal
> > graceful
> > > > >>>>> close
> > > > >>>>> Undertow should be sending Connection:close headers.
> > > > >>>>>
> > > > >>>>> I am going to investigate some more, and see if I can figure out
> > what
> > > > >>>>> is going on. It seems unlikely but is there any chance your code
> > > > >>>>> forcibly
> > > > >>>>> closes the ServerConnection (HttpServerExchange.getConnection())
> > > > >>>>> because
> > > > >>>>> that could potentially cause this issue. Setting the exchange to
> > > > >>>>> non-persistent after headers have been sent could also cause it.
> > > > >>>>>
> > > > >>>>> Stuart
> > > > >>>>>
> > > > >>>>> ----- Original Message -----
> > > > >>>>> > From: "Edgar Espina" <espina.edgar at gmail.com>
> > > > >>>>> > To: "Stuart Douglas" <sdouglas at redhat.com>
> > > > >>>>> > Cc: undertow-dev at lists.jboss.org
> > > > >>>>> > Sent: Monday, 19 January, 2015 2:53:06 PM
> > > > >>>>> > Subject: Re: [undertow-dev] occasional 502 from Apache HTTP
> > Proxy
> > > > >>>>> >
> > > > >>>>> > Done, but makes no difference. Still got the 502 random errors
> > :S
> > > > >>>>> >
> > > > >>>>> > On Sun, Jan 18, 2015 at 11:13 PM, Stuart Douglas <
> > > > >>>>> sdouglas at redhat.com>
> > > > >>>>> > wrote:
> > > > >>>>> >
> > > > >>>>> > > I had a play around with apache locally, and I could
> > reproduce
> > > > >>>>> this in
> > > > >>>>> > > some circumstances, and it looks like we are not setting a
> > high
> > > > >>>>> enough
> > > > >>>>> > > backlog by default.
> > > > >>>>> > >
> > > > >>>>> > > Can you try adding:
> > > > >>>>> > >
> > > > >>>>> > > undertow.setSocketOption(Options.BACKLOG, 1000)
> > > > >>>>> > >
> > > > >>>>> > > To your Undertow builder? I am going to increase this in our
> > > > >>>>> default
> > > > >>>>> > > config upstream.
> > > > >>>>> > >
> > > > >>>>> > > Hopefully this is the issue that you are running into.
> > > > >>>>> > >
> > > > >>>>> > > Stuart
> > > > >>>>> > >
> > > > >>>>> > >
> > > > >>>>> > > ----- Original Message -----
> > > > >>>>> > > > From: "Edgar Espina" <espina.edgar at gmail.com>
> > > > >>>>> > > > To: "Stuart Douglas" <sdouglas at redhat.com>
> > > > >>>>> > > > Cc: undertow-dev at lists.jboss.org
> > > > >>>>> > > > Sent: Monday, 19 January, 2015 12:10:16 PM
> > > > >>>>> > > > Subject: Re: [undertow-dev] occasional 502 from Apache
> > HTTP Proxy
> > > > >>>>> > > >
> > > > >>>>> > > > It is a default undertow instance with a HTTP listener,
> > just set
> > > > >>>>> work
> > > > >>>>> > > > threads to 200. Is there a default idle timeout? I can'
> > tell
> > > > >>>>> from javadoc
> > > > >>>>> > > > and looking at the code.
> > > > >>>>> > > >
> > > > >>>>> > > >
> > > > >>>>> > > > On Sun, Jan 18, 2015 at 9:59 PM, Stuart Douglas <
> > > > >>>>> sdouglas at redhat.com>
> > > > >>>>> > > wrote:
> > > > >>>>> > > >
> > > > >>>>> > > > >
> > > > >>>>> > > > >
> > > > >>>>> > > > > Edgar Espina wrote:
> > > > >>>>> > > > >
> > > > >>>>> > > > >> I'm not, channel does it.
> > > > >>>>> > > > >>
> > > > >>>>> > > > >> Problem is present on both. For example, home page has a
> > > > >>>>> > > content-length
> > > > >>>>> > > > >> header but jquery.js use chunked.
> > > > >>>>> > > > >>
> > > > >>>>> > > > >
> > > > >>>>> > > > > In general if you are serving static resources you are
> > better
> > > > >>>>> off
> > > > >>>>> > > setting
> > > > >>>>> > > > > the content length (slightly more efficient, and the
> > browser
> > > > >>>>> can
> > > > >>>>> > > display
> > > > >>>>> > > > > progress on downloads).
> > > > >>>>> > > > >
> > > > >>>>> > > > >
> > > > >>>>> > > > >> Do I have to call .endExchange? or closing the output
> > stream
> > > > >>>>> is
> > > > >>>>> > > enough?
> > > > >>>>> > > > >>
> > > > >>>>> > > > >
> > > > >>>>> > > > > Closing the output stream is enough. endExchange is
> > > > >>>>> automatically
> > > > >>>>> > > called
> > > > >>>>> > > > > once the call stack returns anyway, unless you have
> > dispatched
> > > > >>>>> the
> > > > >>>>> > > exchange
> > > > >>>>> > > > > or started async IO.
> > > > >>>>> > > > >
> > > > >>>>> > > > > The 'connection reset by peer' error in the apache log
> > while
> > > > >>>>> reading
> > > > >>>>> > > the
> > > > >>>>> > > > > status like kinda indicates that the request does not
> > even get
> > > > >>>>> to this
> > > > >>>>> > > > > point anyway, and the underlying TCP connection is
> > probably
> > > > >>>>> being torn
> > > > >>>>> > > down
> > > > >>>>> > > > > somehow.
> > > > >>>>> > > > >
> > > > >>>>> > > > > Do you have any kind of timeouts set? If you have an idle
> > > > >>>>> timeout set
> > > > >>>>> > > on
> > > > >>>>> > > > > the listener there is a race where Undertow can close the
> > > > >>>>> channel due
> > > > >>>>> > > to
> > > > >>>>> > > > > inactivity just as the front end starts to send a
> > request.
> > > > >>>>> > > > >
> > > > >>>>> > > > > Stuart
> > > > >>>>> > > > >
> > > > >>>>> > > > >
> > > > >>>>> > > > >> Thanks
> > > > >>>>> > > > >>
> > > > >>>>> > > > >> On Sun, Jan 18, 2015 at 9:48 PM, Stuart Douglas <
> > > > >>>>> sdouglas at redhat.com
> > > > >>>>> > > > >> <mailto:sdouglas at redhat.com>> wrote:
> > > > >>>>> > > > >>
> > > > >>>>> > > > >>     Another question, are you setting a content length
> > on the
> > > > >>>>> > > responses?
> > > > >>>>> > > > >>     If not the channel will automatically set one if the
> > > > >>>>> response fits
> > > > >>>>> > > > >>     inside a buffer, otherwise chunked encoding will be
> > used.
> > > > >>>>> > > > >>
> > > > >>>>> > > > >>     It might be helpful to know if this only happens on
> > > > >>>>> chunked, fixed
> > > > >>>>> > > > >>     length or both.
> > > > >>>>> > > > >>
> > > > >>>>> > > > >>     Stuart
> > > > >>>>> > > > >>
> > > > >>>>> > > > >>     Edgar Espina wrote:
> > > > >>>>> > > > >>
> > > > >>>>> > > > >>         latest: 1.2.0.Beta8
> > > > >>>>> > > > >>
> > > > >>>>> > > > >>         On Sun, Jan 18, 2015 at 9:42 PM, Stuart Douglas
> > > > >>>>> > > > >>         <sdouglas at redhat.com <mailto:
> > sdouglas at redhat.com>
> > > > >>>>> > > > >>         <mailto:sdouglas at redhat.com <mailto:
> > > > >>>>> sdouglas at redhat.com>>>
> > > > >>>>> > > wrote:
> > > > >>>>> > > > >>
> > > > >>>>> > > > >>              Also what version of Undertow are you
> > using?
> > > > >>>>> > > > >>
> > > > >>>>> > > > >>              Stuart
> > > > >>>>> > > > >>
> > > > >>>>> > > > >>              ----- Original Message -----
> > > > >>>>> > > > >>         >   From: "Edgar Espina" <
> > espina.edgar at gmail.com
> > > > >>>>> > > > >>         <mailto:espina.edgar at gmail.com>
> > > > >>>>> > > > >>         <mailto:espina.edgar at gmail.com <mailto:
> > > > >>>>> espina.edgar at gmail.com
> > > > >>>>> > > > >> >__>>
> > > > >>>>> > > > >>         >  To: "Stuart Douglas" <sdouglas at redhat.com
> > > > >>>>> > > > >>         <mailto:sdouglas at redhat.com>
> > > > >>>>> > > > >>         <mailto:sdouglas at redhat.com <mailto:
> > > > >>>>> sdouglas at redhat.com>>>
> > > > >>>>> > > > >>         >  Cc: undertow-dev at lists.jboss.org
> > > > >>>>> > > > >>         <mailto:undertow-dev at lists.jboss.org>
> > > > >>>>> > > > >>         <mailto:undertow-dev at lists.__jboss.org
> > > > >>>>> > > > >>         <mailto:undertow-dev at lists.jboss.org>>
> > > > >>>>> > > > >>         >  Sent: Monday, 19 January, 2015 11:13:21 AM
> > > > >>>>> > > > >>         >  Subject: Re: [undertow-dev] occasional 502
> > from
> > > > >>>>> Apache HTTP
> > > > >>>>> > > > >> Proxy
> > > > >>>>> > > > >>         >
> > > > >>>>> > > > >>         >  found this in apache:
> > > > >>>>> > > > >>         >
> > > > >>>>> > > > >>         >  (104) Connection reset by peer: proxy: error
> > > > >>>>> reading status
> > > > >>>>> > > > >>         line from
> > > > >>>>> > > > >>         >  remote server
> > > > >>>>> > > > >>         >
> > > > >>>>> > > > >>         >  but nothing in undertow.
> > > > >>>>> > > > >>         >
> > > > >>>>> > > > >>         >  got 502 on HTTP GET. Sometimes while calling
> > the
> > > > >>>>> home page
> > > > >>>>> > > / or
> > > > >>>>> > > > >>              when page
> > > > >>>>> > > > >>         >  loads OK, got 502 on page resources (js, css,
> > > > >>>>> images).
> > > > >>>>> > > > >>         >
> > > > >>>>> > > > >>         >  I call startBlocking and use the
> > outputstream to
> > > > >>>>> write the
> > > > >>>>> > > > >>              response, when
> > > > >>>>> > > > >>         >  everything has been written I call the
> > > > >>>>> "outputstream.close"
> > > > >>>>> > > > >>              method. Do I
> > > > >>>>> > > > >>         >  need to call exchange.endExchange too?
> > > > >>>>> > > > >>         >
> > > > >>>>> > > > >>         >  Thanks
> > > > >>>>> > > > >>         >
> > > > >>>>> > > > >>         >  On Sun, Jan 18, 2015 at 8:52 PM, Stuart
> > Douglas
> > > > >>>>> > > > >>         <sdouglas at redhat.com <mailto:
> > sdouglas at redhat.com>
> > > > >>>>> > > > >>         <mailto:sdouglas at redhat.com <mailto:
> > > > >>>>> sdouglas at redhat.com>>>
> > > > >>>>> > > wrote:
> > > > >>>>> > > > >>         >
> > > > >>>>> > > > >>         >  > Is there any info in the log? Or is there
> > any
> > > > >>>>> specific
> > > > >>>>> > > type
> > > > >>>>> > > > >> of
> > > > >>>>> > > > >>              request
> > > > >>>>> > > > >>         >  > that causes this?
> > > > >>>>> > > > >>         >  >
> > > > >>>>> > > > >>         >  > Stuart
> > > > >>>>> > > > >>         >  >
> > > > >>>>> > > > >>         >  >
> > > > >>>>> > > > >>         >  > ----- Original Message -----
> > > > >>>>> > > > >>         >  > > From: "Edgar Espina" <
> > espina.edgar at gmail.com
> > > > >>>>> > > > >>         <mailto:espina.edgar at gmail.com>
> > > > >>>>> > > > >>         <mailto:espina.edgar at gmail.com <mailto:
> > > > >>>>> espina.edgar at gmail.com
> > > > >>>>> > > > >> >__>>
> > > > >>>>> > > > >>          > > > To: undertow-dev at lists.jboss.org
> > > > >>>>> > > > >>         <mailto:undertow-dev at lists.jboss.org>
> > > > >>>>> > > > >>         <mailto:undertow-dev at lists.__jboss.org
> > > > >>>>> > > > >>         <mailto:undertow-dev at lists.jboss.org>>
> > > > >>>>> > > > >>          > > > Sent: Monday, 19 January, 2015 9:42:19 AM
> > > > >>>>> > > > >>          > > > Subject: [undertow-dev] occasional 502
> > from
> > > > >>>>> Apache HTTP
> > > > >>>>> > > > >> Proxy
> > > > >>>>> > > > >>          > > >
> > > > >>>>> > > > >>          > > > Hi,
> > > > >>>>> > > > >>          > > >
> > > > >>>>> > > > >>          > > > I've an Undertow application behind
> > apache
> > > > >>>>> reverse
> > > > >>>>> > > proxy,
> > > > >>>>> > > > >>              trying to load
> > > > >>>>> > > > >>          > > a
> > > > >>>>> > > > >>          > > > page displays error 502 proxy error.
> > > > >>>>> > > > >>          > > >
> > > > >>>>> > > > >>          > > > Still couldn't find why so I wonder if
> > any of
> > > > >>>>> you find
> > > > >>>>> > > a
> > > > >>>>> > > > >>              similar problem
> > > > >>>>> > > > >>          > > with
> > > > >>>>> > > > >>          > > > Undertow and Apache.
> > > > >>>>> > > > >>          > > >
> > > > >>>>> > > > >>          > > > Please note this is our first app on top
> > of
> > > > >>>>> Undertow,
> > > > >>>>> > > > >>              existing apps
> > > > >>>>> > > > >>          > > running
> > > > >>>>> > > > >>          > > > on Tomcat/Jetty are OK.
> > > > >>>>> > > > >>          > > >
> > > > >>>>> > > > >>          > > > Appreciate any help.
> > > > >>>>> > > > >>          > > >
> > > > >>>>> > > > >>          > > > Thanks
> > > > >>>>> > > > >>          > > >
> > > > >>>>> > > > >>          > > > --
> > > > >>>>> > > > >>          > > > edgar
> > > > >>>>> > > > >>          > > >
> > > > >>>>> > > > >>          > > >
> > > > >>>>> _________________________________________________
> > > > >>>>> > > > >>          > > > undertow-dev mailing list
> > > > >>>>> > > > >>          > > > undertow-dev at lists.jboss.org
> > > > >>>>> > > > >>         <mailto:undertow-dev at lists.jboss.org>
> > > > >>>>> > > > >>         <mailto:undertow-dev at lists.__jboss.org
> > > > >>>>> > > > >>         <mailto:undertow-dev at lists.jboss.org>>
> > > > >>>>> > > > >>         >  > >
> > > > >>>>> > > https://lists.jboss.org/__mailman/listinfo/undertow-dev
> > > > >>>>> > > > >>         <
> > > > >>>>> https://lists.jboss.org/mailman/listinfo/undertow-dev>
> > > > >>>>> > > > >>         >  >
> > > > >>>>> > > > >>         >
> > > > >>>>> > > > >>         >
> > > > >>>>> > > > >>         >
> > > > >>>>> > > > >>         >  --
> > > > >>>>> > > > >>         >  edgar
> > > > >>>>> > > > >>         >
> > > > >>>>> > > > >>
> > > > >>>>> > > > >>
> > > > >>>>> > > > >>
> > > > >>>>> > > > >>
> > > > >>>>> > > > >>         --
> > > > >>>>> > > > >>         edgar
> > > > >>>>> > > > >>
> > > > >>>>> > > > >>
> > > > >>>>> > > > >>
> > > > >>>>> > > > >>
> > > > >>>>> > > > >> --
> > > > >>>>> > > > >> edgar
> > > > >>>>> > > > >>
> > > > >>>>> > > > >
> > > > >>>>> > > >
> > > > >>>>> > > >
> > > > >>>>> > > > --
> > > > >>>>> > > > edgar
> > > > >>>>> > > >
> > > > >>>>> > >
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> > --
> > > > >>>>> > edgar
> > > > >>>>> >
> > > > >>>>>
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> --
> > > > >>>> edgar
> > > > >>>>
> > > > >>>> _______________________________________________
> > > > >>>> undertow-dev mailing list
> > > > >>>> undertow-dev at lists.jboss.org
> > > > >>>> https://lists.jboss.org/mailman/listinfo/undertow-dev
> > > > >>>>
> > > > >>>
> > > > >>>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> edgar
> > > > >>
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > edgar
> > > >
> > >
> > >
> > >
> > > --
> > > edgar
> > > _______________________________________________
> > > undertow-dev mailing list
> > > undertow-dev at lists.jboss.org
> > > https://lists.jboss.org/mailman/listinfo/undertow-dev
> >
> > --
> > Jason T. Greene
> > WildFly Lead / JBoss EAP Platform Architect
> > JBoss, a division of Red Hat
> >
> >
> 
> 
> --
> edgar
>

More information about the undertow-dev mailing list