Would you be willing to capture the traffic with wireshark or (tcpdump using -s 65535)
when the failure occurs? It would be helpful in diagnosing the exact reason the connection
is lost (tcp, http protocol issue, etc).
If you prefer not to post the dump on list, you can email us privately.
Thanks!
On Jan 21, 2015, at 1:40 PM, Edgar Espina
<espina.edgar(a)gmail.com> wrote:
Nope, it doesn't help.
Stuart, you mentioned that:
"For a normal graceful close Undertow should be sending Connection:close
headers."
Is it mean that on a req/res cycle, Undertow should always set the Connection response
header with close or keep-alive?
Thanks
On Tue, Jan 20, 2015 at 5:07 PM, Stuart Douglas <sdouglas(a)redhat.com> wrote:
----- Original Message -----
> From: "Edgar Espina" <espina.edgar(a)gmail.com>
> To: "Tomaž Cerar" <tomaz.cerar(a)gmail.com>
> Cc: "Stuart Douglas" <sdouglas(a)redhat.com>,
undertow-dev(a)lists.jboss.org
> Sent: Wednesday, 21 January, 2015 2:03:10 AM
> Subject: Re: [undertow-dev] occasional 502 from Apache HTTP Proxy
>
> seems to be some sort of timeout, because everything work as expected, but
> then after waiting for a while and got again the 502.
That is odd, as there is no timeout configured. What does your network config between the
LB and Undertow look like?
Does enabling TCP keep alive (Options.KEEP_ALIVE) help?
Stuart
>
> also, I did set the content-length for static resources and call
> .endExchange after a 304 response, set a backlog of 1000 and 10000 too. But
> it didn't help.
>
>
>
> On Mon, Jan 19, 2015 at 9:57 AM, Tomaž Cerar <tomaz.cerar(a)gmail.com> wrote:
>
> > Maybe related to
> >
http://stackoverflow.com/questions/169453/bad-gateway-502-error-with-apac...
> >
> > or
> >
http://qnalist.com/questions/4502641/users-httpd-mod-proxy-ignores-incomp...
> >
> > On Mon, Jan 19, 2015 at 1:33 PM, Edgar Espina <espina.edgar(a)gmail.com>
> > wrote:
> >
> >> Sure, it is 2.2.15.
> >>
> >>
> >>
> >> On Mon, Jan 19, 2015 at 9:29 AM, Tomaž Cerar <tomaz.cerar(a)gmail.com>
> >> wrote:
> >>
> >>> Edgar,
> >>>
> >>> what is the version of Apache and mod_proxy module you are using.
> >>> maybe it will be easier to reproduce with exact version you have..
> >>>
> >>> --
> >>> tomaz
> >>>
> >>> On Mon, Jan 19, 2015 at 1:15 PM, Edgar Espina
<espina.edgar(a)gmail.com>
> >>> wrote:
> >>>
> >>>> The source code can be found here:
> >>>>
https://github.com/jooby-project/jooby
> >>>>
> >>>> Undertow related classes are here:
> >>>>
https://github.com/jooby-project/jooby/tree/master/jooby/src/main/java/or...
> >>>> .
> >>>> Server is built here:
> >>>>
https://github.com/jooby-project/jooby/blob/master/jooby/src/main/java/or...
> >>>> Response is sent here:
> >>>>
https://github.com/jooby-project/jooby/blob/master/jooby/src/main/java/or...
> >>>>
> >>>> My microweb-framework was built on top of Jetty, so for now I used
the
> >>>> blocking API and follow more or less what we usually do with
Servlets
> >>>> (acquire an outstream).
> >>>>
> >>>> I will review what I'm doing and try to figure it out what is
going on
> >>>> too.
> >>>>
> >>>> Thanks for your help, Stuart.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Mon, Jan 19, 2015 at 2:12 AM, Stuart Douglas
<sdouglas(a)redhat.com>
> >>>> wrote:
> >>>>
> >>>>> My best guess as to what is happening is that Undertow closing
a
> >>>>> connection after a request is done for some reason, and the
apache
> >>>>> attempts
> >>>>> to re-use this connection without realising that it is dead.
> >>>>>
> >>>>> In general this should not happen, Undertow should only
forcibly close
> >>>>> a connection if it knows that it is broken (e.g. a content
length is
> >>>>> set
> >>>>> and the full amount of content is not written). For a normal
graceful
> >>>>> close
> >>>>> Undertow should be sending Connection:close headers.
> >>>>>
> >>>>> I am going to investigate some more, and see if I can figure
out what
> >>>>> is going on. It seems unlikely but is there any chance your
code
> >>>>> forcibly
> >>>>> closes the ServerConnection
(HttpServerExchange.getConnection())
> >>>>> because
> >>>>> that could potentially cause this issue. Setting the exchange
to
> >>>>> non-persistent after headers have been sent could also cause
it.
> >>>>>
> >>>>> Stuart
> >>>>>
> >>>>> ----- Original Message -----
> >>>>> > From: "Edgar Espina"
<espina.edgar(a)gmail.com>
> >>>>> > To: "Stuart Douglas"
<sdouglas(a)redhat.com>
> >>>>> > Cc: undertow-dev(a)lists.jboss.org
> >>>>> > Sent: Monday, 19 January, 2015 2:53:06 PM
> >>>>> > Subject: Re: [undertow-dev] occasional 502 from Apache
HTTP Proxy
> >>>>> >
> >>>>> > Done, but makes no difference. Still got the 502 random
errors :S
> >>>>> >
> >>>>> > On Sun, Jan 18, 2015 at 11:13 PM, Stuart Douglas <
> >>>>> sdouglas(a)redhat.com>
> >>>>> > wrote:
> >>>>> >
> >>>>> > > I had a play around with apache locally, and I could
reproduce
> >>>>> this in
> >>>>> > > some circumstances, and it looks like we are not
setting a high
> >>>>> enough
> >>>>> > > backlog by default.
> >>>>> > >
> >>>>> > > Can you try adding:
> >>>>> > >
> >>>>> > > undertow.setSocketOption(Options.BACKLOG, 1000)
> >>>>> > >
> >>>>> > > To your Undertow builder? I am going to increase this
in our
> >>>>> default
> >>>>> > > config upstream.
> >>>>> > >
> >>>>> > > Hopefully this is the issue that you are running
into.
> >>>>> > >
> >>>>> > > Stuart
> >>>>> > >
> >>>>> > >
> >>>>> > > ----- Original Message -----
> >>>>> > > > From: "Edgar Espina"
<espina.edgar(a)gmail.com>
> >>>>> > > > To: "Stuart Douglas"
<sdouglas(a)redhat.com>
> >>>>> > > > Cc: undertow-dev(a)lists.jboss.org
> >>>>> > > > Sent: Monday, 19 January, 2015 12:10:16 PM
> >>>>> > > > Subject: Re: [undertow-dev] occasional 502 from
Apache HTTP Proxy
> >>>>> > > >
> >>>>> > > > It is a default undertow instance with a HTTP
listener, just set
> >>>>> work
> >>>>> > > > threads to 200. Is there a default idle timeout?
I can' tell
> >>>>> from javadoc
> >>>>> > > > and looking at the code.
> >>>>> > > >
> >>>>> > > >
> >>>>> > > > On Sun, Jan 18, 2015 at 9:59 PM, Stuart Douglas
<
> >>>>> sdouglas(a)redhat.com>
> >>>>> > > wrote:
> >>>>> > > >
> >>>>> > > > >
> >>>>> > > > >
> >>>>> > > > > Edgar Espina wrote:
> >>>>> > > > >
> >>>>> > > > >> I'm not, channel does it.
> >>>>> > > > >>
> >>>>> > > > >> Problem is present on both. For
example, home page has a
> >>>>> > > content-length
> >>>>> > > > >> header but jquery.js use chunked.
> >>>>> > > > >>
> >>>>> > > > >
> >>>>> > > > > In general if you are serving static
resources you are better
> >>>>> off
> >>>>> > > setting
> >>>>> > > > > the content length (slightly more
efficient, and the browser
> >>>>> can
> >>>>> > > display
> >>>>> > > > > progress on downloads).
> >>>>> > > > >
> >>>>> > > > >
> >>>>> > > > >> Do I have to call .endExchange? or
closing the output stream
> >>>>> is
> >>>>> > > enough?
> >>>>> > > > >>
> >>>>> > > > >
> >>>>> > > > > Closing the output stream is enough.
endExchange is
> >>>>> automatically
> >>>>> > > called
> >>>>> > > > > once the call stack returns anyway, unless
you have dispatched
> >>>>> the
> >>>>> > > exchange
> >>>>> > > > > or started async IO.
> >>>>> > > > >
> >>>>> > > > > The 'connection reset by peer'
error in the apache log while
> >>>>> reading
> >>>>> > > the
> >>>>> > > > > status like kinda indicates that the
request does not even get
> >>>>> to this
> >>>>> > > > > point anyway, and the underlying TCP
connection is probably
> >>>>> being torn
> >>>>> > > down
> >>>>> > > > > somehow.
> >>>>> > > > >
> >>>>> > > > > Do you have any kind of timeouts set? If
you have an idle
> >>>>> timeout set
> >>>>> > > on
> >>>>> > > > > the listener there is a race where Undertow
can close the
> >>>>> channel due
> >>>>> > > to
> >>>>> > > > > inactivity just as the front end starts to
send a request.
> >>>>> > > > >
> >>>>> > > > > Stuart
> >>>>> > > > >
> >>>>> > > > >
> >>>>> > > > >> Thanks
> >>>>> > > > >>
> >>>>> > > > >> On Sun, Jan 18, 2015 at 9:48 PM, Stuart
Douglas <
> >>>>> sdouglas(a)redhat.com
> >>>>> > > > >> <mailto:sdouglas@redhat.com>>
wrote:
> >>>>> > > > >>
> >>>>> > > > >> Another question, are you setting a
content length on the
> >>>>> > > responses?
> >>>>> > > > >> If not the channel will
automatically set one if the
> >>>>> response fits
> >>>>> > > > >> inside a buffer, otherwise chunked
encoding will be used.
> >>>>> > > > >>
> >>>>> > > > >> It might be helpful to know if this
only happens on
> >>>>> chunked, fixed
> >>>>> > > > >> length or both.
> >>>>> > > > >>
> >>>>> > > > >> Stuart
> >>>>> > > > >>
> >>>>> > > > >> Edgar Espina wrote:
> >>>>> > > > >>
> >>>>> > > > >> latest: 1.2.0.Beta8
> >>>>> > > > >>
> >>>>> > > > >> On Sun, Jan 18, 2015 at 9:42
PM, Stuart Douglas
> >>>>> > > > >> <sdouglas(a)redhat.com
<mailto:sdouglas@redhat.com>
> >>>>> > > > >> <mailto:sdouglas@redhat.com
<mailto:
> >>>>> sdouglas(a)redhat.com>>>
> >>>>> > > wrote:
> >>>>> > > > >>
> >>>>> > > > >> Also what version of
Undertow are you using?
> >>>>> > > > >>
> >>>>> > > > >> Stuart
> >>>>> > > > >>
> >>>>> > > > >> ----- Original Message
-----
> >>>>> > > > >> > From: "Edgar
Espina" <espina.edgar(a)gmail.com
> >>>>> > > > >>
<mailto:espina.edgar@gmail.com>
> >>>>> > > > >>
<mailto:espina.edgar@gmail.com <mailto:
> >>>>> espina.edgar(a)gmail.com
> >>>>> > > > >> >__>>
> >>>>> > > > >> > To: "Stuart
Douglas" <sdouglas(a)redhat.com
> >>>>> > > > >>
<mailto:sdouglas@redhat.com>
> >>>>> > > > >> <mailto:sdouglas@redhat.com
<mailto:
> >>>>> sdouglas(a)redhat.com>>>
> >>>>> > > > >> > Cc:
undertow-dev(a)lists.jboss.org
> >>>>> > > > >>
<mailto:undertow-dev@lists.jboss.org>
> >>>>> > > > >>
<mailto:undertow-dev@lists.__jboss.org
> >>>>> > > > >>
<mailto:undertow-dev@lists.jboss.org>>
> >>>>> > > > >> > Sent: Monday, 19 January,
2015 11:13:21 AM
> >>>>> > > > >> > Subject: Re:
[undertow-dev] occasional 502 from
> >>>>> Apache HTTP
> >>>>> > > > >> Proxy
> >>>>> > > > >> >
> >>>>> > > > >> > found this in apache:
> >>>>> > > > >> >
> >>>>> > > > >> > (104) Connection reset by
peer: proxy: error
> >>>>> reading status
> >>>>> > > > >> line from
> >>>>> > > > >> > remote server
> >>>>> > > > >> >
> >>>>> > > > >> > but nothing in undertow.
> >>>>> > > > >> >
> >>>>> > > > >> > got 502 on HTTP GET.
Sometimes while calling the
> >>>>> home page
> >>>>> > > / or
> >>>>> > > > >> when page
> >>>>> > > > >> > loads OK, got 502 on page
resources (js, css,
> >>>>> images).
> >>>>> > > > >> >
> >>>>> > > > >> > I call startBlocking and
use the outputstream to
> >>>>> write the
> >>>>> > > > >> response, when
> >>>>> > > > >> > everything has been
written I call the
> >>>>> "outputstream.close"
> >>>>> > > > >> method. Do I
> >>>>> > > > >> > need to call
exchange.endExchange too?
> >>>>> > > > >> >
> >>>>> > > > >> > Thanks
> >>>>> > > > >> >
> >>>>> > > > >> > On Sun, Jan 18, 2015 at
8:52 PM, Stuart Douglas
> >>>>> > > > >> <sdouglas(a)redhat.com
<mailto:sdouglas@redhat.com>
> >>>>> > > > >> <mailto:sdouglas@redhat.com
<mailto:
> >>>>> sdouglas(a)redhat.com>>>
> >>>>> > > wrote:
> >>>>> > > > >> >
> >>>>> > > > >> > > Is there any info in
the log? Or is there any
> >>>>> specific
> >>>>> > > type
> >>>>> > > > >> of
> >>>>> > > > >> request
> >>>>> > > > >> > > that causes this?
> >>>>> > > > >> > >
> >>>>> > > > >> > > Stuart
> >>>>> > > > >> > >
> >>>>> > > > >> > >
> >>>>> > > > >> > > ----- Original
Message -----
> >>>>> > > > >> > > > From:
"Edgar Espina" <espina.edgar(a)gmail.com
> >>>>> > > > >>
<mailto:espina.edgar@gmail.com>
> >>>>> > > > >>
<mailto:espina.edgar@gmail.com <mailto:
> >>>>> espina.edgar(a)gmail.com
> >>>>> > > > >> >__>>
> >>>>> > > > >> > > > To:
undertow-dev(a)lists.jboss.org
> >>>>> > > > >>
<mailto:undertow-dev@lists.jboss.org>
> >>>>> > > > >>
<mailto:undertow-dev@lists.__jboss.org
> >>>>> > > > >>
<mailto:undertow-dev@lists.jboss.org>>
> >>>>> > > > >> > > > Sent: Monday,
19 January, 2015 9:42:19 AM
> >>>>> > > > >> > > > Subject:
[undertow-dev] occasional 502 from
> >>>>> Apache HTTP
> >>>>> > > > >> Proxy
> >>>>> > > > >> > > >
> >>>>> > > > >> > > > Hi,
> >>>>> > > > >> > > >
> >>>>> > > > >> > > > I've an
Undertow application behind apache
> >>>>> reverse
> >>>>> > > proxy,
> >>>>> > > > >> trying to load
> >>>>> > > > >> > > a
> >>>>> > > > >> > > > page displays
error 502 proxy error.
> >>>>> > > > >> > > >
> >>>>> > > > >> > > > Still
couldn't find why so I wonder if any of
> >>>>> you find
> >>>>> > > a
> >>>>> > > > >> similar problem
> >>>>> > > > >> > > with
> >>>>> > > > >> > > > Undertow and
Apache.
> >>>>> > > > >> > > >
> >>>>> > > > >> > > > Please note
this is our first app on top of
> >>>>> Undertow,
> >>>>> > > > >> existing apps
> >>>>> > > > >> > > running
> >>>>> > > > >> > > > on Tomcat/Jetty
are OK.
> >>>>> > > > >> > > >
> >>>>> > > > >> > > > Appreciate any
help.
> >>>>> > > > >> > > >
> >>>>> > > > >> > > > Thanks
> >>>>> > > > >> > > >
> >>>>> > > > >> > > > --
> >>>>> > > > >> > > > edgar
> >>>>> > > > >> > > >
> >>>>> > > > >> > > >
> >>>>> _________________________________________________
> >>>>> > > > >> > > > undertow-dev
mailing list
> >>>>> > > > >> > > >
undertow-dev(a)lists.jboss.org
> >>>>> > > > >>
<mailto:undertow-dev@lists.jboss.org>
> >>>>> > > > >>
<mailto:undertow-dev@lists.__jboss.org
> >>>>> > > > >>
<mailto:undertow-dev@lists.jboss.org>>
> >>>>> > > > >> > > >
> >>>>> > >
https://lists.jboss.org/__mailman/listinfo/undertow-dev
> >>>>> > > > >> <
> >>>>>
https://lists.jboss.org/mailman/listinfo/undertow-dev>
> >>>>> > > > >> > >
> >>>>> > > > >> >
> >>>>> > > > >> >
> >>>>> > > > >> >
> >>>>> > > > >> > --
> >>>>> > > > >> > edgar
> >>>>> > > > >> >
> >>>>> > > > >>
> >>>>> > > > >>
> >>>>> > > > >>
> >>>>> > > > >>
> >>>>> > > > >> --
> >>>>> > > > >> edgar
> >>>>> > > > >>
> >>>>> > > > >>
> >>>>> > > > >>
> >>>>> > > > >>
> >>>>> > > > >> --
> >>>>> > > > >> edgar
> >>>>> > > > >>
> >>>>> > > > >
> >>>>> > > >
> >>>>> > > >
> >>>>> > > > --
> >>>>> > > > edgar
> >>>>> > > >
> >>>>> > >
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> > --
> >>>>> > edgar
> >>>>> >
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> edgar
> >>>>
> >>>> _______________________________________________
> >>>> undertow-dev mailing list
> >>>> undertow-dev(a)lists.jboss.org
> >>>>
https://lists.jboss.org/mailman/listinfo/undertow-dev
> >>>>
> >>>
> >>>
> >>
> >>
> >> --
> >> edgar
> >>
> >
> >
>
>
> --
> edgar
>
--
edgar
_______________________________________________
undertow-dev mailing list
undertow-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/undertow-dev
--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat