we put nginx upfront undertow and everything works as expected.
so we have: apache -> nginx -> undertow :S
apache can't be removed it and as I said before existing apps running on
Tomcat/Jetty work without any issue
we need to move on with our project (wasted a lot of time on this) but I
will try capture the traffic with wireshark and shared it.
still, if you find out something will be helpful too.
On Wed, Jan 21, 2015 at 4:51 PM, Jason Greene <jason.greene(a)redhat.com>
wrote:
Would you be willing to capture the traffic with wireshark or
(tcpdump
using -s 65535) when the failure occurs? It would be helpful in diagnosing
the exact reason the connection is lost (tcp, http protocol issue, etc).
If you prefer not to post the dump on list, you can email us privately.
Thanks!
> On Jan 21, 2015, at 1:40 PM, Edgar Espina <espina.edgar(a)gmail.com>
wrote:
>
> Nope, it doesn't help.
>
> Stuart, you mentioned that:
> "For a normal graceful close Undertow should be sending Connection:close
headers."
>
> Is it mean that on a req/res cycle, Undertow should always set the
Connection response header with close or keep-alive?
>
> Thanks
>
>
>
>
> On Tue, Jan 20, 2015 at 5:07 PM, Stuart Douglas <sdouglas(a)redhat.com>
wrote:
>
>
> ----- Original Message -----
> > From: "Edgar Espina" <espina.edgar(a)gmail.com>
> > To: "Tomaž Cerar" <tomaz.cerar(a)gmail.com>
> > Cc: "Stuart Douglas" <sdouglas(a)redhat.com>,
undertow-dev(a)lists.jboss.org
> > Sent: Wednesday, 21 January, 2015 2:03:10 AM
> > Subject: Re: [undertow-dev] occasional 502 from Apache HTTP Proxy
> >
> > seems to be some sort of timeout, because everything work as expected,
but
> > then after waiting for a while and got again the 502.
>
> That is odd, as there is no timeout configured. What does your network
config between the LB and Undertow look like?
>
> Does enabling TCP keep alive (Options.KEEP_ALIVE) help?
>
> Stuart
>
>
> >
> > also, I did set the content-length for static resources and call
> > .endExchange after a 304 response, set a backlog of 1000 and 10000
too. But
> > it didn't help.
> >
> >
> >
> > On Mon, Jan 19, 2015 at 9:57 AM, Tomaž Cerar <tomaz.cerar(a)gmail.com>
wrote:
> >
> > > Maybe related to
> > >
http://stackoverflow.com/questions/169453/bad-gateway-502-error-with-apac...
> > >
> > > or
> > >
http://qnalist.com/questions/4502641/users-httpd-mod-proxy-ignores-incomp...
> > >
> > > On Mon, Jan 19, 2015 at 1:33 PM, Edgar Espina <
espina.edgar(a)gmail.com>
> > > wrote:
> > >
> > >> Sure, it is 2.2.15.
> > >>
> > >>
> > >>
> > >> On Mon, Jan 19, 2015 at 9:29 AM, Tomaž Cerar
<tomaz.cerar(a)gmail.com
>
> > >> wrote:
> > >>
> > >>> Edgar,
> > >>>
> > >>> what is the version of Apache and mod_proxy module you are using.
> > >>> maybe it will be easier to reproduce with exact version you
have..
> > >>>
> > >>> --
> > >>> tomaz
> > >>>
> > >>> On Mon, Jan 19, 2015 at 1:15 PM, Edgar Espina <
espina.edgar(a)gmail.com>
> > >>> wrote:
> > >>>
> > >>>> The source code can be found here:
> > >>>>
https://github.com/jooby-project/jooby
> > >>>>
> > >>>> Undertow related classes are here:
> > >>>>
https://github.com/jooby-project/jooby/tree/master/jooby/src/main/java/or...
> > >>>> .
> > >>>> Server is built here:
> > >>>>
https://github.com/jooby-project/jooby/blob/master/jooby/src/main/java/or...
> > >>>> Response is sent here:
> > >>>>
https://github.com/jooby-project/jooby/blob/master/jooby/src/main/java/or...
> > >>>>
> > >>>> My microweb-framework was built on top of Jetty, so for now I
used the
> > >>>> blocking API and follow more or less what we usually do with
Servlets
> > >>>> (acquire an outstream).
> > >>>>
> > >>>> I will review what I'm doing and try to figure it out what
is
going on
> > >>>> too.
> > >>>>
> > >>>> Thanks for your help, Stuart.
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>> On Mon, Jan 19, 2015 at 2:12 AM, Stuart Douglas <
sdouglas(a)redhat.com>
> > >>>> wrote:
> > >>>>
> > >>>>> My best guess as to what is happening is that Undertow
closing a
> > >>>>> connection after a request is done for some reason, and
the
apache
> > >>>>> attempts
> > >>>>> to re-use this connection without realising that it is
dead.
> > >>>>>
> > >>>>> In general this should not happen, Undertow should only
forcibly
close
> > >>>>> a connection if it knows that it is broken (e.g. a
content
length is
> > >>>>> set
> > >>>>> and the full amount of content is not written). For a
normal
graceful
> > >>>>> close
> > >>>>> Undertow should be sending Connection:close headers.
> > >>>>>
> > >>>>> I am going to investigate some more, and see if I can
figure out
what
> > >>>>> is going on. It seems unlikely but is there any chance
your code
> > >>>>> forcibly
> > >>>>> closes the ServerConnection
(HttpServerExchange.getConnection())
> > >>>>> because
> > >>>>> that could potentially cause this issue. Setting the
exchange to
> > >>>>> non-persistent after headers have been sent could also
cause it.
> > >>>>>
> > >>>>> Stuart
> > >>>>>
> > >>>>> ----- Original Message -----
> > >>>>> > From: "Edgar Espina"
<espina.edgar(a)gmail.com>
> > >>>>> > To: "Stuart Douglas"
<sdouglas(a)redhat.com>
> > >>>>> > Cc: undertow-dev(a)lists.jboss.org
> > >>>>> > Sent: Monday, 19 January, 2015 2:53:06 PM
> > >>>>> > Subject: Re: [undertow-dev] occasional 502 from
Apache HTTP
Proxy
> > >>>>> >
> > >>>>> > Done, but makes no difference. Still got the 502
random errors
:S
> > >>>>> >
> > >>>>> > On Sun, Jan 18, 2015 at 11:13 PM, Stuart Douglas
<
> > >>>>> sdouglas(a)redhat.com>
> > >>>>> > wrote:
> > >>>>> >
> > >>>>> > > I had a play around with apache locally, and I
could
reproduce
> > >>>>> this in
> > >>>>> > > some circumstances, and it looks like we are not
setting a
high
> > >>>>> enough
> > >>>>> > > backlog by default.
> > >>>>> > >
> > >>>>> > > Can you try adding:
> > >>>>> > >
> > >>>>> > > undertow.setSocketOption(Options.BACKLOG, 1000)
> > >>>>> > >
> > >>>>> > > To your Undertow builder? I am going to increase
this in our
> > >>>>> default
> > >>>>> > > config upstream.
> > >>>>> > >
> > >>>>> > > Hopefully this is the issue that you are running
into.
> > >>>>> > >
> > >>>>> > > Stuart
> > >>>>> > >
> > >>>>> > >
> > >>>>> > > ----- Original Message -----
> > >>>>> > > > From: "Edgar Espina"
<espina.edgar(a)gmail.com>
> > >>>>> > > > To: "Stuart Douglas"
<sdouglas(a)redhat.com>
> > >>>>> > > > Cc: undertow-dev(a)lists.jboss.org
> > >>>>> > > > Sent: Monday, 19 January, 2015 12:10:16 PM
> > >>>>> > > > Subject: Re: [undertow-dev] occasional 502
from Apache
HTTP Proxy
> > >>>>> > > >
> > >>>>> > > > It is a default undertow instance with a
HTTP listener,
just set
> > >>>>> work
> > >>>>> > > > threads to 200. Is there a default idle
timeout? I can'
tell
> > >>>>> from javadoc
> > >>>>> > > > and looking at the code.
> > >>>>> > > >
> > >>>>> > > >
> > >>>>> > > > On Sun, Jan 18, 2015 at 9:59 PM, Stuart
Douglas <
> > >>>>> sdouglas(a)redhat.com>
> > >>>>> > > wrote:
> > >>>>> > > >
> > >>>>> > > > >
> > >>>>> > > > >
> > >>>>> > > > > Edgar Espina wrote:
> > >>>>> > > > >
> > >>>>> > > > >> I'm not, channel does it.
> > >>>>> > > > >>
> > >>>>> > > > >> Problem is present on both. For
example, home page has a
> > >>>>> > > content-length
> > >>>>> > > > >> header but jquery.js use chunked.
> > >>>>> > > > >>
> > >>>>> > > > >
> > >>>>> > > > > In general if you are serving static
resources you are
better
> > >>>>> off
> > >>>>> > > setting
> > >>>>> > > > > the content length (slightly more
efficient, and the
browser
> > >>>>> can
> > >>>>> > > display
> > >>>>> > > > > progress on downloads).
> > >>>>> > > > >
> > >>>>> > > > >
> > >>>>> > > > >> Do I have to call .endExchange? or
closing the output
stream
> > >>>>> is
> > >>>>> > > enough?
> > >>>>> > > > >>
> > >>>>> > > > >
> > >>>>> > > > > Closing the output stream is enough.
endExchange is
> > >>>>> automatically
> > >>>>> > > called
> > >>>>> > > > > once the call stack returns anyway,
unless you have
dispatched
> > >>>>> the
> > >>>>> > > exchange
> > >>>>> > > > > or started async IO.
> > >>>>> > > > >
> > >>>>> > > > > The 'connection reset by peer'
error in the apache log
while
> > >>>>> reading
> > >>>>> > > the
> > >>>>> > > > > status like kinda indicates that the
request does not
even get
> > >>>>> to this
> > >>>>> > > > > point anyway, and the underlying TCP
connection is
probably
> > >>>>> being torn
> > >>>>> > > down
> > >>>>> > > > > somehow.
> > >>>>> > > > >
> > >>>>> > > > > Do you have any kind of timeouts set?
If you have an idle
> > >>>>> timeout set
> > >>>>> > > on
> > >>>>> > > > > the listener there is a race where
Undertow can close the
> > >>>>> channel due
> > >>>>> > > to
> > >>>>> > > > > inactivity just as the front end
starts to send a
request.
> > >>>>> > > > >
> > >>>>> > > > > Stuart
> > >>>>> > > > >
> > >>>>> > > > >
> > >>>>> > > > >> Thanks
> > >>>>> > > > >>
> > >>>>> > > > >> On Sun, Jan 18, 2015 at 9:48 PM,
Stuart Douglas <
> > >>>>> sdouglas(a)redhat.com
> > >>>>> > > > >>
<mailto:sdouglas@redhat.com>> wrote:
> > >>>>> > > > >>
> > >>>>> > > > >> Another question, are you
setting a content length
on the
> > >>>>> > > responses?
> > >>>>> > > > >> If not the channel will
automatically set one if the
> > >>>>> response fits
> > >>>>> > > > >> inside a buffer, otherwise
chunked encoding will be
used.
> > >>>>> > > > >>
> > >>>>> > > > >> It might be helpful to know if
this only happens on
> > >>>>> chunked, fixed
> > >>>>> > > > >> length or both.
> > >>>>> > > > >>
> > >>>>> > > > >> Stuart
> > >>>>> > > > >>
> > >>>>> > > > >> Edgar Espina wrote:
> > >>>>> > > > >>
> > >>>>> > > > >> latest: 1.2.0.Beta8
> > >>>>> > > > >>
> > >>>>> > > > >> On Sun, Jan 18, 2015 at
9:42 PM, Stuart Douglas
> > >>>>> > > > >> <sdouglas(a)redhat.com
<mailto:
sdouglas(a)redhat.com>
> > >>>>> > > > >>
<mailto:sdouglas@redhat.com <mailto:
> > >>>>> sdouglas(a)redhat.com>>>
> > >>>>> > > wrote:
> > >>>>> > > > >>
> > >>>>> > > > >> Also what version of
Undertow are you
using?
> > >>>>> > > > >>
> > >>>>> > > > >> Stuart
> > >>>>> > > > >>
> > >>>>> > > > >> ----- Original
Message -----
> > >>>>> > > > >> > From: "Edgar
Espina" <
espina.edgar(a)gmail.com
> > >>>>> > > > >>
<mailto:espina.edgar@gmail.com>
> > >>>>> > > > >>
<mailto:espina.edgar@gmail.com <mailto:
> > >>>>> espina.edgar(a)gmail.com
> > >>>>> > > > >> >__>>
> > >>>>> > > > >> > To: "Stuart
Douglas" <sdouglas(a)redhat.com
> > >>>>> > > > >>
<mailto:sdouglas@redhat.com>
> > >>>>> > > > >>
<mailto:sdouglas@redhat.com <mailto:
> > >>>>> sdouglas(a)redhat.com>>>
> > >>>>> > > > >> > Cc:
undertow-dev(a)lists.jboss.org
> > >>>>> > > > >>
<mailto:undertow-dev@lists.jboss.org>
> > >>>>> > > > >>
<mailto:undertow-dev@lists.__jboss.org
> > >>>>> > > > >>
<mailto:undertow-dev@lists.jboss.org>>
> > >>>>> > > > >> > Sent: Monday, 19
January, 2015 11:13:21 AM
> > >>>>> > > > >> > Subject: Re:
[undertow-dev] occasional 502
from
> > >>>>> Apache HTTP
> > >>>>> > > > >> Proxy
> > >>>>> > > > >> >
> > >>>>> > > > >> > found this in
apache:
> > >>>>> > > > >> >
> > >>>>> > > > >> > (104) Connection
reset by peer: proxy: error
> > >>>>> reading status
> > >>>>> > > > >> line from
> > >>>>> > > > >> > remote server
> > >>>>> > > > >> >
> > >>>>> > > > >> > but nothing in
undertow.
> > >>>>> > > > >> >
> > >>>>> > > > >> > got 502 on HTTP GET.
Sometimes while calling
the
> > >>>>> home page
> > >>>>> > > / or
> > >>>>> > > > >> when page
> > >>>>> > > > >> > loads OK, got 502 on
page resources (js, css,
> > >>>>> images).
> > >>>>> > > > >> >
> > >>>>> > > > >> > I call startBlocking
and use the
outputstream to
> > >>>>> write the
> > >>>>> > > > >> response, when
> > >>>>> > > > >> > everything has been
written I call the
> > >>>>> "outputstream.close"
> > >>>>> > > > >> method. Do I
> > >>>>> > > > >> > need to call
exchange.endExchange too?
> > >>>>> > > > >> >
> > >>>>> > > > >> > Thanks
> > >>>>> > > > >> >
> > >>>>> > > > >> > On Sun, Jan 18, 2015
at 8:52 PM, Stuart
Douglas
> > >>>>> > > > >> <sdouglas(a)redhat.com
<mailto:
sdouglas(a)redhat.com>
> > >>>>> > > > >>
<mailto:sdouglas@redhat.com <mailto:
> > >>>>> sdouglas(a)redhat.com>>>
> > >>>>> > > wrote:
> > >>>>> > > > >> >
> > >>>>> > > > >> > > Is there any
info in the log? Or is there
any
> > >>>>> specific
> > >>>>> > > type
> > >>>>> > > > >> of
> > >>>>> > > > >> request
> > >>>>> > > > >> > > that causes
this?
> > >>>>> > > > >> > >
> > >>>>> > > > >> > > Stuart
> > >>>>> > > > >> > >
> > >>>>> > > > >> > >
> > >>>>> > > > >> > > ----- Original
Message -----
> > >>>>> > > > >> > > > From:
"Edgar Espina" <
espina.edgar(a)gmail.com
> > >>>>> > > > >>
<mailto:espina.edgar@gmail.com>
> > >>>>> > > > >>
<mailto:espina.edgar@gmail.com <mailto:
> > >>>>> espina.edgar(a)gmail.com
> > >>>>> > > > >> >__>>
> > >>>>> > > > >> > > > To:
undertow-dev(a)lists.jboss.org
> > >>>>> > > > >>
<mailto:undertow-dev@lists.jboss.org>
> > >>>>> > > > >>
<mailto:undertow-dev@lists.__jboss.org
> > >>>>> > > > >>
<mailto:undertow-dev@lists.jboss.org>>
> > >>>>> > > > >> > > > Sent:
Monday, 19 January, 2015 9:42:19 AM
> > >>>>> > > > >> > > > Subject:
[undertow-dev] occasional 502
from
> > >>>>> Apache HTTP
> > >>>>> > > > >> Proxy
> > >>>>> > > > >> > > >
> > >>>>> > > > >> > > > Hi,
> > >>>>> > > > >> > > >
> > >>>>> > > > >> > > > I've
an Undertow application behind
apache
> > >>>>> reverse
> > >>>>> > > proxy,
> > >>>>> > > > >> trying to load
> > >>>>> > > > >> > > a
> > >>>>> > > > >> > > > page
displays error 502 proxy error.
> > >>>>> > > > >> > > >
> > >>>>> > > > >> > > > Still
couldn't find why so I wonder if
any of
> > >>>>> you find
> > >>>>> > > a
> > >>>>> > > > >> similar problem
> > >>>>> > > > >> > > with
> > >>>>> > > > >> > > > Undertow
and Apache.
> > >>>>> > > > >> > > >
> > >>>>> > > > >> > > > Please
note this is our first app on top
of
> > >>>>> Undertow,
> > >>>>> > > > >> existing apps
> > >>>>> > > > >> > > running
> > >>>>> > > > >> > > > on
Tomcat/Jetty are OK.
> > >>>>> > > > >> > > >
> > >>>>> > > > >> > > > Appreciate
any help.
> > >>>>> > > > >> > > >
> > >>>>> > > > >> > > > Thanks
> > >>>>> > > > >> > > >
> > >>>>> > > > >> > > > --
> > >>>>> > > > >> > > > edgar
> > >>>>> > > > >> > > >
> > >>>>> > > > >> > > >
> > >>>>> _________________________________________________
> > >>>>> > > > >> > > >
undertow-dev mailing list
> > >>>>> > > > >> > > >
undertow-dev(a)lists.jboss.org
> > >>>>> > > > >>
<mailto:undertow-dev@lists.jboss.org>
> > >>>>> > > > >>
<mailto:undertow-dev@lists.__jboss.org
> > >>>>> > > > >>
<mailto:undertow-dev@lists.jboss.org>>
> > >>>>> > > > >> > > >
> > >>>>> > >
https://lists.jboss.org/__mailman/listinfo/undertow-dev
> > >>>>> > > > >> <
> > >>>>>
https://lists.jboss.org/mailman/listinfo/undertow-dev>
> > >>>>> > > > >> > >
> > >>>>> > > > >> >
> > >>>>> > > > >> >
> > >>>>> > > > >> >
> > >>>>> > > > >> > --
> > >>>>> > > > >> > edgar
> > >>>>> > > > >> >
> > >>>>> > > > >>
> > >>>>> > > > >>
> > >>>>> > > > >>
> > >>>>> > > > >>
> > >>>>> > > > >> --
> > >>>>> > > > >> edgar
> > >>>>> > > > >>
> > >>>>> > > > >>
> > >>>>> > > > >>
> > >>>>> > > > >>
> > >>>>> > > > >> --
> > >>>>> > > > >> edgar
> > >>>>> > > > >>
> > >>>>> > > > >
> > >>>>> > > >
> > >>>>> > > >
> > >>>>> > > > --
> > >>>>> > > > edgar
> > >>>>> > > >
> > >>>>> > >
> > >>>>> >
> > >>>>> >
> > >>>>> >
> > >>>>> > --
> > >>>>> > edgar
> > >>>>> >
> > >>>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> edgar
> > >>>>
> > >>>> _______________________________________________
> > >>>> undertow-dev mailing list
> > >>>> undertow-dev(a)lists.jboss.org
> > >>>>
https://lists.jboss.org/mailman/listinfo/undertow-dev
> > >>>>
> > >>>
> > >>>
> > >>
> > >>
> > >> --
> > >> edgar
> > >>
> > >
> > >
> >
> >
> > --
> > edgar
> >
>
>
>
> --
> edgar
> _______________________________________________
> undertow-dev mailing list
> undertow-dev(a)lists.jboss.org
>
https://lists.jboss.org/mailman/listinfo/undertow-dev
--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat