12:29 p.m.
I am hearing that some work is being put into an APR based connector for Undertow. I am
wondering why we are doing this?
Based on my own testing, and testing the performance team has done with AS 7.x/EAP
6.x/Wildfly, I see no performance reason to do it, and it creates more work for getting
product out the door, and maintaining product.
We have found that the APR based connector can exhibit very poor scalability, and consume
large amounts of CPU resources when in a network environment that is having problems (TCP
resets, dropped and retransmitted packets, etc.).
In this situation, especially where we see these poor network conditions, the APR
connector spends a lot of its time managing the connection data structure. That data
structure is a ring (doubly linked list with head and tail pointing at each other, which
you already know). That ring is sequentially searched for the socket to remove and it
locks all access with a mutex before it does this. This causes both a CPU consumption
issue, as it sequentially walks the linked list, and a blocking problem. When we first
discovered this, I went into the code to see how easy it would be to change it. I
don't think this is a simple fix, and since APR is the underpinning of other things,
like Apache HTTPD, not a trivial thing to get accepted by the community either. This
spurred us to look at the NIO2 based connector, which at the time was unsupported, and it
didn't exhibit any of this poor behavior. That's what spurred the effort to move
the NIO2 based connector to be fully supported.
In the absence of poor network conditions, the APR connector behaves and performs well,
but in the presence of poor network conditions its not good. So, with a viable
alternative, which Undertow provides out of the box with its XNIO based architecture,
there simple seems no good reason to do with for Undertow.
Thanks.
--
Andrig (Andy) T. Miller
Global Platform Director, JBoss Middleware
Red Hat, Inc.
12:51 p.m.
We have no intention to do an APR connector. We have had some occasional discussions about
whether or not we will need openssl support. Perhaps that’s where it is coming from?
JFC and Remy had some numbers on JBossWeb that showed poor perf on JSSE vs OpenSSL using
APR. It’s somewhat apples vs oranges but looking at the JSSE impl we do see a number of
inefficiencies. As an example JSSE seems to do a lot of copying (e.g. anything in a buffer
becomes many byte arrays which gets copied again to a buffer). We also see that SHA
support on intel is still in a big chunk of java code, and not intrinsic, wheras on
openssl its optimized assembler. The new AES GCM ciphers are also known to be slow, but
Andrew Haley mentioned that he is in the process of optimizing those. All of this is of
course, anecdotal, and until we spend some cycles doing a more apples to apples test I
don’t think we can conclude this is a necessary direction.
The other case where this might matter is how we find a way to support HTTP2. HTTP2
requires that we support the ALPN extension to TLS. Unfortunately JSSE has no support for
it, and has been rather slow to implement, and it has to come into 9 before there is any
possibility of a backport to 8:
http://openjdk.java.net/jeps/244
https://bugs.openjdk.java.net/browse/JDK-8051498
<https://bugs.openjdk.java.net/browse/JDK-8051498>
In the meantime we have been reusing a hack that the Jetty guys are maintaining as a
separate project, and it requires using bootclasspath to replace core JSSE classes. These
hacks are not z-stream compatible, so any JDK bug fix update can break the hack (and this
has already happened about 2-3 times).
So on that perspective we have three options:
1) Red Hat does a Red Hat only hack for IcedTea distributions (not sure if the JDK team
would be willing to maintain it or not, would have to ask)
2) Introduce openssl support, which supports ALPN
3) Wait for JSSE, and live with the hack (warning people appropriately)
At this point, I don’t think we can commit to having anything for the WildFly 10 schedule,
unless the performance difference really mandates such a thing, and then we might have to
shuffle things around.
On Jun 15, 2015, at 12:29 PM, Andrig T. Miller
<anmiller(a)redhat.com> wrote:
I am hearing that some work is being put into an APR based connector for Undertow. I am
wondering why we are doing this?
Based on my own testing, and testing the performance team has done with AS 7.x/EAP
6.x/Wildfly, I see no performance reason to do it, and it creates more work for getting
product out the door, and maintaining product.
We have found that the APR based connector can exhibit very poor scalability, and consume
large amounts of CPU resources when in a network environment that is having problems (TCP
resets, dropped and retransmitted packets, etc.).
In this situation, especially where we see these poor network conditions, the APR
connector spends a lot of its time managing the connection data structure. That data
structure is a ring (doubly linked list with head and tail pointing at each other, which
you already know). That ring is sequentially searched for the socket to remove and it
locks all access with a mutex before it does this. This causes both a CPU consumption
issue, as it sequentially walks the linked list, and a blocking problem. When we first
discovered this, I went into the code to see how easy it would be to change it. I
don't think this is a simple fix, and since APR is the underpinning of other things,
like Apache HTTPD, not a trivial thing to get accepted by the community either. This
spurred us to look at the NIO2 based connector, which at the time was unsupported, and it
didn't exhibit any of this poor behavior. That's what spurred the effort to move
the NIO2 based connector to be fully supported.
In the absence of poor network conditions, the APR connector behaves and performs well,
but in the presence of poor network conditions its not good. So, with a viable
alternative, which Undertow provides out of the box with its XNIO based architecture,
there simple seems no good reason to do with for Undertow.
Thanks.
--
Andrig (Andy) T. Miller
Global Platform Director, JBoss Middleware
Red Hat, Inc.
_______________________________________________
undertow-dev mailing list
undertow-dev(a)lists.jboss.org <mailto:undertow-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/undertow-dev
<https://lists.jboss.org/mailman/listinfo/undertow-dev>
--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat
1:01 p.m.
Oh and I forgot to mention that IMO I think we should prefer contributing optimizations
back to the JDK if at all possible, and if not it should be the long term goal.
On Jun 15, 2015, at 12:51 PM, Jason Greene
<jason.greene(a)redhat.com> wrote:
We have no intention to do an APR connector. We have had some occasional discussions
about whether or not we will need openssl support. Perhaps that’s where it is coming
from?
JFC and Remy had some numbers on JBossWeb that showed poor perf on JSSE vs OpenSSL using
APR. It’s somewhat apples vs oranges but looking at the JSSE impl we do see a number of
inefficiencies. As an example JSSE seems to do a lot of copying (e.g. anything in a buffer
becomes many byte arrays which gets copied again to a buffer). We also see that SHA
support on intel is still in a big chunk of java code, and not intrinsic, wheras on
openssl its optimized assembler. The new AES GCM ciphers are also known to be slow, but
Andrew Haley mentioned that he is in the process of optimizing those. All of this is of
course, anecdotal, and until we spend some cycles doing a more apples to apples test I
don’t think we can conclude this is a necessary direction.
The other case where this might matter is how we find a way to support HTTP2. HTTP2
requires that we support the ALPN extension to TLS. Unfortunately JSSE has no support for
it, and has been rather slow to implement, and it has to come into 9 before there is any
possibility of a backport to 8:
http://openjdk.java.net/jeps/244 <http://openjdk.java.net/jeps/244>
https://bugs.openjdk.java.net/browse/JDK-8051498
<https://bugs.openjdk.java.net/browse/JDK-8051498>
In the meantime we have been reusing a hack that the Jetty guys are maintaining as a
separate project, and it requires using bootclasspath to replace core JSSE classes. These
hacks are not z-stream compatible, so any JDK bug fix update can break the hack (and this
has already happened about 2-3 times).
So on that perspective we have three options:
1) Red Hat does a Red Hat only hack for IcedTea distributions (not sure if the JDK team
would be willing to maintain it or not, would have to ask)
2) Introduce openssl support, which supports ALPN
3) Wait for JSSE, and live with the hack (warning people appropriately)
At this point, I don’t think we can commit to having anything for the WildFly 10
schedule, unless the performance difference really mandates such a thing, and then we
might have to shuffle things around.
> On Jun 15, 2015, at 12:29 PM, Andrig T. Miller <anmiller(a)redhat.com
<mailto:anmiller@redhat.com>> wrote:
>
> I am hearing that some work is being put into an APR based connector for Undertow. I
am wondering why we are doing this?
>
> Based on my own testing, and testing the performance team has done with AS 7.x/EAP
6.x/Wildfly, I see no performance reason to do it, and it creates more work for getting
product out the door, and maintaining product.
>
> We have found that the APR based connector can exhibit very poor scalability, and
consume large amounts of CPU resources when in a network environment that is having
problems (TCP resets, dropped and retransmitted packets, etc.).
>
> In this situation, especially where we see these poor network conditions, the APR
connector spends a lot of its time managing the connection data structure. That data
structure is a ring (doubly linked list with head and tail pointing at each other, which
you already know). That ring is sequentially searched for the socket to remove and it
locks all access with a mutex before it does this. This causes both a CPU consumption
issue, as it sequentially walks the linked list, and a blocking problem. When we first
discovered this, I went into the code to see how easy it would be to change it. I
don't think this is a simple fix, and since APR is the underpinning of other things,
like Apache HTTPD, not a trivial thing to get accepted by the community either. This
spurred us to look at the NIO2 based connector, which at the time was unsupported, and it
didn't exhibit any of this poor behavior. That's what spurred the effort to move
the NIO2 based connector to be fully supported.
>
> In the absence of poor network conditions, the APR connector behaves and performs
well, but in the presence of poor network conditions its not good. So, with a viable
alternative, which Undertow provides out of the box with its XNIO based architecture,
there simple seems no good reason to do with for Undertow.
>
> Thanks.
>
> --
> Andrig (Andy) T. Miller
> Global Platform Director, JBoss Middleware
> Red Hat, Inc.
> _______________________________________________
> undertow-dev mailing list
> undertow-dev(a)lists.jboss.org <mailto:undertow-dev@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/undertow-dev
<https://lists.jboss.org/mailman/listinfo/undertow-dev>
--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat
_______________________________________________
undertow-dev mailing list
undertow-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/undertow-dev
--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat
9:08 a.m.
Okay. I don't have a problem with the openssl part of things, just didn't want to
see effort expended on APR, as it was mentioned in a different e-mail thread.
Thanks Jason.
Andy
----- Original Message -----
From: "Jason Greene" <jason.greene(a)redhat.com>
To: "Andrig T. Miller" <anmiller(a)redhat.com>
Cc: "undertow-dev" <undertow-dev(a)lists.jboss.org>
Sent: Monday, June 15, 2015 12:01:42 PM
Subject: Re: [undertow-dev] Question about APR based connector in
Undertow
Oh and I forgot to mention that IMO I think we should prefer
contributing optimizations back to the JDK if at all possible, and
if not it should be the long term goal.
> On Jun 15, 2015, at 12:51 PM, Jason Greene <
> jason.greene(a)redhat.com
> > wrote:
> We have no intention to do an APR connector. We have had some
> occasional discussions about whether or not we will need openssl
> support. Perhaps that’s where it is coming from?
> JFC and Remy had some numbers on JBossWeb that showed poor perf
on
> JSSE vs OpenSSL using APR. It’s somewhat apples vs oranges but
> looking at the JSSE impl we do see a number of inefficiencies. As
> an
> example JSSE seems to do a lot of copying (e.g. anything in a
> buffer
> becomes many byte arrays which gets copied again to a buffer). We
> also see that SHA support on intel is still in a big chunk of java
> code, and not intrinsic, wheras on openssl its optimized assembler.
> The new AES GCM ciphers are also known to be slow, but Andrew Haley
> mentioned that he is in the process of optimizing those. All of
> this
> is of course, anecdotal, and until we spend some cycles doing a
> more
> apples to apples test I don’t think we can conclude this is a
> necessary direction.
> The other case where this might matter is how we find a way to
> support HTTP2. HTTP2 requires that we support the ALPN extension to
> TLS. Unfortunately JSSE has no support for it, and has been rather
> slow to implement, and it has to come into 9 before there is any
> possibility of a backport to 8:
> In the meantime we have been reusing a hack that the Jetty guys
are
> maintaining as a separate project, and it requires using
> bootclasspath to replace core JSSE classes. These hacks are not
> z-stream compatible, so any JDK bug fix update can break the hack
> (and this has already happened about 2-3 times).
> So on that perspective we have three options:
> 1) Red Hat does a Red Hat only hack for IcedTea distributions
(not
> sure if the JDK team would be willing to maintain it or not, would
> have to ask)
> 2) Introduce openssl support, which supports ALPN
> 3) Wait for JSSE, and live with the hack (warning people
> appropriately)
> At this point, I don’t think we can commit to having anything
for
> the
> WildFly 10 schedule, unless the performance difference really
> mandates such a thing, and then we might have to shuffle things
> around.
> > On Jun 15, 2015, at 12:29 PM, Andrig T. Miller <
> > anmiller(a)redhat.com
> > > wrote:
>
> > I am hearing that some work is being put into an APR based
> > connector
> > for Undertow. I am wondering why we are doing this?
>
> > Based on my own testing, and testing the performance team
has
> > done
> > with AS 7.x/EAP 6.x/Wildfly, I see no performance reason to do
> > it,
> > and it creates more work for getting product out the door, and
> > maintaining product.
>
> > We have found that the APR based connector can exhibit very
poor
> > scalability, and consume large amounts of CPU resources when in a
> > network environment that is having problems (TCP resets, dropped
> > and
> > retransmitted packets, etc.).
>
> > In this situation, especially where we see these poor
network
> > conditions, the APR connector spends a lot of its time managing
> > the
> > connection data structure. That data structure is a ring (doubly
> > linked list with head and tail pointing at each other, which you
> > already know). That ring is sequentially searched for the socket
> > to
> > remove and it locks all access with a mutex before it does this.
> > This causes both a CPU consumption issue, as it sequentially
> > walks
> > the linked list, and a blocking problem. When we first discovered
> > this, I went into the code to see how easy it would be to change
> > it.
> > I don't think this is a simple fix, and since APR is the
> > underpinning of other things, like Apache HTTPD, not a trivial
> > thing
> > to get accepted by the community either. This spurred us to look
> > at
> > the NIO2 based connector, which at the time was unsupported, and
> > it
> > didn't exhibit any of this poor behavior. That's what spurred the
> > effort to move the NIO2 based connector to be fully supported.
>
> > In the absence of poor network conditions, the APR
connector
> > behaves
> > and performs well, but in the presence of poor network conditions
> > its not good. So, with a viable alternative, which Undertow
> > provides
> > out of the box with its XNIO based architecture, there simple
> > seems
> > no good reason to do with for Undertow.
>
> > Thanks.
>
> > --
>
> > Andrig (Andy) T. Miller
>
> > Global Platform Director, JBoss Middleware
>
> > Red Hat, Inc.
>
> > _______________________________________________
>
> > undertow-dev mailing list
>
> > undertow-dev(a)lists.jboss.org
>
> > https://lists.jboss.org/mailman/listinfo/undertow-dev
>
> --
> Jason T. Greene
> WildFly Lead / JBoss EAP Platform Architect
> JBoss, a division of Red Hat
> _______________________________________________
> undertow-dev mailing list
> undertow-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/undertow-dev
--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat
11:23 a.m.
On 06/16/2015 04:08 PM, Andrig T. Miller wrote:
Okay. I don't have a problem with the openssl part of things,
just
didn't want to see effort expended on APR, as it was mentioned in a
different e-mail thread.
I think we are on the same page APR is useless without OpenSSL, the APR
connector won't make sense in undertow.
The new thing I have just merged in tc-native allows to use NIO/NIO2
with OpenSSL, Remy has the code for the NIO/NIO2 connectors of Tomcat
that should be portable to undertow.
Cheers
Jean-Frederic
Thanks Jason.
Andy
------------------------------------------------------------------------
*From: *"Jason Greene" <jason.greene(a)redhat.com>
*To: *"Andrig T. Miller" <anmiller(a)redhat.com>
*Cc: *"undertow-dev" <undertow-dev(a)lists.jboss.org>
*Sent: *Monday, June 15, 2015 12:01:42 PM
*Subject: *Re: [undertow-dev] Question about APR based connector in
Undertow
Oh and I forgot to mention that IMO I think we should prefer
contributing optimizations back to the JDK if at all possible, and
if not it should be the long term goal.
On Jun 15, 2015, at 12:51 PM, Jason Greene
<jason.greene(a)redhat.com <mailto:jason.greene@redhat.com>> wrote:
We have no intention to do an APR connector. We have had some
occasional discussions about whether or not we will need openssl
support. Perhaps that’s where it is coming from?
JFC and Remy had some numbers on JBossWeb that showed poor perf
on JSSE vs OpenSSL using APR. It’s somewhat apples vs oranges
but looking at the JSSE impl we do see a number of
inefficiencies. As an example JSSE seems to do a lot of copying
(e.g. anything in a buffer becomes many byte arrays which gets
copied again to a buffer). We also see that SHA support on intel
is still in a big chunk of java code, and not intrinsic, wheras
on openssl its optimized assembler. The new AES GCM ciphers are
also known to be slow, but Andrew Haley mentioned that he is in
the process of optimizing those. All of this is of course,
anecdotal, and until we spend some cycles doing a more apples to
apples test I don’t think we can conclude this is a necessary
direction.
The other case where this might matter is how we find a way to
support HTTP2. HTTP2 requires that we support the ALPN extension
to TLS. Unfortunately JSSE has no support for it, and has been
rather slow to implement, and it has to come into 9 before there
is any possibility of a backport to 8:
http://openjdk.java.net/jeps/244
https://bugs.openjdk.java.net/browse/JDK-8051498
In the meantime we have been reusing a hack that the Jetty guys
are maintaining as a separate project, and it requires using
bootclasspath to replace core JSSE classes. These hacks are not
z-stream compatible, so any JDK bug fix update can break the
hack (and this has already happened about 2-3 times).
So on that perspective we have three options:
1) Red Hat does a Red Hat only hack for IcedTea distributions
(not sure if the JDK team would be willing to maintain it or
not, would have to ask)
2) Introduce openssl support, which supports ALPN
3) Wait for JSSE, and live with the hack (warning people
appropriately)
At this point, I don’t think we can commit to having anything
for the WildFly 10 schedule, unless the performance difference
really mandates such a thing, and then we might have to shuffle
things around.
On Jun 15, 2015, at 12:29 PM, Andrig T. Miller
<anmiller(a)redhat.com <mailto:anmiller@redhat.com>> wrote:
I am hearing that some work is being put into an APR based
connector for Undertow. I am wondering why we are doing this?
Based on my own testing, and testing the performance team
has done with AS 7.x/EAP 6.x/Wildfly, I see no performance
reason to do it, and it creates more work for getting
product out the door, and maintaining product.
We have found that the APR based connector can exhibit very
poor scalability, and consume large amounts of CPU resources
when in a network environment that is having problems (TCP
resets, dropped and retransmitted packets, etc.).
In this situation, especially where we see these poor
network conditions, the APR connector spends a lot of its
time managing the connection data structure. That data
structure is a ring (doubly linked list with head and tail
pointing at each other, which you already know). That ring
is sequentially searched for the socket to remove and it
locks all access with a mutex before it does this. This
causes both a CPU consumption issue, as it sequentially
walks the linked list, and a blocking problem. When we
first discovered this, I went into the code to see how easy
it would be to change it. I don't think this is a simple
fix, and since APR is the underpinning of other things, like
Apache HTTPD, not a trivial thing to get accepted by the
community either. This spurred us to look at the NIO2 based
connector, which at the time was unsupported, and it didn't
exhibit any of this poor behavior. That's what spurred the
effort to move the NIO2 based connector to be fully supported.
In the absence of poor network conditions, the APR connector
behaves and performs well, but in the presence of poor
network conditions its not good. So, with a viable
alternative, which Undertow provides out of the box with its
XNIO based architecture, there simple seems no good reason
to do with for Undertow.
Thanks.
--
Andrig (Andy) T. Miller
Global Platform Director, JBoss Middleware
Red Hat, Inc.
_______________________________________________
undertow-dev mailing list
undertow-dev(a)lists.jboss.org
<mailto:undertow-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/undertow-dev
--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat
_______________________________________________
undertow-dev mailing list
undertow-dev(a)lists.jboss.org <mailto:undertow-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/undertow-dev
--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat
_______________________________________________
undertow-dev mailing list
undertow-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/undertow-dev
3249
days inactive
3252
days old
4 comments
3 participants
participants (3)
-
Andrig T. Miller -
Jason Greene -
jean-frederic clere