[undertow-dev] Question about APR based connector in Undertow

[Jason Greene]

We have no intention to do an APR connector. We have had some occasional discussions about whether or not we will need openssl support. Perhaps that’s where it is coming from?

JFC and Remy had some numbers on JBossWeb that showed poor perf on JSSE vs OpenSSL using APR. It’s somewhat apples vs oranges but looking at the JSSE impl we do see a number of inefficiencies. As an example JSSE seems to do a lot of copying (e.g. anything in a buffer becomes many byte arrays which gets copied again to a buffer). We also see that SHA support on intel is still in a big chunk of java code, and not intrinsic, wheras on openssl its optimized assembler. The new AES GCM ciphers are also known to be slow, but Andrew Haley mentioned that he is in the process of optimizing those. All of this is of course, anecdotal, and until we spend some cycles doing a more apples to apples test I don’t think we can conclude this is a necessary direction.

The other case where this might matter is how we find a way to support HTTP2. HTTP2 requires that we support the ALPN extension to TLS. Unfortunately JSSE has no support for it, and has been rather slow to implement, and it has to come into 9 before there is any possibility of a backport to 8:

http://openjdk.java.net/jeps/244
https://bugs.openjdk.java.net/browse/JDK-8051498 <https://bugs.openjdk.java.net/browse/JDK-8051498>

In the meantime we have been reusing a hack that the Jetty guys are maintaining as a separate project, and it requires using bootclasspath to replace core JSSE classes. These hacks are not z-stream compatible, so any JDK bug fix update can break the hack (and this has already happened about 2-3 times).

So on that perspective we have three options:

1)  Red Hat does a Red Hat only hack for IcedTea distributions (not sure if the JDK team would be willing to maintain it or not, would have to ask)
2)  Introduce openssl support, which supports ALPN
3)  Wait for JSSE, and live with the hack (warning people appropriately)

At this point, I don’t think we can commit to having anything for the WildFly 10 schedule, unless the performance difference really mandates such a thing, and then we might have to shuffle things around. 

> On Jun 15, 2015, at 12:29 PM, Andrig T. Miller <anmiller at redhat.com> wrote:
> 
> I am hearing that some work is being put into an APR based connector for Undertow.  I am wondering why we are doing this?
> 
> Based on my own testing, and testing the performance team has done with AS 7.x/EAP 6.x/Wildfly, I see no performance reason to do it, and it creates more work for getting product out the door, and maintaining product.
> 
> We have found that the APR based connector can exhibit very poor scalability, and consume large amounts of CPU resources when in a network environment that is having problems (TCP resets, dropped and retransmitted packets, etc.).
> 
> In this situation, especially where we see these poor network conditions, the APR connector spends a lot of its time managing the connection data structure.  That data structure is a ring (doubly linked list with head and tail pointing at each other, which you already know).  That ring is sequentially searched for the socket to remove and it locks all access with a mutex before it does this.  This causes both a CPU consumption issue, as it sequentially walks the linked list, and a blocking problem.  When we first discovered this, I went into the code to see how easy it would be to change it.  I don't think this is a simple fix, and since APR is the underpinning of other things, like Apache HTTPD, not a trivial thing to get accepted by the community either.  This spurred us to look at the NIO2 based connector, which at the time was unsupported, and it didn't exhibit any of this poor behavior.  That's what spurred the effort to move the NIO2 based connector to be fully supported.
> 
> In the absence of poor network conditions, the APR connector behaves and performs well, but in the presence of poor network conditions its not good.  So, with a viable alternative, which Undertow provides out of the box with its XNIO based architecture, there simple seems no good reason to do with for Undertow.
> 
> Thanks.
> 
> -- 
> Andrig (Andy) T. Miller
> Global Platform Director, JBoss Middleware
> Red Hat, Inc.
> _______________________________________________
> undertow-dev mailing list
> undertow-dev at lists.jboss.org <mailto:undertow-dev at lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/undertow-dev <https://lists.jboss.org/mailman/listinfo/undertow-dev>
--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/undertow-dev/attachments/20150615/7b5845e1/attachment.html