OpenSSL via Wildfly-OpenSSL crashes JVM when mod_cluster used ------------------------------------------------------------- Key: WFLY-8753 URL: https://issues.jboss.org/browse/WFLY-8753 Project: WildFly Issue Type: Bug Components: Security Affects Versions: 11.0.0.Beta1 Environment: RHEL 7 x86_64 Reporter: Michal Karm Babacek Assignee: Stuart Douglas Priority: Critical Labels: wildfly-openssl Attachments: crashes_on_mod_cluster_connection.server.log, hs_err_pid12281.log, new_hs_err_pid13002.log, server.log, standalone-ha.xml h3. Previously I thought it happens just on server shutdown... {quote} Worker node with this mod_cluster and Elytron configuration [^standalone-ha.xml] keeps complaining on being unable to agree on a cipher suite with the Undertow balancer, whose Elytron configuration is identical, i.e. {code} <server-ssl-contexts> <server-ssl-context name="serverSSLContext" providers="openssl" need-client-auth="false" key-managers="keyManager" protocols="TLSv1.2" trust-managers="trustManager"/> </server-ssl-contexts> <client-ssl-contexts> <client-ssl-context name="clientSSLContext" providers="openssl" key-managers="clientKeyManager" protocols="TLSv1.2" trust-managers="trustManager"/> </client-ssl-contexts>{code} see:{code} 15:19:25,492 FINE [org.wildfly.openssl.OpenSSLEngine] (UndertowEventHandlerAdapter - 1) The version of SSL in use does not support cipher ordering 15:19:25,541 ERROR [org.jboss.mod_cluster.undertow] (UndertowEventHandlerAdapter - 1) null: java.lang.NullPointerException at org.wildfly.openssl.CipherSuiteConverter.toJava(CipherSuiteConverter.java:284) at org.wildfly.openssl.OpenSSLEngine.toJavaCipherSuite(OpenSSLEngine.java:1021) at org.wildfly.openssl.OpenSSlSession.initCipherSuite(OpenSSlSession.java:305) at org.wildfly.openssl.OpenSSlSession.initialised(OpenSSlSession.java:296) at org.wildfly.openssl.OpenSSLSessionContext.clientSessionCreated(OpenSSLSessionContext.java:114) at org.wildfly.openssl.OpenSSLClientSessionContext.storeClientSideSession(OpenSSLClientSessionContext.java:92) at org.wildfly.openssl.OpenSSLEngine.handshakeFinished(OpenSSLEngine.java:940) at org.wildfly.openssl.OpenSSLEngine.getHandshakeStatus(OpenSSLEngine.java:989) at org.wildfly.openssl.OpenSSLEngine.unwrap(OpenSSLEngine.java:606) at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) at org.wildfly.openssl.OpenSSLSocket.runHandshake(OpenSSLSocket.java:316) at org.wildfly.openssl.OpenSSLSocket.write(OpenSSLSocket.java:462) at org.wildfly.openssl.OpenSSLOutputStream.write(OpenSSLOutputStream.java:46) at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291) at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295) at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141) at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229) at java.io.BufferedWriter.flush(BufferedWriter.java:254) at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequest(DefaultMCMPHandler.java:526) at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequest(DefaultMCMPHandler.java:605) at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.status(DefaultMCMPHandler.java:387) at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.status(DefaultMCMPHandler.java:365) at org.jboss.modcluster.ModClusterService.status(ModClusterService.java:454) at org.wildfly.mod_cluster.undertow.UndertowEventHandlerAdapter.run(UndertowEventHandlerAdapter.java:169) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) at org.jboss.threads.JBossThread.run(JBossThread.java:320) {code} Full log: [^server.log]. I might or might not have it misconfigured, but what definitely should not happen is that when I try to shut the server down: {code} INFO [org.jboss.as.server] (Thread-2) WFLYSRV0220: Server shutdown has been requested via an OS signal {code} it crashes JVM (doublefree?): {code} # JRE version: Java(TM) SE Runtime Environment (8.0_121-b13) (build 1.8.0_121-b13) # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 compressed oops) # Problematic frame: # C [libcrypto.so.1.0.2h+0x117a64] sk_pop_free+0x24 # {code} Core: [^hs_err_pid12281.log] {quote} Now, with this configuration, it happens pretty much immediately after the worker tries to register with the balancer: h3. Elytron {code} <server-ssl-contexts> <server-ssl-context name="serverSSLContext" cipher-suite-filter="TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256" providers="openssl" need-client-auth="false" key-managers="keyManager" protocols="TLSv1.2" trust-managers="trustManager"/> </server-ssl-contexts> <client-ssl-contexts> <client-ssl-context name="clientSSLContext" cipher-suite-filter="TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256" providers="openssl" key-managers="clientKeyManager" protocols="TLSv1.2" trust-managers="trustManager"/> </client-ssl-contexts> {code} h3. Undertow {code} <server name="default-server"> <http-listener name="default" socket-binding="http" redirect-socket="https" enable-http2="true"/> <https-listener name="https" socket-binding="https" enable-http2="true" ssl-context="serverSSLContext"/> <host name="default-host" alias="localhost"> <location name="/" handler="welcome-content"/> <access-log/> <filter-ref name="server-header"/> <filter-ref name="x-powered-by-header"/> </host> </server> {code} h3. mod_cluster {code} <subsystem xmlns="urn:jboss:domain:modcluster:3.0"> <mod-cluster-config advertise-socket="modcluster" connector="https" ssl-context="clientSSLContext"> <dynamic-load-provider> <load-metric type="cpu"/> </dynamic-load-provider> </mod-cluster-config> </subsystem> {code} See new [^crashes_on_mod_cluster_connection.server.log] and [^new_hs_err_pid13002.log]. I must admit I am somewhat getting into low spirits with this whole {{client <\-\-H/2\-\->mod_cluster balancer<\-\-H/2\-\->mod_cluster worker}} and "just use Elytron" situation. Would it be possible for you to actually give it a try? Take 1 Wildfly and Undertow mod_cluster balancer, set it up with H/2 via Elytron, take 1 Wildfly node as H/2 mod_cluster worker configured with Elytron, start it...