[jboss-jira] [JBoss JIRA] (WFLY-8753) OpenSSL via Wildfly-OpenSSL crashes VM on application server shutdown when mod_cluster used

Thu May 11 16:15:00 EDT 2017

     [ https://issues.jboss.org/browse/WFLY-8753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michal Karm Babacek updated WFLY-8753:
--------------------------------------
    Attachment: new_hs_err_pid13002.log
                crashes_on_mod_cluster_connection.server.log


> OpenSSL via Wildfly-OpenSSL crashes VM on application server shutdown when mod_cluster used
> -------------------------------------------------------------------------------------------
>
>                 Key: WFLY-8753
>                 URL: https://issues.jboss.org/browse/WFLY-8753
>             Project: WildFly
>          Issue Type: Bug
>          Components: Security
>    Affects Versions: 11.0.0.Beta1
>         Environment: RHEL 7 x86_64
>            Reporter: Michal Karm Babacek
>            Assignee: Stuart Douglas
>            Priority: Critical
>              Labels: wildfly-openssl
>         Attachments: crashes_on_mod_cluster_connection.server.log, hs_err_pid12281.log, new_hs_err_pid13002.log, server.log, standalone-ha.xml
>
>
> h3. Previously I thought it happens just on server shutdown...
> {quote}
> Worker node with this mod_cluster and Elytron configuration [^standalone-ha.xml] keeps complaining on being unable to agree on a cipher suite with the Undertow balancer, whose Elytron configuration is identical, i.e. {code}
> <server-ssl-contexts>
>     <server-ssl-context name="serverSSLContext" providers="openssl" need-client-auth="false" key-managers="keyManager"  protocols="TLSv1.2" trust-managers="trustManager"/>
> </server-ssl-contexts>
> <client-ssl-contexts>
>     <client-ssl-context name="clientSSLContext" providers="openssl" key-managers="clientKeyManager" protocols="TLSv1.2" trust-managers="trustManager"/>
> </client-ssl-contexts>{code}
> see:{code}
> 15:19:25,492 FINE  [org.wildfly.openssl.OpenSSLEngine] (UndertowEventHandlerAdapter - 1) The version of SSL in use does not support cipher ordering
> 15:19:25,541 ERROR [org.jboss.mod_cluster.undertow] (UndertowEventHandlerAdapter - 1) null: java.lang.NullPointerException
>     at org.wildfly.openssl.CipherSuiteConverter.toJava(CipherSuiteConverter.java:284)
>     at org.wildfly.openssl.OpenSSLEngine.toJavaCipherSuite(OpenSSLEngine.java:1021)
>     at org.wildfly.openssl.OpenSSlSession.initCipherSuite(OpenSSlSession.java:305)
>     at org.wildfly.openssl.OpenSSlSession.initialised(OpenSSlSession.java:296)
>     at org.wildfly.openssl.OpenSSLSessionContext.clientSessionCreated(OpenSSLSessionContext.java:114)
>     at org.wildfly.openssl.OpenSSLClientSessionContext.storeClientSideSession(OpenSSLClientSessionContext.java:92)
>     at org.wildfly.openssl.OpenSSLEngine.handshakeFinished(OpenSSLEngine.java:940)
>     at org.wildfly.openssl.OpenSSLEngine.getHandshakeStatus(OpenSSLEngine.java:989)
>     at org.wildfly.openssl.OpenSSLEngine.unwrap(OpenSSLEngine.java:606)
>     at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
>     at org.wildfly.openssl.OpenSSLSocket.runHandshake(OpenSSLSocket.java:316)
>     at org.wildfly.openssl.OpenSSLSocket.write(OpenSSLSocket.java:462)
>     at org.wildfly.openssl.OpenSSLOutputStream.write(OpenSSLOutputStream.java:46)
>     at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
>     at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
>     at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295)
>     at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
>     at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
>     at java.io.BufferedWriter.flush(BufferedWriter.java:254)
>     at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequest(DefaultMCMPHandler.java:526)
>     at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequest(DefaultMCMPHandler.java:605)
>     at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.status(DefaultMCMPHandler.java:387)
>     at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.status(DefaultMCMPHandler.java:365)
>     at org.jboss.modcluster.ModClusterService.status(ModClusterService.java:454)
>     at org.wildfly.mod_cluster.undertow.UndertowEventHandlerAdapter.run(UndertowEventHandlerAdapter.java:169)
>     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>     at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>     at java.lang.Thread.run(Thread.java:745)
>     at org.jboss.threads.JBossThread.run(JBossThread.java:320)
> {code}
> Full log: [^server.log].
> I might or might not have it misconfigured, but what definitely should not happen is that when I try to shut the server down:
> {code}
> INFO  [org.jboss.as.server] (Thread-2) WFLYSRV0220: Server shutdown has been requested via an OS signal
> {code}
> it crashes JVM (doublefree?):
> {code}
> # JRE version: Java(TM) SE Runtime Environment (8.0_121-b13) (build 1.8.0_121-b13)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 compressed oops)
> # Problematic frame:
> # C  [libcrypto.so.1.0.2h+0x117a64]  sk_pop_free+0x24
> #
> {code}
> Core: [^hs_err_pid12281.log]
> {quote}
> Now, with this configuration, it happens pretty much immediately after the worker tries to register with the balancer:
> h3. Elytron
> {code}
> <server-ssl-contexts>
>     <server-ssl-context name="serverSSLContext" cipher-suite-filter="TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256" providers="openssl" need-client-auth="false" key-managers="keyManager"  protocols="TLSv1.2" trust-managers="trustManager"/>
> </server-ssl-contexts>
> <client-ssl-contexts>
>     <client-ssl-context name="clientSSLContext" cipher-suite-filter="TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256" providers="openssl" key-managers="clientKeyManager" protocols="TLSv1.2" trust-managers="trustManager"/>
> </client-ssl-contexts>
> {code}
> h3. Undertow
> {code}
> <server name="default-server">
>     <http-listener name="default" socket-binding="http" redirect-socket="https" enable-http2="true"/>
>     <https-listener name="https" socket-binding="https" enable-http2="true" ssl-context="serverSSLContext"/>
>     <host name="default-host" alias="localhost">
>         <location name="/" handler="welcome-content"/>
>         <access-log/>
>         <filter-ref name="server-header"/>
>         <filter-ref name="x-powered-by-header"/>
>     </host>
> </server>
> {code}
> h3. mod_cluster
> {code}
> <subsystem xmlns="urn:jboss:domain:modcluster:3.0">
>     <mod-cluster-config advertise-socket="modcluster" connector="https" ssl-context="clientSSLContext">
>         <dynamic-load-provider>
>             <load-metric type="cpu"/>
>         </dynamic-load-provider>
>     </mod-cluster-config>
> </subsystem>
> {code}
> See new [^crashes_on_mod_cluster_connection.server.log] and [^new_hs_err_pid13002.log].
> I must admit I am somewhat getting into low spirits with this whole {{client <\-\-H/2\-\->mod_cluster balancer<\-\-H/2\-\->mod_cluster worker}} and "just use Elytron" situation. Would it be possible for you to actually give it a try? Take 1 Wildfly and Undertow mod_cluster balancer, set it up with H/2 via Elytron, take 1 Wildfly node as H/2 mod_cluster worker configured with Elytron, start it...


--
This message was sent by Atlassian JIRA
(v7.2.3#72005)