[jboss-jira] [JBoss JIRA] (WFLY-7704) JVM segfault: mod_cluster subsystem cannot handle wildfly-openssl integration

JVM segfault: mod_cluster subsystem cannot handle wildfly-openssl integration ----------------------------------------------------------------------------- Key: WFLY-7704 URL: https://issues.jboss.org/browse/WFLY-7704 Project: WildFly Issue Type: Bug Affects Versions: 10.1.0.Final Reporter: Michal Karm Babacek Assignee: Stuart Douglas Priority: Critical h3. Preface mod_cluster subsystem doesn't use Security Realms, unfortunately, so one must replicate SSL configuration both in security realms and in mod_cluster subsystem. Apparently, there is a confusion about setting protocol and cipher suite in integration between mod_cluster subsystem and wildfly-openssl: {noformat} at org.wildfly.openssl.OpenSSLEngine.setEnabledProtocols(OpenSSLEngine.java:754) at org.wildfly.openssl.OpenSSLSocket.setEnabledCipherSuites(OpenSSLSocket.java:204) at org.jboss.modcluster.mcmp.impl.JSSESocketFactory.initSocket(JSSESocketFactory.java:384) at org.jboss.modcluster.mcmp.impl.JSSESocketFactory.createSocket(JSSESocketFactory.java:124) {noformat} h3. Configuration {code} <security-realm name="JBossTestServer"> <server-identities> <ssl protocol="openssl.TLS"> <engine enabled-cipher-suites="TLS_RSA_WITH_AES_128_GCM_SHA256"/> <keystore provider="JKS" path="/opt/noe-tests/resources/ssl/proper/server-cert-key.jks" keystore-password="tomcat" alias="javaserver"/> </ssl> </server-identities> <authentication> <truststore path="/opt/noe-tests/resources/ssl/proper/ca-cert.jks" keystore-password="tomcat"/> </authentication> </security-realm> <subsystem xmlns="urn:jboss:domain:modcluster:2.0"> <mod-cluster-config advertise-socket="modcluster" connector="https"> <dynamic-load-provider> <load-metric type="cpu"/> </dynamic-load-provider> <ssl key-alias="javaclient" password="tomcat" certificate-key-file="/opt/noe-tests/resources/ssl/proper/client-cert-key.jks" cipher-suite="TLS_RSA_WITH_AES_128_GCM_SHA256" protocol="openssl.TLS" ca-certificate-file="/opt/noe-tests/resources/ssl/proper/ca-cert.jks"/> </mod-cluster-config> </subsystem> [org.wildfly.openssl.SSL] OpenSSL Version OpenSSL 1.0.2h-fips 3 May 2016 {code} h3. JVM segfault Java Stackstrace on MCMP handler registration: [OpenSSLEngine.java:L754|https://github.com/wildfly/wildfly-openssl/blob/1...] {noformat} 12:01:02,249 ERROR [org.jboss.mod_cluster.undertow] (UndertowEventHandlerAdapter - 1) Unsupported protocol TLS_RSA_WITH_AES_128_GCM_SHA256: java.lang.IllegalArgumentException: Unsupported protocol TLS_RSA_WITH_AES_128_GCM_SHA256 at org.wildfly.openssl.OpenSSLEngine.setEnabledProtocols(OpenSSLEngine.java:754) at org.wildfly.openssl.OpenSSLSocket.setEnabledCipherSuites(OpenSSLSocket.java:204) at org.jboss.modcluster.mcmp.impl.JSSESocketFactory.initSocket(JSSESocketFactory.java:384) at org.jboss.modcluster.mcmp.impl.JSSESocketFactory.createSocket(JSSESocketFactory.java:124) at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler$Proxy.getConnection(DefaultMCMPHandler.java:850) at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler$Proxy.getConnectionWriter(DefaultMCMPHandler.java:886) at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequest(DefaultMCMPHandler.java:514) at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequest(DefaultMCMPHandler.java:605) at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.status(DefaultMCMPHandler.java:387) at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.status(DefaultMCMPHandler.java:365) at org.jboss.modcluster.ModClusterService.status(ModClusterService.java:454) at org.wildfly.mod_cluster.undertow.UndertowEventHandlerAdapter.run(UndertowEventHandlerAdapter.java:179) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) at org.jboss.threads.JBossThread.run(JBossThread.java:320) {noformat}causes JVM segfault: {noformat}# # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x00007fd40e7798c5, pid=29489, tid=0x00007fd44f7f7700 #{noformat} Java and Native stacktrace: for a call from Java [byte\[\] getSessionId0(long ssl)|https://github.com/wildfly/wildfly-openssl/blob/1.0.0.Alpha4/java/sr...] to C [getting session from underlying OpenSSL integration fails|https://github.com/wildfly/wildfly-openssl/blob/1.0.0.Alpha4/libwfs...: [0x00007fd44f6f7000,0x00007fd44f7f8000], sp=0x00007fd44f7f6358, free space=1020k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) C [libssl.so+0x478c5] SSL_SESSION_get_id+0x5 C [libwfssl.so+0x4a0b] Java_org_wildfly_openssl_SSLImpl_getSessionId0+0x6b j org.wildfly.openssl.SSLImpl.getSessionId0(J)[B+0 j org.wildfly.openssl.SSLImpl.getSessionId(J)[B+1 j org.wildfly.openssl.OpenSSLEngine.shutdown()V+42 j org.wildfly.openssl.OpenSSLEngine.finalize()V+5 J 1218 C1 java.lang.ref.Finalizer.runFinalizer(Lsun/misc/JavaLangAccess;)V (62 bytes) @ 0x00007fd4593dbf84 [0x00007fd4593dba00+0x584] J 1217 C1 java.lang.ref.Finalizer.access$100(Ljava/lang/ref/Finalizer;Lsun/misc/JavaLangAccess;)V (6 bytes) @ 0x00007fd4593db69c [0x00007fd4593db640+0x5c] j java.lang.ref.Finalizer$FinalizerThread.run()V+45 v ~StubRoutines::call_stub V [libjvm.so+0x657fbb] V [libjvm.so+0x6593b7] V [libjvm.so+0x659877] V [libjvm.so+0x6a9371] V [libjvm.so+0x9de335] V [libjvm.so+0x9de590] V [libjvm.so+0x8a18b2] C [libpthread.so.0+0x7aa1] Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) j org.wildfly.openssl.SSLImpl.getSessionId0(J)[B+0 j org.wildfly.openssl.SSLImpl.getSessionId(J)[B+1 j org.wildfly.openssl.OpenSSLEngine.shutdown()V+42 j org.wildfly.openssl.OpenSSLEngine.finalize()V+5 J 1218 C1 java.lang.ref.Finalizer.runFinalizer(Lsun/misc/JavaLangAccess;)V (62 bytes) @ 0x00007fd4593dbf84 [0x00007fd4593dba00+0x584] J 1217 C1 java.lang.ref.Finalizer.access$100(Ljava/lang/ref/Finalizer;Lsun/misc/JavaLangAccess;)V (6 bytes) @ 0x00007fd4593db69c [0x00007fd4593db640+0x5c] j java.lang.ref.Finalizer$FinalizerThread.run()V+45 v ~StubRoutines::call_stub{noformat} h3. OpenSSL 1.0.2h This is the declaration of the called [SSL_SESSION_get_id|https://github.com/openssl/openssl/blob/OpenSSL_1_0_2h...] and this is the definition in [ssl_sess.c|https://github.com/openssl/openssl/blob/OpenSSL_1_0_2h/ssl/ssl...] h3. Conclusion IMHO it wouldn't do any harm to check whether {{session}} ain't NULL before passing it on down to OpenSSL, but it is only the aftermath, not the original cause I suppose. It might be worth checking whether the bug is actually not simply in the mod_cluster java part, in calling {{org.wildfly.openssl.OpenSSLEngine}} with confused protocols/cipher suites. Cheers -K-