[jboss-jira] [JBoss JIRA] (WFLY-10138) TLS using PKCS11 and JDK9+ does not work by default

TLS using PKCS11 and JDK9+ does not work by default --------------------------------------------------- Key: WFLY-10138 URL: https://issues.jboss.org/browse/WFLY-10138 Project: WildFly Issue Type: Bug Components: Documentation, Security Affects Versions: 12.0.0.Final Environment: java version "9.0.4" Java(TM) SE Runtime Environment (build 9.0.4+11) Java HotSpot(TM) 64-Bit Server VM (build 9.0.4+11, mixed mode) Reporter: Martin Choma Priority: Critical Attachments: TLS_with_ExtendedMasterSecret, TLS_wo_ExtendedMAsterSecret Since JDK 9.0.4 default behaviour changed and extended master secret extension is turned on by default [1]. This fails on java using sun.security.pkcs11.SunPKCS11 provider. (FIPS compliant java) {code} 17:32:48,377 INFO [stdout] (default task-1) SESSION KEYGEN: 17:32:48,378 INFO [stdout] (default task-1) PreMaster Secret: 17:32:48,378 INFO [stdout] (default task-1) (key bytes not available) 17:32:48,378 INFO [stdout] (default task-1) RSA master secret generation error: 17:32:48,378 INFO [stdout] (default task-1) java.security.InvalidAlgorithmParameterException: Key format must be RAW 17:32:48,378 INFO [stdout] (default task-1) at java.base/com.sun.crypto.provider.TlsMasterSecretGenerator.engineInit(TlsMasterSecretGenerator.java:69) 17:32:48,378 INFO [stdout] (default task-1) at java.base/javax.crypto.KeyGenerator.init(KeyGenerator.java:477) 17:32:48,378 INFO [stdout] (default task-1) at java.base/javax.crypto.KeyGenerator.init(KeyGenerator.java:453) 17:32:48,378 INFO [stdout] (default task-1) at java.base/sun.security.ssl.Handshaker.calculateMasterSecret(Handshaker.java:1334) 17:32:48,378 INFO [stdout] (default task-1) at java.base/sun.security.ssl.Handshaker.calculateKeys(Handshaker.java:1235) 17:32:48,378 INFO [stdout] (default task-1) at java.base/sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:318) 17:32:48,378 INFO [stdout] (default task-1) at java.base/sun.security.ssl.Handshaker.processLoop(Handshaker.java:1092) 17:32:48,379 INFO [stdout] (default task-1) at java.base/sun.security.ssl.Handshaker$1.run(Handshaker.java:1031) 17:32:48,379 INFO [stdout] (default task-1) at java.base/sun.security.ssl.Handshaker$1.run(Handshaker.java:1028) 17:32:48,379 INFO [stdout] (default task-1) at java.base/java.security.AccessController.doPrivileged(Native Method) 17:32:48,379 INFO [stdout] (default task-1) at java.base/sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1534) 17:32:48,379 INFO [stdout] (default task-1) at io.undertow.core@2.0.0.SP1-redhat-1//io.undertow.protocols.ssl.SslConduit$5.run(SslConduit.java:1047) 17:32:48,379 INFO [stdout] (default task-1) at org.jboss.threads@2.3.1.Final-redhat-1//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) 17:32:48,379 INFO [stdout] (default task-1) at org.jboss.threads@2.3.1.Final-redhat-1//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) 17:32:48,379 INFO [stdout] (default task-1) at org.jboss.threads@2.3.1.Final-redhat-1//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) 17:32:48,379 INFO [stdout] (default task-1) at org.jboss.threads@2.3.1.Final-redhat-1//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) 17:32:48,379 INFO [stdout] (default task-1) at java.base/java.lang.Thread.run(Thread.java:844) 17:32:48,379 INFO [stdout] (default I/O-7) default I/O-7, fatal error: 80: problem unwrapping net record 17:32:48,379 INFO [stdout] (default I/O-7) java.lang.RuntimeException: java.security.InvalidAlgorithmParameterException: Key format must be RAW {code} This default extension behaviour can be switched off by system property {{-Djdk.tls.useExtendedMasterSecret=false}} on client or on server side. [1] https://bugs.java.com/view_bug.do?bug_id=JDK-8148421