Error on startup when multiple FIPS Credential Stores are configured -------------------------------------------------------------------- Key: WFCORE-3408 URL: https://issues.jboss.org/browse/WFCORE-3408 Project: WildFly Core Issue Type: Bug Components: Security Affects Versions: 4.0.0.Alpha2 Reporter: Tomas Hofman Assignee: Tomas Hofman Priority: Critical In case there is multiple external PKCS11 credential stores configured, intermittently it happens on startup exception occurs. Seems more CS are configured, it is bigger chance to hit error. If only one CS is configured error does not occur. We have automatic tests with 3 CS and so far we have not hit this issue. With 5 CS from reproducer it happens nearly with each restart. I hit this during discussing analysis of advanced keystore features [1] with Farah. Multiple CS can be expected by users, when they want to logically separate CS files for some reason. However technically everything can be stored in one CS. That is analogy of https://issues.jboss.org/browse/JBEAP-11693 for PKCS11 credential store. Fix for that issue was proper synchronization of PKCS11 keystore loading. I assume something similar will be needed here. If there is missing synchronization of PKCS11 keystore in Credential store implementation, that can in theory occur in combination of 1 CS and multiple PKCS11 keystores as well. However this scenario is tested for and such problem haven't occurred yet. [~bmaxwell] Is GSS is ok with this being critical since a workaround exists? External Credential Store is new feature of Elytron tracked by EAP7-277. {code:title=KeyStoreException: invalid KeyStore state: found N secret keys sharing CKA_LABEL [my-key]} 09:56:15,574 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([ ("subsystem" => "elytron"), ("credential-store" => "MyStore") ]) - failure description: {"WFLYCTL0080: Failed services" => {"org.wildfly.security.credential-store.MyStore" => "WFLYELY00004: Unable to start the service. Caused by: org.wildfly.security.credential.store.CredentialStoreException: ELY09514: Unable to initialize credential store Caused by: org.wildfly.security.credential.store.CredentialStoreException: java.security.KeyStoreException: expected but could not find secret key Caused by: java.security.KeyStoreException: expected but could not find secret key"}} 09:56:15,575 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([ ("subsystem" => "elytron"), ("credential-store" => "MyStore4") ]) - failure description: {"WFLYCTL0080: Failed services" => {"org.wildfly.security.credential-store.MyStore4" => "WFLYELY00004: Unable to start the service. Caused by: org.wildfly.security.credential.store.CredentialStoreException: ELY09514: Unable to initialize credential store Caused by: org.wildfly.security.credential.store.CredentialStoreException: java.security.KeyStoreException: invalid KeyStore state: found 2 secret keys sharing CKA_LABEL [my-key] Caused by: java.security.KeyStoreException: invalid KeyStore state: found 2 secret keys sharing CKA_LABEL [my-key]"}} 09:56:15,576 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([ ("subsystem" => "elytron"), ("credential-store" => "MyStore5") ]) - failure description: {"WFLYCTL0080: Failed services" => {"org.wildfly.security.credential-store.MyStore5" => "WFLYELY00004: Unable to start the service. Caused by: org.wildfly.security.credential.store.CredentialStoreException: ELY09514: Unable to initialize credential store Caused by: org.wildfly.security.credential.store.CredentialStoreException: java.security.KeyStoreException: invalid KeyStore state: found 3 secret keys sharing CKA_LABEL [my-key] Caused by: java.security.KeyStoreException: invalid KeyStore state: found 3 secret keys sharing CKA_LABEL [my-key]"}} {code} [1] https://developer.jboss.org/wiki/AnalysisDesign-AdvancedElytronKey-storeM...