[keycloak-user] Slave in Domain mode saying "missing/unavailable dependencies org.wildfly.undertow.listener.https is missing [org.wildfly.core.management.security.realm.ssl-realm]"

Hello, I have a successful dev system stood-up with a 2 node cluster using AWS and working 100% using Active Directory as the back-end via LDAPS. We are moving it to production and automation as much of the process as possible via Cloud Formation, bootstraps with S3 buckets and scripts. So it's not just a fork lift. We are on the home stretch with the Master up and running in Domain mode using TCCPING (We might go to Native S3 Ping) and when I bring up one Slave, it launches all services and clusters up, but port https errors out so port 443 does not start up and it looks like 8080 is missing along with 7600, yet it is clustering, per the logs on the Master. Here is the output of running services. tcp 0 0 127.0.0.1:42363 0.0.0.0:* LISTEN 31686/java tcp 0 0 10.122.160.37:3456 0.0.0.0:* LISTEN 31701/java tcp 0 0 0.0.0.0:8259 0.0.0.0:* LISTEN 31782/java tcp 0 0 127.0.0.1:7850 0.0.0.0:* LISTEN 31782/java tcp 0 0 0.0.0.0:8330 0.0.0.0:* LISTEN 31782/java tcp 0 0 127.0.0.1:38507 0.0.0.0:* LISTEN 31782/java tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1413/sshd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1397/master tcp6 0 0 :::22 :::* LISTEN 1413/sshd tcp6 0 0 ::1:25 :::* LISTEN 1397/master Here is what is used to launch the Slave: /opt/keycloak/bin/domain.sh --host-config=host-slave.xml -Djboss.bind.address=0.0.0.0 -Djboss.bind.address.management=10.122.160.37 -Djboss.bind.address:10.122.160.37 -Djboss.bind.address.private:10.122.160.37 -Djboss.https.port=443 -Djboss.domain.master.address=10.122.160.147 -Djboss.tx.node.id <http://djboss.tx.node.id/>=prod-slave-a -Djboss.node.name <http://djboss.node.name/>=prod-slave-a -Djava.security.egd=file:/dev/./urandom This is the actual script which is used to launch it. I've manually added the IPs and Hostname of the Slave above. #!/bin/bash myip=$(ip addr show eth0 | grep -Po 'inet \K[\d.]+') echo $myip echo $HOSTNAME /opt/keycloak/bin/domain.sh --host-config=host-slave.xml -Djboss.bind.address=0.0.0.0 -Djboss.bind.address.management=${myip} -Djboss.bind.address=${myip} -Djboss.bind.address.private=${myip} -Djboss.https.port=443 -Djboss.domain.master.address=10.122.160.147 - Djboss.tx.node.id <http://djboss.tx.node.id/>=${hostname} -Djboss.node.name <http://djboss.node.name/>=${hostname} -Djava.security.egd=file:/dev/./urandom & The Master Address above is hardcoded for the time being until we work out getting that automated as well. Below is the error where is says there are missing dependencies. This is the exact same configuration for our dev environment. That information is only found in the domain.xml configuration file. Here is the actual start-up from the debug logs: [Server:prod-slave-a] 14:47:05,630 DEBUG [org.jboss.as.config] (MSC service thread 1-1) VM Arguments: -D[Server:prod-slave-a] -D[pcid:1223106776] -Xms64m -Xmx512m -XX:MetaspaceSize=96m -XX:MaxMetaspaceSize=256m -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Djava.security.egd=file:/dev/./urandom -Djboss.bind.address=0.0.0.0 -Djboss.bind.address.management=10.122.160.37 -Djboss.bind.address.private:10.122.160.37=true -Djboss.bind.address:10.122.160.37=true -Djboss.cluster.tcp.initial_hosts=10.122.160.147[7600],10.122.160.37[7600] -Djboss.domain.master.address=10.122.160.147 -Djboss.home.dir=/opt/keycloak -Djboss.https.port=443 -Djboss.modules.system.pkgs=org.jboss.byteman - Djboss.node.name <http://djboss.node.name/>=prod-slave-a -Djboss.tx.node.id <http://djboss.tx.node.id/>=prod-slave-a -Djboss.server.log.dir=/opt/keycloak/domain/servers/prod-slave-a/log -Djboss.server.temp.dir=/opt/keycloak/domain/servers/prod-slave-a/tmp -Djboss.server.data.dir=/opt/keycloak/domain/servers/prod-slave-a/data -Dlogging.configuration=file:/opt/keycloak/domain/servers/prod-slave-a/data/logging.properties [Server:prod-slave-a] 14:34:26,278 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 50) WFLYUT0021: Registered web context: '/auth' for server 'default-server' [Server:prod-slave-a] 14:34:26,290 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([ [Server:prod-slave-a] ("subsystem" => "undertow"), [Server:prod-slave-a] ("server" => "default-server"), [Server:prod-slave-a] ("https-listener" => "https") [Server:prod-slave-a] ]) - failure description: { [Server:prod-slave-a] "WFLYCTL0412: Required services that are not installed:" => ["org.wildfly.core.management.security.realm.ssl-realm"], [Server:prod-slave-a] "WFLYCTL0180: Services with missing/unavailable dependencies" => ["org.wildfly.undertow.listener.https is missing [org.wildfly.core.management.security.realm.ssl-realm]"] [Server:prod-slave-a] } [Server:prod-slave-a] 14:34:26,358 INFO [org.jboss.as.server] (ServerService Thread Pool -- 40) WFLYSRV0010: Deployed "keycloak-server.war" (runtime-name : "keycloak-server.war") [Server:prod-slave-a] 14:34:26,364 INFO [org.jboss.as.controller] (Controller Boot Thread) WFLYCTL0183: Service status report [Server:prod--slave-a] WFLYCTL0184: New missing/unsatisfied dependencies: [Server:prod-slave-a] service org.wildfly.core.management.security.realm.ssl-realm (missing) dependents: [service org.wildfly.undertow.listener.https] [Server:prod-slave-a] [Server:prod-slave-a] 14:34:26,402 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server [Server:prod-slave-a] 14:34:26,407 ERROR [org.jboss.as] (Controller Boot Thread) WFLYSRV0026: Keycloak 7.0.1 (WildFly Core 9.0.2.Final) started (with errors) in 16346ms - Started 656 of 962 services (1 services failed or missing dependencies, 690 services are lazy, passive or on-demand) Some of the output from the Master showing that it is clustering: tail -n 50 -f /opt/keycloak/domain/servers/prod-a/log/server.log 2019-11-19 14:46:32,379 INFO [org.infinispan.CLUSTER] (thread-303,ejb,true) ISPN100001: Node prod-sentinel-slave-a left the cluster 2019-11-19 14:46:32,379 INFO [org.infinispan.CLUSTER] (thread-303,ejb,true) ISPN000094: Received new cluster view for channel ejb: [true|16] (1) [true] 2019-11-19 14:46:32,379 INFO [org.infinispan.CLUSTER] (thread-303,ejb,true) ISPN100001: Node prod-sentinel-slave-a left the cluster 2019-11-19 14:47:10,566 INFO [org.infinispan.CLUSTER] (thread-303,ejb,true) ISPN000094: Received new cluster view for channel ejb: [true|17] (2) [true, prod-slave-a] 2019-11-19 14:47:10,567 INFO [org.infinispan.CLUSTER] (thread-303,ejb,true) ISPN100000: Node prod-sentinel-slave-a joined the cluster 2019-11-19 14:47:10,567 INFO [org.infinispan.CLUSTER] (thread-303,ejb,true) ISPN000094: Received new cluster view for channel ejb: [true|17] (2) [true, prod-slave-a] 2019-11-19 14:47:10,567 INFO [org.infinispan.CLUSTER] (thread-303,ejb,true) ISPN100000: Node prod-sentinel-slave-a joined the cluster 2019-11-19 14:47:10,567 INFO [org.infinispan.CLUSTER] (thread-303,ejb,true) ISPN000094: Received new cluster view for channel ejb: [true|17] (2) [true, prod-slave-a] 2019-11-19 14:47:10,567 INFO [org.infinispan.CLUSTER] (thread-303,ejb,true) ISPN100000: Node prod-sentinel-slave-a joined the cluster 2019-11-19 14:47:10,568 INFO [org.infinispan.CLUSTER] (thread-303,ejb,true) ISPN000094: Received new cluster view for channel ejb: [true|17] (2) [true, prod-slave-a] 2019-11-19 14:47:10,568 INFO [org.infinispan.CLUSTER] (thread-303,ejb,true) ISPN100000: Node prod-sentinel-slave-a joined the cluster 2019-11-19 14:47:10,568 INFO [org.infinispan.CLUSTER] (thread-303,ejb,true) ISPN000094: Received new cluster view for channel ejb: [true|17] (2) [true, prod-slave-a] 2019-11-19 14:47:10,568 INFO [org.infinispan.CLUSTER] (thread-303,ejb,true) ISPN100000: Node prod-sentinel-slave-a joined the cluster Any help would be appreciated. I never did encounter this error in dev, although we had numerous ones. Thanks