[keycloak-user] Domain mode cluster, slave authentication?

Hi, Has anyone used the latest WildFly 11 version of Keycloak in domain mode? I could get it to work on a local instance with host-master and host-slave config files. But using the same host-slave config on a different instance it would fail to authenticate. Error : [Host Controller] 09:07:25,741 INFO [org.jboss.remoting] (MSC service thread 1-1) JBoss Remoting version 5.0.5.Final [Host Controller] 09:07:25,874 INFO [org.jboss.as.remoting] (MSC service thread 1-2) WFLYRMT0001: Listening on 127.0.0.1:3456 [Host Controller] 09:07:26,167 WARN [org.jboss.as.host.controller] (Controller Boot Thread) WFLYHC0001: Could not connect to remote domain controller remote://192.168.33.10:9999: java.lang.IllegalStateException: WFLYHC0043: Unable to connect due to authentication failure. [Host Controller] at org.jboss.as.host.controller. RemoteDomainConnectionService.rethrowIrrecoverableConnectionFailures( RemoteDomainConnectionService.java:674) [Host Controller] at org.jboss.as.host.controller. RemoteDomainConnectionService.register(RemoteDomainConnectionService. java:293) [Host Controller] at org.jboss.as.host.controller. DomainModelControllerService.connectToDomainMaster( DomainModelControllerService.java:938) [Host Controller] at org.jboss.as.host.controller. DomainModelControllerService.boot(DomainModelControllerService.java:692) [Host Controller] at org.jboss.as.controller.AbstractControllerService$1. run(AbstractControllerService.java:370) [Host Controller] at java.lang.Thread.run(Thread.java:748) [Host Controller] Caused by: javax.security.sasl.SaslException: Authentication failed: all available authentication mechanisms failed: [Host Controller] DIGEST-MD5: javax.security.sasl.SaslException: DIGEST-MD5: Server rejected authentication [Host Controller] at org.jboss.remoting3.remote. ClientConnectionOpenListener.allMechanismsFailed( ClientConnectionOpenListener.java:109) [Host Controller] at org.jboss.remoting3.remote. ClientConnectionOpenListener$Capabilities.handleEvent( ClientConnectionOpenListener.java:446) [Host Controller] at org.jboss.remoting3.remote. ClientConnectionOpenListener$Capabilities.handleEvent( ClientConnectionOpenListener.java:242) [Host Controller] at org.xnio.ChannelListeners.invokeChannelListener( ChannelListeners.java:92) [Host Controller] at org.xnio.conduits.ReadReadyHandler$ ChannelListenerHandler.readReady(ReadReadyHandler.java:66) [Host Controller] at org.xnio.nio.NioSocketConduit. handleReady(NioSocketConduit.java:89) [Host Controller] at org.xnio.nio.WorkerThread.run(WorkerThread.java:591) [Host Controller] at ...asynchronous invocation...(Unknown Source) [Host Controller] at org.jboss.remoting3.EndpointImpl.connect( EndpointImpl.java:570) [Host Controller] at org.jboss.remoting3.EndpointImpl.connect( EndpointImpl.java:532) [Host Controller] at org.jboss.remoting3.EndpointImpl.connect( EndpointImpl.java:520) [Host Controller] at org.jboss.as.protocol.ProtocolConnectionUtils.connect( ProtocolConnectionUtils.java:204) [Host Controller] at org.jboss.as.protocol.ProtocolConnectionUtils. connectSync(ProtocolConnectionUtils.java:120) [Host Controller] at org.jboss.as.host.controller. RemoteDomainConnection.lambda$openConnection$0(RemoteDomainConnection.java: 223) [Host Controller] at org.wildfly.common.context. Contextual.runExceptionAction(Contextual.java:108) [Host Controller] at org.wildfly.security.auth.client.AuthenticationContext. run(AuthenticationContext.java:268) [Host Controller] at org.jboss.as.host.controller.RemoteDomainConnection. openConnection(RemoteDomainConnection.java:223) [Host Controller] at org.jboss.as.host.controller.RemoteDomainConnection$ InitialConnectTask.connect(RemoteDomainConnection.java:592) [Host Controller] at org.jboss.as.protocol.ProtocolConnectionManager. connect(ProtocolConnectionManager.java:70) [Host Controller] at org.jboss.as.host.controller.RemoteDomainConnection. connect(RemoteDomainConnection.java:147) [Host Controller] at org.jboss.as.host.controller. RemoteDomainConnectionService.register(RemoteDomainConnectionService. java:288) [Host Controller] ... 4 more [Host Controller] Suppressed: javax.security.sasl.SaslException: DIGEST-MD5: Server rejected authentication [Host Controller] at org.jboss.remoting3.remote. ClientConnectionOpenListener$Authentication.handleEvent( ClientConnectionOpenListener.java:736) [Host Controller] at org.jboss.remoting3.remote. ClientConnectionOpenListener$Authentication.handleEvent( ClientConnectionOpenListener.java:578) [Host Controller] at org.xnio.ChannelListeners.invokeChannelListener( ChannelListeners.java:92) [Host Controller] at org.xnio.conduits.ReadReadyHandler$ ChannelListenerHandler.readReady(ReadReadyHandler.java:66) [Host Controller] at org.xnio.nio.NioSocketConduit. handleReady(NioSocketConduit.java:89) [Host Controller] at org.xnio.nio.WorkerThread.run(WorkerThread.java:591) [Host Controller] [Host Controller] 09:07:26,169 WARN [org.jboss.as.host.controller] (Controller Boot Thread) WFLYHC0147: No domain controller discovery options remain. [Host Controller] 09:07:26,169 ERROR [org.jboss.as.host.controller] (Controller Boot Thread) WFLYHC0002: Could not connect to master. Error was: java.lang.IllegalStateException: WFLYHC0120: Tried all domain controller discovery option(s) but unable to connect [Host Controller] 09:07:26,170 FATAL [org.jboss.as.host.controller] (Controller Boot Thread) WFLYHC0178: Aborting with exit code 99 After poking around a bit I found the slave makes a connection with this bit of host-slave.xml : <domain-controller> <remote username="$local" security-realm="ManagementRealm"> <discovery-options> <static-discovery name="primary" protocol="${jboss.domain.master.protocol:remote}" host="${jboss.domain.master.address:192.168.33.10}" port="${jboss.domain.master.port:9999}"/> </discovery-options> </remote> </domain-controller> I changed $local to admin and it connects fine. But if I understand Wildfly, then you shouldn't need to specify a username at all, and the remote server will interpret $local (or none) as the local default user, which would be "admin". I tried leaving out the username and that didn't work either. Have I managed to configure my user wrong on the master somehow? I used this command (and obviously have the correct secret in the host-slave.xml) : bin/add-user.sh -u admin -p password -r ManagementRealm -ds -e Or is this actually expected behaviour?