[keycloak-user] Keycloak Domain TCP Clustering of Sessions in AWS not auto-failing over to node

[JTK]

I have two nodes setup in a cluster using TCP port 7600 and I see them join
the cluster in the logs.
On Master: [Host Controller] 15:07:18,293 INFO
 [org.jboss.as.domain.controller] (Host Controller Service Threads - 7)
WFLYHC0019: Registered remote slave host "dev-slave1", JBoss Keycloak 6.0.1
(WildFly 8.0.0.Final)
On Slave: [Host Controller] 15:03:12,603 INFO
 [org.jboss.as.host.controller] (Host Controller Service Threads - 3)
WFLYHC0148: Connected to master host controller at remote://10.10.10.77:9999

In the WildFly admin panel I see the server group: auth-server-group which
is ha and then I see both servers in the group and they are both green.

I've set the distributed-cache setup to 2 in domain.xml, so it should be
sharing session information:
                    <distributed-cache name="sessions" owners="2"/>
                    <distributed-cache name="authenticationSessions"
owners="2"/>
                    <distributed-cache name="offlineSessions" owners="2"/>
                    <distributed-cache name="clientSessions" owners="2"/>
                    <distributed-cache name="offlineClientSessions"
owners="2"/>
                    <distributed-cache name="loginFailures" owners="2"/>
                    <distributed-cache name="actionTokens" owners="2">

Here is the logs on the master showing there a new cluster has been
received:
2019-08-26 15:03:19,776 INFO  [org.infinispan.CLUSTER] (MSC service thread
1-1) ISPN000094: Received new cluster view for channel ejb: [dev-master|0]
(1) [dev-master]
2019-08-26 15:03:19,779 INFO  [org.infinispan.CLUSTER] (MSC service thread
1-3) ISPN000094: Received new cluster view for channel ejb: [dev-master|0]
(1) [dev-master]
2019-08-26 15:03:19,780 INFO  [org.infinispan.CLUSTER] (MSC service thread
1-2) ISPN000094: Received new cluster view for channel ejb: [dev-master|0]
(1) [dev-master]
2019-08-26 15:03:19,780 INFO  [org.infinispan.CLUSTER] (MSC service thread
1-4) ISPN000094: Received new cluster view for channel ejb: [dev-master|0]
(1) [dev-master]
2019-08-26 15:03:19,875 INFO  [org.infinispan.CLUSTER] (MSC service thread
1-1) ISPN000094: Received new cluster view for channel ejb: [dev-master|0]
(1) [dev-master]

And on the slave:
2019-08-26 15:07:29,567 INFO  [org.infinispan.CLUSTER] (MSC service thread
1-2) ISPN000094: Received new cluster view for channel ejb: [dev-slave1|0]
(1) [dev-slave1]
2019-08-26 15:07:29,572 INFO  [org.infinispan.CLUSTER] (MSC service thread
1-3) ISPN000094: Received new cluster view for channel ejb: [dev-slave1|0]
(1) [dev-slave1]
2019-08-26 15:07:29,572 INFO  [org.infinispan.CLUSTER] (MSC service thread
1-4) ISPN000094: Received new cluster view for channel ejb: [dev-slave1|0]
(1) [dev-slave1]
2019-08-26 15:07:29,574 INFO  [org.infinispan.CLUSTER] (MSC service thread
1-1) ISPN000094: Received new cluster view for channel ejb: [dev-slave1|0]
(1) [dev-slave1]
2019-08-26 15:07:29,635 INFO  [org.infinispan.CLUSTER] (MSC service thread
1-3) ISPN000094: Received new cluster view for channel ejb: [dev-slave1|0]
(1) [dev-slave1]

I believe I read somewhere that I was supposed to see the master and slave
together in the logs an not just master or slave. Maybe this is my issue,
but I don't know how to resolve it.

I can't use multi-cast as it's disabled in AWS and almost all cloud
providers.

When I launch the master and let it come up, then launch the slave I can
see all the traffic for the session on the master. As soon as I stop the
master, the slave is looking for the master, but when I click on the
website, it just hangs waiting for a connection and then eventually logs me
out, and I end up logging back in, and now I'm on the slave node. The
shared sessions are not happening. Is there something else I need to do or
set?

I have this setup in my domain.xml configuration as well:
        <server-group name="auth-server-group" profile="ha">
            <jvm name="default">
                <heap size="64m" max-size="512m"/>
            </jvm>
            <socket-binding-group ref="ha-sockets"/>
            <system-properties>
                <property name="jboss.cluster.tcp.initial_hosts"
value="10.10.10.77[7600],10.10.10.27[7600]"/>
            </system-properties>
        </server-group>

In my host.xml on the slave I have this setup to reach back to the master
as the domain controller
    <domain-controller>
        <remote protocol="remote" host="${jboss.domain.master.address}"
port="${jboss.domain.master.port:9999}" security-realm="ManagementRealm"/>
   </domain-controller>

Any help would be appreciated