[keycloak-user] Keycloak Domain TCP Clustering of Sessions in AWS not auto-failing over to node

Tue Aug 27 06:08:35 EDT 2019

On Mon, Aug 26, 2019 at 5:32 PM JTK <jonesy at sydow.org> wrote:

> I have two nodes setup in a cluster using TCP port 7600 and I see them join
> the cluster in the logs.
> On Master: [Host Controller] 15:07:18,293 INFO
>  [org.jboss.as.domain.controller] (Host Controller Service Threads - 7)
> WFLYHC0019: Registered remote slave host "dev-slave1", JBoss Keycloak 6.0.1
> (WildFly 8.0.0.Final)
> On Slave: [Host Controller] 15:03:12,603 INFO
>  [org.jboss.as.host.controller] (Host Controller Service Threads - 3)
> WFLYHC0148: Connected to master host controller at remote://
> 10.10.10.77:9999
>
> In the WildFly admin panel I see the server group: auth-server-group which
> is ha and then I see both servers in the group and they are both green.
>
> I've set the distributed-cache setup to 2 in domain.xml, so it should be
> sharing session information:
>                     <distributed-cache name="sessions" owners="2"/>
>                     <distributed-cache name="authenticationSessions"
> owners="2"/>
>                     <distributed-cache name="offlineSessions" owners="2"/>
>                     <distributed-cache name="clientSessions" owners="2"/>
>                     <distributed-cache name="offlineClientSessions"
> owners="2"/>
>                     <distributed-cache name="loginFailures" owners="2"/>
>                     <distributed-cache name="actionTokens" owners="2">
>
> Here is the logs on the master showing there a new cluster has been
> received:
> 2019-08-26 15:03:19,776 INFO  [org.infinispan.CLUSTER] (MSC service thread
> 1-1) ISPN000094: Received new cluster view for channel ejb: [dev-master|0]
> (1) [dev-master]
> 2019-08-26 15:03:19,779 INFO  [org.infinispan.CLUSTER] (MSC service thread
> 1-3) ISPN000094: Received new cluster view for channel ejb: [dev-master|0]
> (1) [dev-master]
> 2019-08-26 15:03:19,780 INFO  [org.infinispan.CLUSTER] (MSC service thread
> 1-2) ISPN000094: Received new cluster view for channel ejb: [dev-master|0]
> (1) [dev-master]
> 2019-08-26 15:03:19,780 INFO  [org.infinispan.CLUSTER] (MSC service thread
> 1-4) ISPN000094: Received new cluster view for channel ejb: [dev-master|0]
> (1) [dev-master]
> 2019-08-26 15:03:19,875 INFO  [org.infinispan.CLUSTER] (MSC service thread
> 1-1) ISPN000094: Received new cluster view for channel ejb: [dev-master|0]
> (1) [dev-master]
>
> And on the slave:
> 2019-08-26 15:07:29,567 INFO  [org.infinispan.CLUSTER] (MSC service thread
> 1-2) ISPN000094: Received new cluster view for channel ejb: [dev-slave1|0]
> (1) [dev-slave1]
> 2019-08-26 15:07:29,572 INFO  [org.infinispan.CLUSTER] (MSC service thread
> 1-3) ISPN000094: Received new cluster view for channel ejb: [dev-slave1|0]
> (1) [dev-slave1]
> 2019-08-26 15:07:29,572 INFO  [org.infinispan.CLUSTER] (MSC service thread
> 1-4) ISPN000094: Received new cluster view for channel ejb: [dev-slave1|0]
> (1) [dev-slave1]
> 2019-08-26 15:07:29,574 INFO  [org.infinispan.CLUSTER] (MSC service thread
> 1-1) ISPN000094: Received new cluster view for channel ejb: [dev-slave1|0]
> (1) [dev-slave1]
> 2019-08-26 15:07:29,635 INFO  [org.infinispan.CLUSTER] (MSC service thread
> 1-3) ISPN000094: Received new cluster view for channel ejb: [dev-slave1|0]
> (1) [dev-slave1]
>

This definitely doesn't look right. The view id (which is increasing
monotonically) is 0. Which means this is an initial view and none of the
new members joined. Clearly, the discovery protocol is not configured
properly and both nodes are in separate (singleton) clusters.

> I believe I read somewhere that I was supposed to see the master and slave
> together in the logs an not just master or slave. Maybe this is my issue,
> but I don't know how to resolve it.
>
> I can't use multi-cast as it's disabled in AWS and almost all cloud
> providers.
>

The easiest option is to use TCPPING. However, it requires you to put all
nodes IPs in its configuration [1]. There are other options as well, e.g.
S3 Ping [2] and its rewritten (and much better) version - Native S3 Ping
[3].

You may also be interested in using JDBC_PING. Please have a look at the
our blogs [4][5].

[1] http://jgroups.org/manual4/index.html#TCPPING_Prot
[2] http://jgroups.org/manual4/index.html#_s3_ping
[3] https://github.com/jgroups-extras/native-s3-ping
[4] https://www.keycloak.org/2019/05/keycloak-cluster-setup.html
[5] https://www.keycloak.org/2019/08/keycloak-jdbc-ping.html

>
> When I launch the master and let it come up, then launch the slave I can
> see all the traffic for the session on the master. As soon as I stop the
> master, the slave is looking for the master, but when I click on the
> website, it just hangs waiting for a connection and then eventually logs me
> out, and I end up logging back in, and now I'm on the slave node. The
> shared sessions are not happening. Is there something else I need to do or
> set?
>

It looks like a consequence of the JGroups discovery issue. Please try to
fix the clustering problem and then see if this one appears again.

>
> I have this setup in my domain.xml configuration as well:
>         <server-group name="auth-server-group" profile="ha">
>             <jvm name="default">
>                 <heap size="64m" max-size="512m"/>
>             </jvm>
>             <socket-binding-group ref="ha-sockets"/>
>             <system-properties>
>                 <property name="jboss.cluster.tcp.initial_hosts"
> value="10.10.10.77[7600],10.10.10.27[7600]"/>
>             </system-properties>
>         </server-group>
>
> In my host.xml on the slave I have this setup to reach back to the master
> as the domain controller
>     <domain-controller>
>         <remote protocol="remote" host="${jboss.domain.master.address}"
> port="${jboss.domain.master.port:9999}" security-realm="ManagementRealm"/>
>    </domain-controller>
>
> Any help would be appreciated
> _______________________________________________
> keycloak-user mailing list
> keycloak-user at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user
>