[keycloak-dev] Keycloak Cluster Setup And Configuration

[张立强]

Hi there,

I am writing this mail just for knowledge sharing, through lots of efforts
we achieved keycloak cluster setting in some scenes, maybe you guys have
done this by some solutions but hope this can help you anyway.
We are using 4.8.3.Final docker image in our deployment, using this version
because RH-SSO 7.3.0.GA is derived from 4.8.3.Final(please refer to
https://www.keycloak.org/support.html) and we believe this is a rather
stable build.
Also we did some extensions(custom theme, custom user federation, but the
two cli script files is the most important matter for this mail, attached
you can find the two files) base on the official docker image
<https://hub.docker.com/r/jboss/keycloak/>.
[image: QQ20190424-151432 (1).jpg]

First of all we have to know that for a keycloak cluster, all keycloak
instances should use same database and this is very simple, another thing
is about cache(generally there are two kinds of cache in keycloak, 1st is
persistent data cache read from database aim to improve performance like
realm/client/user, 2nd is the non-persistent data cache like
sessions/clientSessions, the 2nd is very important for a cluster) which is
a little bit complex to configure, we have to make sure the consistent of
cache data in a cluster view.
Totally we have 3 solutions for clustering, and all of the solutions are
base on the discovery protocols of JGroups <http://jgroups.org/>, as
keycloak use a distributed cache called Infinispan <http://infinispan.org/>
and the Infinispan use JGroups to discover nodes.

1. PING
PING <http://jgroups.org/manual/#PING> is the default enabled clustering
solution of keycloak using UDP protocol, and you don't need to do any
configuration for this.
But this solution is only available when multicast network is enabled and
port 55200 should be exposed, e.g. bare metals, VMs, docker containers in
same host.
[image: image.png]
We tested this by two keycloak containers in same host.
[image: QQ20190424-150949 (1).jpg]

As you see from logs, after started the two keycloak instances discovered
each other and clustered.

2. TCPPING
TCPPING <http://jgroups.org/manual/#TCPPING_Prot> use TCP protocol and 7600
port should be exposed.
This solution can be used when multicast is not available, e.g. deployments
cross DC, containers cross host.
We tested this by two keycloak containers cross host.
[image: image.png]
And for our own solution we need to set three below environment variables
for containers.
*#IP address of this host*
*JGROUPS_DISCOVERY_EXTERNAL_IP=172.21.48.39*
*#protocol*
*JGROUPS_DISCOVERY_PROTOCOL=TCPPING*
*#IP and Port of all host*
*JGROUPS_DISCOVERY_PROPERTIES=initial_hosts="172.21.48.4[7600],172.21.48.39[7600]"*
After started we can see the keycloak instances discovered each other and
clustered.
[image: QQ20190424-150720 (1).jpg]

3. JDBC_PING
JDBC_PING <http://jgroups.org/manual/#_jdbc_ping> use TCP protocol and 7600
port should be expose which is similar as TCPPING, but the difference
between them is, TCPPING require you configure the IP and port of all
instances,  for JDBC_PING you just need to configure the IP and port of
current instance, this is because in JDBC_PING solution each instance
insert its own information into database and the instances discover peers
by the ping data which is from database.
We tested this by two keycloak containers cross host.
[image: image.png]
And for our own solution we need to set two below environment variables for
containers.
*#IP address of this host*
*JGROUPS_DISCOVERY_EXTERNAL_IP=172.21.48.39*
*#protocol*
*JGROUPS_DISCOVERY_PROTOCOL=JDBC_PING*
After started the ping data of all instances haven been saved in database,
and from logs we can see the keycloak instances discovered each other and
clustered.
[image: image.png]
[image: QQ20190424-151128 (1).jpg]

-----
I believe the above solutions is available for most scenes, but for some
scene this is enough, e.g. kubernetes.
In kubernetes, multicast is available only for the containers in same node,
for the pods cross node multicast is not working, furthermore for a pod
there is no static ip that you can use to configure TCPPING or JDBC_PING.
But that's ok because we can use KUBE_PING
<http://jgroups.org/manual/#_kube_ping> in kubernetes. And also don't
worry, KUBE_PING is not the only choice, actually JDBC_PING is another
option. In the attached JDBC_PING.cli we have handled this,  if you don't
set the JGROUPS_DISCOVERY_EXTERNAL_IP environment variable, the pod ip will
be used, that means in kubernetes you can just set
JGROUPS_DISCOVERY_PROTOCOL=JDBC_PING then your keycloak cluster is ok.


----MAIL END----

张立强

研发小头目, FIT2CLOUD


中国领先的云管理平台及服务提供商，助力企业 管好云 · 用好云 · 控好云
-------------- next part --------------

-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 18324 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/keycloak-dev/attachments/20190424/90daf0c2/attachment-0003.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 8762 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/keycloak-dev/attachments/20190424/90daf0c2/attachment-0004.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 39242 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/keycloak-dev/attachments/20190424/90daf0c2/attachment-0005.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: QQ20190424-150720 (1).jpg
Type: image/jpeg
Size: 154243 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/keycloak-dev/attachments/20190424/90daf0c2/attachment-0004.jpg 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: QQ20190424-150949 (1).jpg
Type: image/jpeg
Size: 150139 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/keycloak-dev/attachments/20190424/90daf0c2/attachment-0005.jpg 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: QQ20190424-151128 (1).jpg
Type: image/jpeg
Size: 138614 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/keycloak-dev/attachments/20190424/90daf0c2/attachment-0006.jpg 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: QQ20190424-151432 (1).jpg
Type: image/jpeg
Size: 47830 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/keycloak-dev/attachments/20190424/90daf0c2/attachment-0007.jpg