[keycloak-user] Enabling High Availability for Keycloak 3.1.0 on AWS ECS Instance

[Jyoti Kumar Singh]

Hi,

I have tried the JDBC_PING option which Tonnis has mentioned :-
https://github.com/devsu/docker-keycloak/tree/master/server-ha-mysql

After that I could see node discovery is happening but JOIN operation is
getting timed out which eventually not forming clustering between two ECS
instances. Is there any configuration am I missing here ?


#Logs:-

2017-09-19 10:59:52,907 WARN  [org.jgroups.protocols.UDP] (MSC service
thread 1-1) [f0be09280f90] KEYCLOAK 3.1.0.Final-MySQL JGRP000015: the
receive buffer of socket ManagedMulticastSocketBinding was set to 25MB, but
the OS only allocated 212.99KB. This might lead to performance problems.
Please set your max receive buffer in the OS correctly (e.g.
net.core.rmem_max on Linux)
2017-09-19 10:59:59,475 WARN  [org.jgroups.protocols.pbcast.GMS] (MSC
service thread 1-1) [f0be09280f90] KEYCLOAK 3.1.0.Final-MySQL f0be09280f90:
JOIN(f0be09280f90) sent to 16c566cfa08e timed out (after 3000 ms), on try 1
2017-09-19 11:00:02,490 WARN  [org.jgroups.protocols.pbcast.GMS] (MSC
service thread 1-1) [f0be09280f90] KEYCLOAK 3.1.0.Final-MySQL f0be09280f90:
JOIN(f0be09280f90) sent to 16c566cfa08e timed out (after 3000 ms), on try 2
2017-09-19 11:00:05,508 WARN  [org.jgroups.protocols.pbcast.GMS] (MSC
service thread 1-1) [f0be09280f90] KEYCLOAK 3.1.0.Final-MySQL f0be09280f90:
JOIN(f0be09280f90) sent to 16c566cfa08e timed out (after 3000 ms), on try 3
2017-09-19 11:00:08,527 WARN  [org.jgroups.protocols.pbcast.GMS] (MSC
service thread 1-1) [f0be09280f90] KEYCLOAK 3.1.0.Final-MySQL f0be09280f90:
JOIN(f0be09280f90) sent to 16c566cfa08e timed out (after 3000 ms), on try 4
2017-09-19 11:00:11,542 WARN  [org.jgroups.protocols.pbcast.GMS] (MSC
service thread 1-1) [f0be09280f90] KEYCLOAK 3.1.0.Final-MySQL f0be09280f90:
JOIN(f0be09280f90) sent to 16c566cfa08e timed out (after 3000 ms), on try 5
2017-09-19 11:00:14,558 WARN  [org.jgroups.protocols.pbcast.GMS] (MSC
service thread 1-1) [f0be09280f90] KEYCLOAK 3.1.0.Final-MySQL f0be09280f90:
JOIN(f0be09280f90) sent to 16c566cfa08e timed out (after 3000 ms), on try 6
2017-09-19 11:00:17,579 WARN  [org.jgroups.protocols.pbcast.GMS] (MSC
service thread 1-1) [f0be09280f90] KEYCLOAK 3.1.0.Final-MySQL f0be09280f90:
JOIN(f0be09280f90) sent to 16c566cfa08e timed out (after 3000 ms), on try 7
2017-09-19 11:00:20,596 WARN  [org.jgroups.protocols.pbcast.GMS] (MSC
service thread 1-1) [f0be09280f90] KEYCLOAK 3.1.0.Final-MySQL f0be09280f90:
JOIN(f0be09280f90) sent to 16c566cfa08e timed out (after 3000 ms), on try 8
2017-09-19 11:00:23,611 WARN  [org.jgroups.protocols.pbcast.GMS] (MSC
service thread 1-1) [f0be09280f90] KEYCLOAK 3.1.0.Final-MySQL f0be09280f90:
JOIN(f0be09280f90) sent to 16c566cfa08e timed out (after 3000 ms), on try 9
2017-09-19 11:00:26,627 WARN  [org.jgroups.protocols.pbcast.GMS] (MSC
service thread 1-1) [f0be09280f90] KEYCLOAK 3.1.0.Final-MySQL f0be09280f90:
JOIN(f0be09280f90) sent to 16c566cfa08e timed out (after 3000 ms), on try 10
2017-09-19 11:00:26,627 WARN  [org.jgroups.protocols.pbcast.GMS] (MSC
service thread 1-1) [f0be09280f90] KEYCLOAK 3.1.0.Final-MySQL f0be09280f90:
too many JOIN attempts (10): becoming singleton

On Thu, Sep 14, 2017 at 10:48 PM, Jyoti Kumar Singh <jyoti.tech90 at gmail.com>
wrote:

> Hi Tonnis,
>
> Thank you very much for sharing the valuable information. I am checking on
> this, hopefully I will also be able to achieve the HA.
>
> Thanks Again !
>
> On Sep 14, 2017 10:00 PM, "Tonnis Wildeboer" <tonnis at autonomic.ai> wrote:
>
>> Jyoti,
>>
>> I have been working on similar goal and was finally successful yesterday.
>> We are using postgres and kubernetes.
>>
>> Here are the key sources of information that enabled me to succeed:
>>
>> The big key is here:
>> https://github.com/devsu/docker-keycloak/tree/master/server-ha-mysql
>> Use the .xsl templates here to transform on the standalone-ha.xml and you
>> can see what is being done.
>>
>> I suggest that you simply use JDBC_PING, since you already have a shared
>> database.
>> I think it is instructive to understand what JDBC_PING (and JGroups in
>> general) are doing:
>> http://jgroups.org/manual4/index.html
>> https://developer.jboss.org/wiki/JDBCPING
>>
>> You may benefit from this also, specifically, the need to bind
>> jgroups-tcp and jgroups-tcp-fd to the proper interface. Not sure about your
>> situation.
>>
>> --Tonnis
>>
>> ____________________
>> Tonnis Wildeboer
>> Autonomic.ai Engineering
>>
>> On 09/14/2017 03:32 AM, Jyoti Kumar Singh wrote:
>>
>>> Hi Team,
>>>
>>> I am trying to enable high availability for Keycloak 3.1.0 on AWS ECS
>>> instances.
>>>
>>> I am running two ECS instances in a cluster setup and also I have
>>> setup Keycloak
>>> in a clustered mode. To achieve this, I am using "
>>> */standalone/configuration/standalone-ha.xml *" file while building the
>>> docker image. Shared MySQL DB and Load Balancer setup are also in place.
>>>
>>> But when I checked Keycloak logs I am not seeing clustered nodes related
>>> information in logs. I am seeing nodes are not able to see each other.
>>> But
>>> same settings are working fine in DCOS Marathon platform.
>>>
>>> Interestingly if I run two Keycloak instances in one AWS ECS instance on
>>> different ports, I could see clustering related logs in Keycloak.
>>>
>>> Is there any standard guidelines which I can follow to achieve HA in AWS
>>> ECS instance ?? I followed the below discussion thread but it didn't
>>> help
>>> me to fix the issue.
>>>
>>> #Link:
>>> http://lists.jboss.org/pipermail/keycloak-user/2016-February/004940.html
>>>
>>>
>>


-- 

*With Regards, Jyoti Kumar Singh*