Re: [keycloak-user] Enabling High Availability for Keycloak 3.1.0 on AWS ECS Instance

Hi, I have tried the JDBC_PING option which Tonnis has mentioned :- https://github.com/devsu/docker-keycloak/tree/master/server-ha-mysql <https://github.com/devsu/docker-keycloak/tree/master/server-ha-mysql> After that I could see node discovery is happening but JOIN operation is getting timed out which eventually not forming clustering between two ECS instances. Is there any configuration am I missing here ? #Logs:- 2017-09-19 10:59:52,907 WARN [org.jgroups.protocols.UDP] (MSC service thread 1-1) [f0be09280f90] KEYCLOAK 3.1.0.Final-MySQL JGRP000015: the receive buffer of socket ManagedMulticastSocketBinding was set to 25MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux) 2017-09-19 10:59:59,475 WARN [org.jgroups.protocols.pbcast.GMS] (MSC service thread 1-1) [f0be09280f90] KEYCLOAK 3.1.0.Final-MySQL f0be09280f90: JOIN(f0be09280f90) sent to 16c566cfa08e timed out (after 3000 ms), on try 1 2017-09-19 11:00:02,490 WARN [org.jgroups.protocols.pbcast.GMS] (MSC service thread 1-1) [f0be09280f90] KEYCLOAK 3.1.0.Final-MySQL f0be09280f90: JOIN(f0be09280f90) sent to 16c566cfa08e timed out (after 3000 ms), on try 2 2017-09-19 11:00:05,508 WARN [org.jgroups.protocols.pbcast.GMS] (MSC service thread 1-1) [f0be09280f90] KEYCLOAK 3.1.0.Final-MySQL f0be09280f90: JOIN(f0be09280f90) sent to 16c566cfa08e timed out (after 3000 ms), on try 3 2017-09-19 11:00:08,527 WARN [org.jgroups.protocols.pbcast.GMS] (MSC service thread 1-1) [f0be09280f90] KEYCLOAK 3.1.0.Final-MySQL f0be09280f90: JOIN(f0be09280f90) sent to 16c566cfa08e timed out (after 3000 ms), on try 4 2017-09-19 11:00:11,542 WARN [org.jgroups.protocols.pbcast.GMS] (MSC service thread 1-1) [f0be09280f90] KEYCLOAK 3.1.0.Final-MySQL f0be09280f90: JOIN(f0be09280f90) sent to 16c566cfa08e timed out (after 3000 ms), on try 5 2017-09-19 11:00:14,558 WARN [org.jgroups.protocols.pbcast.GMS] (MSC service thread 1-1) [f0be09280f90] KEYCLOAK 3.1.0.Final-MySQL f0be09280f90: JOIN(f0be09280f90) sent to 16c566cfa08e timed out (after 3000 ms), on try 6 2017-09-19 11:00:17,579 WARN [org.jgroups.protocols.pbcast.GMS] (MSC service thread 1-1) [f0be09280f90] KEYCLOAK 3.1.0.Final-MySQL f0be09280f90: JOIN(f0be09280f90) sent to 16c566cfa08e timed out (after 3000 ms), on try 7 2017-09-19 11:00:20,596 WARN [org.jgroups.protocols.pbcast.GMS] (MSC service thread 1-1) [f0be09280f90] KEYCLOAK 3.1.0.Final-MySQL f0be09280f90: JOIN(f0be09280f90) sent to 16c566cfa08e timed out (after 3000 ms), on try 8 2017-09-19 11:00:23,611 WARN [org.jgroups.protocols.pbcast.GMS] (MSC service thread 1-1) [f0be09280f90] KEYCLOAK 3.1.0.Final-MySQL f0be09280f90: JOIN(f0be09280f90) sent to 16c566cfa08e timed out (after 3000 ms), on try 9 2017-09-19 11:00:26,627 WARN [org.jgroups.protocols.pbcast.GMS] (MSC service thread 1-1) [f0be09280f90] KEYCLOAK 3.1.0.Final-MySQL f0be09280f90: JOIN(f0be09280f90) sent to 16c566cfa08e timed out (after 3000 ms), on try 10 2017-09-19 11:00:26,627 WARN [org.jgroups.protocols.pbcast.GMS] (MSC service thread 1-1) [f0be09280f90] KEYCLOAK 3.1.0.Final-MySQL f0be09280f90: too many JOIN attempts (10): becoming singleton On Thu, Sep 14, 2017 at 10:48 PM, Jyoti Kumar Singh <jyoti.tech90(a)gmail.com <mailto:jyoti.tech90@gmail.com>> wrote: Hi Tonnis, Thank you very much for sharing the valuable information. I am checking on this, hopefully I will also be able to achieve the HA. Thanks Again ! On Sep 14, 2017 10:00 PM, "Tonnis Wildeboer" <tonnis(a)autonomic.ai <mailto:tonnis@autonomic.ai>> wrote: Jyoti, I have been working on similar goal and was finally successful yesterday. We are using postgres and kubernetes. Here are the key sources of information that enabled me to succeed: The big key is here: https://github.com/devsu/docker-keycloak/tree/master/server-ha-mysql <https://github.com/devsu/docker-keycloak/tree/master/server-ha-mysql> Use the .xsl templates here to transform on the standalone-ha.xml and you can see what is being done. I suggest that you simply use JDBC_PING, since you already have a shared database. I think it is instructive to understand what JDBC_PING (and JGroups in general) are doing: http://jgroups.org/manual4/index.html <http://jgroups.org/manual4/index.html> https://developer.jboss.org/wiki/JDBCPING <https://developer.jboss.org/wiki/JDBCPING> You may benefit from this also, specifically, the need to bind jgroups-tcp and jgroups-tcp-fd to the proper interface. Not sure about your situation. --Tonnis ____________________ Tonnis Wildeboer Autonomic.ai Engineering On 09/14/2017 03:32 AM, Jyoti Kumar Singh wrote: Hi Team, I am trying to enable high availability for Keycloak 3.1.0 on AWS ECS instances. I am running two ECS instances in a cluster setup and also I have setup Keycloak in a clustered mode. To achieve this, I am using " */standalone/configuration/standalone-ha.xml *" file while building the docker image. Shared MySQL DB and Load Balancer setup are also in place. But when I checked Keycloak logs I am not seeing clustered nodes related information in logs. I am seeing nodes are not able to see each other. But same settings are working fine in DCOS Marathon platform. Interestingly if I run two Keycloak instances in one AWS ECS instance on different ports, I could see clustering related logs in Keycloak. Is there any standard guidelines which I can follow to achieve HA in AWS ECS instance ?? I followed the below discussion thread but it didn't help me to fix the issue. #Link: http://lists.jboss.org/pipermail/keycloak-user/2016-February/004940.html <http://lists.jboss.org/pipermail/keycloak-user/2016-February/004940.html> -- *With Regards, Jyoti Kumar Singh*