[keycloak-user] KeyCloak HA on AWS EC2 with docker - cluster is up but login fails
Scott Rossillo
srossillo at smartling.com
Wed Aug 17 17:34:31 EDT 2016
If the JDBC-PING question is for me, it just seemed to be the most logical choice since we have a relational database anyway. S3_PING or the other AWS options should work well but are more complex to configure, IMO.
Scott Rossillo
Smartling | Senior Software Engineer
srossillo at smartling.com
> On Aug 17, 2016, at 10:54 AM, Bill Burke <bburke at redhat.com> wrote:
>
> Do you have to use JDBC-PING?
>
> On 8/17/16 10:50 AM, Scott Rossillo wrote:
>> Have you looked at this Gist of mine [0]? I posted to the mailing list once before. Maybe I should make a more official document but it may help if you’re using docker.
>>
>>
>> [0]: https://gist.github.com/foo4u/ad2fa7251ac5b4d4fd318f668f50f7ac <https://gist.github.com/foo4u/ad2fa7251ac5b4d4fd318f668f50f7ac>
>>
>> Scott Rossillo
>> Smartling | Senior Software Engineer
>> srossillo at smartling.com <mailto:srossillo at smartling.com>
>>
>>> On Aug 17, 2016, at 4:08 AM, Haim Vana <haimv at perfectomobile.com <mailto:haimv at perfectomobile.com>> wrote:
>>>
>>> Thanks, the below is the exact post we were using as a reference.
>>>
>>> Any other idea what might cause it ? or what to search in the logs or JMX ?
>>>
>>>
>>> From: Aikeaguinea [mailto:aikeaguinea at xsmail.com <mailto:aikeaguinea at xsmail.com>]
>>> Sent: Tuesday, August 16, 2016 4:59 PM
>>> To: Haim Vana <haimv at perfectomobile.com <mailto:haimv at perfectomobile.com>>; keycloak-user at lists.jboss.org <mailto:keycloak-user at lists.jboss.org>
>>> Subject: Re: [keycloak-user] KeyCloak HA on AWS EC2 with docker - cluster is up but login fails
>>>
>>> Yes, this gets more complicated than your standard installation. AWS doesn't allow UDP communication in S3, and you also need to configure your Infinispan cache to work while you're running in Docker.
>>>
>>> There was a thread on this list "Using Keycloak in AWS EC2. What are people using? / Infinispan not working" where this was discussed; this is from that three describing howI got things working:
>>>
>>> ________________________________________________________
>>>
>>> I just got JGroups/Infinispan with JDBC_PING working from inside a
>>> Docker cluster in ECS on EC2. I use JDBC_PING rather than S3_PING, since
>>> I need a database anyway and didn't want to have to set up an S3 bucket
>>> just for this one purpose. Nicolás, if you're on AWS the default UDP
>>> transport for JGroups doesn't work because multicast isn't supported
>>> inside EC2, which may be your problem.
>>>
>>> Here are the configurations you'd need:
>>>
>>> 1. The JGroups module has to reference to the db module. So in
>>> jgroups-module.xml I have:
>>>
>>> <dependencies>
>>> <module name="javax.api"/>
>>> <module name="org.postgresql.jdbc"/>
>>> </dependencies>
>>>
>>> 2. The standalone-ha.xml has a JGroups subsystem (with TCP and
>>> JDBC_PING) that looks like the configuration below; I read certain
>>> variables from the environment, but may use the Wildfly vault tool for
>>> some of them. The external_addr property configurations are only needed
>>> if you're inside a Docker container, since Wildfly has to read the
>>> address of the EC2 instance hosting the container to register itself
>>> with JGroups. For the initialize_sql you can generally use the default,
>>> but for Postgres I needed a custom DDL because I needed the BYTEA data
>>> type which isn't in the default DDL.
>>>
>>> <subsystem xmlns="urn:jboss:domain:jgroups:4.0">
>>> <channels default="ee">
>>> <channel name="ee" stack="tcp"/>
>>> </channels>
>>>
>>> <stacks default="tcp">
>>> <stack name="tcp">
>>> <transport type="TCP" socket-binding="jgroups-tcp">
>>> <property
>>> name="external_addr">${env.EXTERNAL_HOST_IP}</property>
>>> </transport>
>>>
>>> <protocol type="JDBC_PING">
>>> <property
>>> name="connection_driver">org.postgresql.Driver</property>
>>> <property
>>> name="connection_url">jdbc:postgresql://${env.POSTGRES_TCP_ADDR}:${env.POSTGRES_TCP_PORT}/${env.POSTGRES_DATABASE}</property>
>>> <property
>>> name="connection_username">${env.POSTGRES_USER}</property>
>>> <property
>>> name="connection_password">${env.POSTGRES_PASSWORD}</property>
>>> <property name="initialize_sql">
>>> CREATE TABLE IF NOT EXISTS jgroupsping (
>>> own_addr VARCHAR(200) NOT NULL,
>>> cluster_name VARCHAR(200) NOT NULL,
>>> ping_data BYTEA DEFAULT NULL,
>>> PRIMARY KEY (own_addr, cluster_name)
>>> )
>>> </property>
>>> </protocol>
>>>
>>> <protocol type="MERGE3"/>
>>> <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd">
>>> <property
>>> name="external_addr">${env.EXTERNAL_HOST_IP}</property>
>>> </protocol>
>>>
>>> <protocol type="FD"/>
>>> <protocol type="VERIFY_SUSPECT"/>
>>> <protocol type="pbcast.NAKACK2"/>
>>> <protocol type="UNICAST3"/>
>>> <protocol type="pbcast.STABLE"/>
>>> <protocol type="pbcast.GMS"/>
>>> <protocol type="MFC"/>
>>> <protocol type="FRAG2"/>
>>> </stack>
>>> </stacks>
>>> </subsystem>
>>>
>>> 3. If you're in a Docker container, you have to expose the JGroups ports
>>> so they are visible from outside the container, so in standalone-ha.xml
>>> in the socket bindings I have changed to the public interface:
>>>
>>> <socket-binding name="jgroups-tcp" interface="public"
>>> port="7600"/>
>>> <socket-binding name="jgroups-tcp-fd" interface="public"
>>> port="57600"/>
>>>
>>> 4. For Docker, the startup script needs to pass the EXTERNAL_HOST_IP
>>> variable. I have a wrapper start script that first queries the AWS
>>> instance metadata service at 169.254.169.254 for the host's private IP address:
>>>
>>> export EXTERNAL_HOST_IP=$(curl -s
>>> 169.254.169.254/latest/meta-data/local-ipv4)
>>> exec $WILDFLY_HOME/bin/standalone.sh -c standalone-keycloak-ha.xml
>>> -Djboss.node.name=$HOSTNAME -Djgroups.bind_addr=global -b $HOSTNAME
>>>
>>>
>>> On Tue, Aug 16, 2016, at 09:01 AM, Haim Vana wrote:
>>> Hi,
>>>
>>> We are trying to set KeyCloak 1.9.3 with HA on AWS EC2 with docker, the cluster is up without errors however the login fails with the below error:
>>>
>>> WARN [org.keycloak.events] (default task-10) type=LOGIN_ERROR, realmId=master, clientId=null, userId=null, ipAddress=172.30.200.171, error=invalid_code
>>>
>>> we have followed this (http://lists.jboss.org/pipermail/keycloak-user/2016-February/004940.html <https://emea01.safelinks.protection.outlook.com/?url=http%3a%2f%2flists.jboss.org%2fpipermail%2fkeycloak-user%2f2016-February%2f004940.html&data=01%7c01%7chaimv%40perfectomobile.com%7cfdfb32069e4647fa9d9308d3c5dd7f6f%7cceb4c662d6994e7da0bd272619a46977%7c1&sdata=Qfqh1zMdVGYCoDKw6fweSKtX5MG3h0wl33JFtYDmzvA%3d> ) post but used S3_PING instead of JDBC_PING.
>>>
>>> It seems that the nodes detect each other:
>>>
>>> INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,ee,6dbce1e2a05a) ISPN000094: Received new cluster view for channel keycloak: [6dbce1e2a05a|1] (2) [6dbce1e2a05a, 75f2b2e98cfd]
>>>
>>> We suspect that the nodes doesn't communicate with each other, when we queried the jboss mbean "jboss.as.expr:subsystem=jgroups,channel=ee" the result was:
>>> jgroups,channel=ee = [6dbce1e2a05a|1] (2) [6dbce1e2a05a, 75f2b2e98cfd]
>>> jgroups,channel=ee receivedMessages = 0
>>> jgroups,channel=ee sentMessages = 0
>>>
>>> And for the second node:
>>> jgroups,channel=ee = [6dbce1e2a05a|1] (2) [6dbce1e2a05a, 75f2b2e98cfd]
>>> jgroups,channel=ee receivedMessages = 0
>>> jgroups,channel=ee sentMessages = 5
>>>
>>>
>>> We also verified that the TCP ports 57600 and 7600 are open.
>>>
>>> Any idea what might cause it ?
>>>
>>>
>>> Here is the relevant standalone-ha.xml configuration and below is that startup command:
>>>
>>> <subsystem xmlns="urn:jboss:domain:jgroups:4.0">
>>> <channels default="ee">
>>> <channel name="ee" stack="tcp"/>
>>> </channels>
>>> <stacks>
>>> <stack name="udp">
>>> <transport type="UDP" socket-binding="jgroups-udp"/>
>>> <protocol type="PING"/>
>>> <protocol type="MERGE3"/>
>>> <protocol type="FD_SOCK" socket-binding="jgroups-udp-fd"/>
>>> <protocol type="FD_ALL"/>
>>> <protocol type="VERIFY_SUSPECT"/>
>>> <protocol type="pbcast.NAKACK2"/>
>>> <protocol type="UNICAST3"/>
>>> <protocol type="pbcast.STABLE"/>
>>> <protocol type="pbcast.GMS"/>
>>> <protocol type="UFC"/>
>>> <protocol type="MFC"/>
>>> <protocol type="FRAG2"/>
>>> </stack>
>>> <stack name="tcp">
>>> <transport type="TCP" socket-binding="jgroups-tcp">
>>> <property name="external_addr">200.129.4.189</property>
>>> </transport>
>>> <protocol type="S3_PING">
>>> <property name="access_key">AAAAAAAAAAAAAA</property>
>>> <property name="secret_access_key">BBBBBBBBBBBBBB</property>
>>> <property name="location">CCCCCCCCCCCCCCCCCCCC</property>
>>> </protocol>
>>> <protocol type="MERGE3"/>
>>> <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd">
>>> <property name="external_addr">200.129.4.189</property>
>>> </protocol>
>>> <protocol type="FD"/>
>>> <protocol type="VERIFY_SUSPECT"/>
>>> <protocol type="pbcast.NAKACK2"/>
>>> <protocol type="UNICAST3"/>
>>> <protocol type="pbcast.STABLE"/>
>>> <protocol type="pbcast.GMS"/>
>>> <protocol type="MFC"/>
>>> <protocol type="FRAG2"/>
>>> </stack>
>>> </stacks>
>>> </subsystem>
>>>
>>>
>>> <socket-binding name="jgroups-tcp" interface="public" port="7600"/>
>>> <socket-binding name="jgroups-tcp-fd" interface="public" port="57600"/>
>>>
>>> And we start the server using the below ($INTERNAL_HOST_IP is the container internal IP address):
>>> standalone.sh -c=standalone-ha.xml -b=$INTERNAL_HOST_IP -bmanagement=$INTERNAL_HOST_IP -bprivate=$INTERNAL_HOST_IP
>>>
>>>
>>> Any help will be appreciated.
>>>
>>>
>>> Thanks,
>>> Haim.
>>>
>>>
>>> The information contained in this message is proprietary to the sender, protected from disclosure, and may be privileged. The information is intended to be conveyed only to the designated recipient(s) of the message. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, use, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by replying to the message and deleting it from your computer. Thank you.
>>> _______________________________________________
>>> keycloak-user mailing list
>>> keycloak-user at lists.jboss.org <mailto:keycloak-user at lists.jboss.org>
>>> https://lists.jboss.org/mailman/listinfo/keycloak-user <https://emea01.safelinks.protection.outlook.com/?url=https%3a%2f%2flists.jboss.org%2fmailman%2flistinfo%2fkeycloak-user&data=01%7c01%7chaimv%40perfectomobile.com%7cfdfb32069e4647fa9d9308d3c5dd7f6f%7cceb4c662d6994e7da0bd272619a46977%7c1&sdata=4HJT%2bfld%2bfvDf2moOzm6coVt9NvNwAGWVtU9TBPgMZY%3d>
>>>
>>> --
>>> Aikeaguinea
>>> aikeaguinea at xsmail.com <mailto:aikeaguinea at xsmail.com>
>>>
>>>
>>>
>>> --
>>> http://www.fastmail.com <http://www.fastmail.com/> - Same, same, but different...
>>> The information contained in this message is proprietary to the sender, protected from disclosure, and may be privileged. The information is intended to be conveyed only to the designated recipient(s) of the message. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, use, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by replying to the message and deleting it from your computer. Thank you. _______________________________________________
>>> keycloak-user mailing list
>>> keycloak-user at lists.jboss.org <mailto:keycloak-user at lists.jboss.org>
>>> https://lists.jboss.org/mailman/listinfo/keycloak-user <https://lists.jboss.org/mailman/listinfo/keycloak-user>
>>
>>
>> _______________________________________________
>> keycloak-user mailing list
>> keycloak-user at lists.jboss.org <mailto:keycloak-user at lists.jboss.org>
>> https://lists.jboss.org/mailman/listinfo/keycloak-user <https://lists.jboss.org/mailman/listinfo/keycloak-user>
> _______________________________________________
> keycloak-user mailing list
> keycloak-user at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/keycloak-user/attachments/20160817/993bbee9/attachment-0001.html
More information about the keycloak-user
mailing list