Hi, I've tried in different docker environments (compose, kubernetes, standalone) to get a HA setup running using https://hub.docker.com/r/ jboss/keycloak-ha-postgres/. Keycloak nodes start, but are unaware of each other. I fail to reach the JGroups ports from any other container or host system. That is expected, as https://keycloak.gitbooks.io/server-installation-and- configuration/content/v/2.3/topics/clustering/multicast.html advises you to configure jboss.bind.address.private. But when I try -Djboss.bind.address.private=0.0.0.0 there's an error during startup: MSC000001: Failed to start service jboss.jgroups.channel.ee: org.jboss.msc.service.StartException in service jboss.jgroups.channel.ee: java.security.PrivilegedActionException: java.net.BindException: [UDP] / 0.0.0.0 is not a valid address on any local network interface at org.wildfly.clustering.jgroups.spi.service.ChannelBuilder.start( ChannelBuilder.java:80) Caused by: java.security.PrivilegedActionException: java.net.BindException: [UDP] /0.0.0.0 is not a valid address on any local network interface at org.wildfly.security.manager.WildFlySecurityManager.doChecked( WildFlySecurityManager.java:640) Caused by: java.net.BindException: [UDP] /0.0.0.0 is not a valid address on any local network interface at org.jgroups.util.Util.checkIfValidAddress(Util.java:3522) ... or if I switch to stack="tcp" in the jgroups subsystem: MSC000001: Failed to start service jboss.jgroups.channel.ee: org.jboss.msc.service.StartException in service jboss.jgroups.channel.ee: java.security.PrivilegedActionException: java.net.BindException: [TCP] / 0.0.0.0 is not a valid address on any local network interface I guess this is a generic Wildfly topic, but I'm curious how the official Keycloak docker containers are tested. In a docker environment, what can we bind to other than 0.0.0.0 or 127.0.0.1? Is there a way to allow a "privileged action"? regards Staffan Olsson _______________________________________________ keycloak-user mailing list keycloak-user(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/keycloak-user

JGroups members use multicast communication over UDP to broadcast their presence to other instances on a network. Google Cloud Platform, like most cloud providers and enterprise networks, does not support multicast

I have verified that `hostname -i` works with Minikube, but not yet a multi-node cluster. Created PR https://github.com/jboss-dockerfiles/keycloak/pull/59 for the official HA docker image. Is the following warning in keycloak logs something that affects clustering? WARN [org.jboss.as.txn] (ServerService Thread Pool -- 49) WFLYTX0013: Node identifier property is set to the default value. Please make sure it is unique. It can be remedied using https://github.com/Reposoft/keycloak-ha-kubernetes/commit/413665f0c0827f8... but I have avoided config file changes. /Staffan On Tue, Nov 8, 2016 at 3:06 PM, Alan Gibson <alan.gibson(a)gmail.com&gt; wrote: > Hi Staffan, > > We've got 3 clustered Keycloak nodes running in Docker with host (not > bridge) networking and managed by Mesos/Marathon. Cluster communications > run over UDP. We start them with the following command. > > /opt/jboss/docker-entrypoint.sh -Dkeycloak.migration.action={{keycloak_migration_action}} > -Dkeycloak.migration.provider={{keycloak_migration_provider}} > -Dkeycloak.migration.file={{keycloak_migration_file}} > -Dkeycloak.migration.strategy={{keycloak_migration_strategy}} > -Djboss.jgroups.stack={{keycloak_jgroups_stack}} > -Djboss.jgroups.udp.port={{keycloak_jgroups_udp_port}} > -Djboss.jgroups.udp.multicast.port={{keycloak_jgroups_udp_multicast_port}} > -Djboss.jgroups.udp.fd.port={{keycloak_jgroups_udp_fd_port}} > -Djboss.management.http.port=$PORT1 -Djboss.http.port=$PORT0 > -Djboss.bind.address.private=$(hostname -i) -b 0.0.0.0 -bmanagement > 0.0.0.0 --server-config standalone-ha.xml > > keycloak_jgroups_stack: udp > keycloak_jgroups_udp_port: 5520 > keycloak_jgroups_udp_multicast_port: 4568 > keycloak_jgroups_udp_fd_port: 5420 > > The magic ingredient is using getting the jboss.bind.address.private > address from the shell with $(hostname -i). Note that this is definitely > not foolproof, so YMMV. > > Br, Alan > > On Tue, Nov 8, 2016 at 11:59 AM, Staffan <solsson(a)gmail.com&gt; wrote: > >> Hi, >> >> I've tried in different docker environments (compose, kubernetes, >> standalone) to get a HA setup running using https://hub.docker.com/r/ >> jboss/keycloak-ha-postgres/ >> <https://hub.docker.com/r/jboss/keycloak-ha-postgres/>;. >> >> Keycloak nodes start, but are unaware of each other. I fail to reach the >> JGroups ports from any other container or host system. That is expected, >> as >> https://keycloak.gitbooks.io/server-installation-and-configu >> ration/content/v/2.3/topics/clustering/multicast.html >> advises you to configure jboss.bind.address.private. >> >> But when I try -Djboss.bind.address.private=0.0.0.0 there's an error >> during >> startup: >> >> MSC000001: Failed to start service jboss.jgroups.channel.ee: >> org.jboss.msc.service.StartException in service jboss.jgroups.channel.ee: >> java.security.PrivilegedActionException: java.net.BindException: [UDP] / >> 0.0.0.0 is not a valid address on any local network interface >> at org.wildfly.clustering.jgroups.spi.service.ChannelBuilder.start( >> ChannelBuilder.java:80) >> Caused by: java.security.PrivilegedActionException: >> java.net.BindException: >> [UDP] /0.0.0.0 is not a valid address on any local network interface >> at org.wildfly.security.manager.WildFlySecurityManager.doChecked( >> WildFlySecurityManager.java:640) >> Caused by: java.net.BindException: [UDP] /0.0.0.0 is not a valid address >> on >> any local network interface >> at org.jgroups.util.Util.checkIfValidAddress(Util.java:3522) >> >> ... or if I switch to stack="tcp" in the jgroups subsystem: >> >> MSC000001: Failed to start service jboss.jgroups.channel.ee: >> org.jboss.msc.service.StartException in service jboss.jgroups.channel.ee: >> java.security.PrivilegedActionException: java.net.BindException: [TCP] / >> 0.0.0.0 is not a valid address on any local network interface >> >> I guess this is a generic Wildfly topic, but I'm curious how the official >> Keycloak docker containers are tested. In a docker environment, what can >> we >> bind to other than 0.0.0.0 or 127.0.0.1? Is there a way to allow a >> "privileged action"? >> >> regards >> Staffan Olsson >> _______________________________________________ >> keycloak-user mailing list >> keycloak-user(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/keycloak-user >> > > _______________________________________________ keycloak-user mailing list keycloak-user(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/keycloak-user

Hi, After lots of trial-and-error we're now running Keycloak HA reliably in Kubernetes. I've summed up our conclusions in https://github.com/jboss-dockerfiles/keycloak/pull/62. The JDBC_PING class works out of the box with MySQL and the `datasource_jndi_name` property is a great fit with Keycloak's config. I tried KUBE_PING but gave up when the jars weren't available in Keycloak, and I found docs insufficient for how to build + bundle + configure. I also tried GOOGLE_PING and, with https://github.com/minio/minio, S3_PING but ran into access control issues. The other ping impls may have their advantages, but JDBC_PING worked out of the box. /Staffan On Thu, Nov 10, 2016 at 9:46 AM, Iván Perdomo <ivan(a)akvo.org <mailto:ivan@akvo.org>> wrote: Hi, I'm also interesting on running Keycloak HA with Kubernetes/Google Cloud Platform. > JGroups members use multicast communication over UDP to broadcast > their presence to other instances on a network. Google Cloud > Platform, like most cloud providers and enterprise networks, does not > support multicast https://cloudplatform.googleblog.com/2016/02/JGroups-based-clustering-and... <https://cloudplatform.googleblog.com/2016/02/JGroups-based-clustering-and... Do we need to take the route and use GOOGLE_PING method for node discovery? Any hints on this topic are quite valuable. Thanks, On 11/09/2016 02:54 PM, Staffan wrote: > I have verified that `hostname -i` works with Minikube, but not yet a > multi-node cluster. > > Created PR https://github.com/jboss-dockerfiles/keycloak/pull/59 <https://github.com/jboss-dockerfiles/keycloak/pull/59> for the > official HA docker image. > > Is the following warning in keycloak logs something that affects clustering? > > WARN [org.jboss.as.txn] (ServerService Thread Pool -- 49) WFLYTX0013: Node > identifier property is set to the default value. Please make sure it is > unique. > > It can be remedied using > https://github.com/Reposoft/keycloak-ha-kubernetes/commit/413665f0c0827f8... <https://github.com/Reposoft/keycloak-ha-kubernetes/commit/413665f0c0827f8... > but I have avoided config file changes. > > /Staffan > > On Tue, Nov 8, 2016 at 3:06 PM, Alan Gibson <alan.gibson(a)gmail.com <mailto:alan.gibson@gmail.com>> wrote: > >> Hi Staffan, >> >> We've got 3 clustered Keycloak nodes running in Docker with host (not >> bridge) networking and managed by Mesos/Marathon. Cluster communications >> run over UDP. We start them with the following command. >> >> /opt/jboss/docker-entrypoint.sh -Dkeycloak.migration.action={{keycloak_migration_action}} >> -Dkeycloak.migration.provider={{keycloak_migration_provider}} >> -Dkeycloak.migration.file={{keycloak_migration_file}} >> -Dkeycloak.migration.strategy={{keycloak_migration_strategy}} >> -Djboss.jgroups.stack={{keycloak_jgroups_stack}} >> -Djboss.jgroups.udp.port={{keycloak_jgroups_udp_port}} >> -Djboss.jgroups.udp.multicast.port={{keycloak_jgroups_udp_multicast_port}} >> -Djboss.jgroups.udp.fd.port={{keycloak_jgroups_udp_fd_port}} >> -Djboss.management.http.port=$PORT1 -Djboss.http.port=$PORT0 >> -Djboss.bind.address.private=$(hostname -i) -b 0.0.0.0 -bmanagement >> 0.0.0.0 --server-config standalone-ha.xml >> >> keycloak_jgroups_stack: udp >> keycloak_jgroups_udp_port: 5520 >> keycloak_jgroups_udp_multicast_port: 4568 >> keycloak_jgroups_udp_fd_port: 5420 >> >> The magic ingredient is using getting the jboss.bind.address.private >> address from the shell with $(hostname -i). Note that this is definitely >> not foolproof, so YMMV. >> >> Br, Alan >> >> On Tue, Nov 8, 2016 at 11:59 AM, Staffan <solsson(a)gmail.com <mailto:solsson@gmail.com>> wrote: >> >>> Hi, >>> >>> I've tried in different docker environments (compose, kubernetes, >>> standalone) to get a HA setup running using https://hub.docker.com/r/ >>> jboss/keycloak-ha-postgres/ >>> <https://hub.docker.com/r/jboss/keycloak-ha-postgres/ <https://hub.docker.com/r/jboss/keycloak-ha-postgres/>>;. >>> >>> Keycloak nodes start, but are unaware of each other. I fail to reach the >>> JGroups ports from any other container or host system. That is expected, >>> as >>> https://keycloak.gitbooks.io/server-installation-and-configu <https://keycloak.gitbooks.io/server-installation-and-configu> >>> ration/content/v/2.3/topics/clustering/multicast.html >>> advises you to configure jboss.bind.address.private. >>> >>> But when I try -Djboss.bind.address.private=0.0.0.0 there's an error >>> during >>> startup: >>> >>> MSC000001: Failed to start service jboss.jgroups.channel.ee <http://jboss.jgroups.channel.ee>;: >>> org.jboss.msc.service.StartException in service jboss.jgroups.channel.ee <http://jboss.jgroups.channel.ee>;: >>> java.security.PrivilegedActionException: java.net.BindException: [UDP] / >>> 0.0.0.0 is not a valid address on any local network interface >>> at org.wildfly.clustering.jgroups.spi.service.ChannelBuilder.start( >>> ChannelBuilder.java:80) >>> Caused by: java.security.PrivilegedActionException: >>> java.net.BindException: >>> [UDP] /0.0.0.0 <http://0.0.0.0> is not a valid address on any local network interface >>> at org.wildfly.security.manager.WildFlySecurityManager.doChecked( >>> WildFlySecurityManager.java:640) >>> Caused by: java.net.BindException: [UDP] /0.0.0.0 <http://0.0.0.0> is not a valid address >>> on >>> any local network interface >>> at org.jgroups.util.Util.checkIfValidAddress(Util.java:3522) >>> >>> ... or if I switch to stack="tcp" in the jgroups subsystem: >>> >>> MSC000001: Failed to start service jboss.jgroups.channel.ee <http://jboss.jgroups.channel.ee>;: >>> org.jboss.msc.service.StartException in service jboss.jgroups.channel.ee <http://jboss.jgroups.channel.ee>;: >>> java.security.PrivilegedActionException: java.net.BindException: [TCP] / >>> 0.0.0.0 is not a valid address on any local network interface >>> >>> I guess this is a generic Wildfly topic, but I'm curious how the official >>> Keycloak docker containers are tested. In a docker environment, what can >>> we >>> bind to other than 0.0.0.0 or 127.0.0.1? Is there a way to allow a >>> "privileged action"? >>> >>> regards >>> Staffan Olsson >>> _______________________________________________ >>> keycloak-user mailing list >>> keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org> >>> https://lists.jboss.org/mailman/listinfo/keycloak-user <https://lists.jboss.org/mailman/listinfo/keycloak-user> >>> >> >> > _______________________________________________ > keycloak-user mailing list > keycloak-user(a)lists.jboss.org <mailto:keycloak-user@lists.jboss.org> > https://lists.jboss.org/mailman/listinfo/keycloak-user <https://lists.jboss.org/mailman/listinfo/keycloak-user> > -- Iván