[keycloak-user] Expose JGroups ports in Docker keycloak-ha-postgres
Iván Perdomo
ivan at akvo.org
Tue Nov 29 03:53:56 EST 2016
Hi,
We have also have a running proof-of-concept using Kubernetes on GCP and
CloudSQL (hosted MySQL) as database.
I our case we use GOOGLE_PING for node discovery and disable `ip_mcast`
at JGroups level.
https://github.com/akvo/akvo-platform/tree/master/k8s/keycloak-ha-mysql
On 11/29/2016 09:44 AM, Staffan wrote:
> Hi,
>
> After lots of trial-and-error we're now running Keycloak HA reliably in
> Kubernetes. I've summed up our conclusions in
> https://github.com/jboss-dockerfiles/keycloak/pull/62.
>
> The JDBC_PING class works out of the box with MySQL and the
> `datasource_jndi_name` property is a great fit with Keycloak's config.
>
> I tried KUBE_PING but gave up when the jars weren't available in
> Keycloak, and I found docs insufficient for how to build + bundle +
> configure.
>
> I also tried GOOGLE_PING and, with https://github.com/minio/minio,
> S3_PING but ran into access control issues.
>
> The other ping impls may have their advantages, but JDBC_PING worked out
> of the box.
>
> /Staffan
>
> On Thu, Nov 10, 2016 at 9:46 AM, Iván Perdomo <ivan at akvo.org
> <mailto:ivan at akvo.org>> wrote:
>
> Hi,
>
> I'm also interesting on running Keycloak HA with Kubernetes/Google Cloud
> Platform.
>
> > JGroups members use multicast communication over UDP to broadcast
> > their presence to other instances on a network. Google Cloud
> > Platform, like most cloud providers and enterprise networks, does not
> > support multicast
>
> https://cloudplatform.googleblog.com/2016/02/JGroups-based-clustering-and-node-discovery-with-Google-Cloud-Storage.html
> <https://cloudplatform.googleblog.com/2016/02/JGroups-based-clustering-and-node-discovery-with-Google-Cloud-Storage.html>
>
> Do we need to take the route and use GOOGLE_PING method for node
> discovery?
>
> Any hints on this topic are quite valuable.
>
> Thanks,
>
> On 11/09/2016 02:54 PM, Staffan wrote:
> > I have verified that `hostname -i` works with Minikube, but not yet a
> > multi-node cluster.
> >
> > Created PR https://github.com/jboss-dockerfiles/keycloak/pull/59
> <https://github.com/jboss-dockerfiles/keycloak/pull/59> for the
> > official HA docker image.
> >
> > Is the following warning in keycloak logs something that affects
> clustering?
> >
> > WARN [org.jboss.as.txn] (ServerService Thread Pool -- 49)
> WFLYTX0013: Node
> > identifier property is set to the default value. Please make sure
> it is
> > unique.
> >
> > It can be remedied using
> >
> https://github.com/Reposoft/keycloak-ha-kubernetes/commit/413665f0c0827f8fa35379cc1f78098124290cd8
> <https://github.com/Reposoft/keycloak-ha-kubernetes/commit/413665f0c0827f8fa35379cc1f78098124290cd8>
> > but I have avoided config file changes.
> >
> > /Staffan
> >
> > On Tue, Nov 8, 2016 at 3:06 PM, Alan Gibson <alan.gibson at gmail.com
> <mailto:alan.gibson at gmail.com>> wrote:
> >
> >> Hi Staffan,
> >>
> >> We've got 3 clustered Keycloak nodes running in Docker with host (not
> >> bridge) networking and managed by Mesos/Marathon. Cluster
> communications
> >> run over UDP. We start them with the following command.
> >>
> >> /opt/jboss/docker-entrypoint.sh
> -Dkeycloak.migration.action={{keycloak_migration_action}}
> >> -Dkeycloak.migration.provider={{keycloak_migration_provider}}
> >> -Dkeycloak.migration.file={{keycloak_migration_file}}
> >> -Dkeycloak.migration.strategy={{keycloak_migration_strategy}}
> >> -Djboss.jgroups.stack={{keycloak_jgroups_stack}}
> >> -Djboss.jgroups.udp.port={{keycloak_jgroups_udp_port}}
> >>
> -Djboss.jgroups.udp.multicast.port={{keycloak_jgroups_udp_multicast_port}}
> >> -Djboss.jgroups.udp.fd.port={{keycloak_jgroups_udp_fd_port}}
> >> -Djboss.management.http.port=$PORT1 -Djboss.http.port=$PORT0
> >> -Djboss.bind.address.private=$(hostname -i) -b 0.0.0.0 -bmanagement
> >> 0.0.0.0 --server-config standalone-ha.xml
> >>
> >> keycloak_jgroups_stack: udp
> >> keycloak_jgroups_udp_port: 5520
> >> keycloak_jgroups_udp_multicast_port: 4568
> >> keycloak_jgroups_udp_fd_port: 5420
> >>
> >> The magic ingredient is using getting the jboss.bind.address.private
> >> address from the shell with $(hostname -i). Note that this is
> definitely
> >> not foolproof, so YMMV.
> >>
> >> Br, Alan
> >>
> >> On Tue, Nov 8, 2016 at 11:59 AM, Staffan <solsson at gmail.com
> <mailto:solsson at gmail.com>> wrote:
> >>
> >>> Hi,
> >>>
> >>> I've tried in different docker environments (compose, kubernetes,
> >>> standalone) to get a HA setup running using
> https://hub.docker.com/r/
> >>> jboss/keycloak-ha-postgres/
> >>> <https://hub.docker.com/r/jboss/keycloak-ha-postgres/
> <https://hub.docker.com/r/jboss/keycloak-ha-postgres/>>.
> >>>
> >>> Keycloak nodes start, but are unaware of each other. I fail to
> reach the
> >>> JGroups ports from any other container or host system. That is
> expected,
> >>> as
> >>> https://keycloak.gitbooks.io/server-installation-and-configu
> <https://keycloak.gitbooks.io/server-installation-and-configu>
> >>> ration/content/v/2.3/topics/clustering/multicast.html
> >>> advises you to configure jboss.bind.address.private.
> >>>
> >>> But when I try -Djboss.bind.address.private=0.0.0.0 there's an error
> >>> during
> >>> startup:
> >>>
> >>> MSC000001: Failed to start service jboss.jgroups.channel.ee
> <http://jboss.jgroups.channel.ee>:
> >>> org.jboss.msc.service.StartException in service
> jboss.jgroups.channel.ee <http://jboss.jgroups.channel.ee>:
> >>> java.security.PrivilegedActionException: java.net.BindException:
> [UDP] /
> >>> 0.0.0.0 is not a valid address on any local network interface
> >>> at
> org.wildfly.clustering.jgroups.spi.service.ChannelBuilder.start(
> >>> ChannelBuilder.java:80)
> >>> Caused by: java.security.PrivilegedActionException:
> >>> java.net.BindException:
> >>> [UDP] /0.0.0.0 <http://0.0.0.0> is not a valid address on any
> local network interface
> >>> at
> org.wildfly.security.manager.WildFlySecurityManager.doChecked(
> >>> WildFlySecurityManager.java:640)
> >>> Caused by: java.net.BindException: [UDP] /0.0.0.0
> <http://0.0.0.0> is not a valid address
> >>> on
> >>> any local network interface
> >>> at org.jgroups.util.Util.checkIfValidAddress(Util.java:3522)
> >>>
> >>> ... or if I switch to stack="tcp" in the jgroups subsystem:
> >>>
> >>> MSC000001: Failed to start service jboss.jgroups.channel.ee
> <http://jboss.jgroups.channel.ee>:
> >>> org.jboss.msc.service.StartException in service
> jboss.jgroups.channel.ee <http://jboss.jgroups.channel.ee>:
> >>> java.security.PrivilegedActionException: java.net.BindException:
> [TCP] /
> >>> 0.0.0.0 is not a valid address on any local network interface
> >>>
> >>> I guess this is a generic Wildfly topic, but I'm curious how the
> official
> >>> Keycloak docker containers are tested. In a docker environment,
> what can
> >>> we
> >>> bind to other than 0.0.0.0 or 127.0.0.1? Is there a way to allow a
> >>> "privileged action"?
> >>>
> >>> regards
> >>> Staffan Olsson
> >>> _______________________________________________
> >>> keycloak-user mailing list
> >>> keycloak-user at lists.jboss.org <mailto:keycloak-user at lists.jboss.org>
> >>> https://lists.jboss.org/mailman/listinfo/keycloak-user
> <https://lists.jboss.org/mailman/listinfo/keycloak-user>
> >>>
> >>
> >>
> > _______________________________________________
> > keycloak-user mailing list
> > keycloak-user at lists.jboss.org <mailto:keycloak-user at lists.jboss.org>
> > https://lists.jboss.org/mailman/listinfo/keycloak-user
> <https://lists.jboss.org/mailman/listinfo/keycloak-user>
> >
>
> --
> Iván
>
>
--
Iván
More information about the keycloak-user
mailing list