[keycloak-user] Expose JGroups ports in Docker keycloak-ha-postgres

[Staffan]

Hi,

After lots of trial-and-error we're now running Keycloak HA reliably in
Kubernetes. I've summed up our conclusions in
https://github.com/jboss-dockerfiles/keycloak/pull/62.

The JDBC_PING class works out of the box with MySQL and the
`datasource_jndi_name` property is a great fit with Keycloak's config.

I tried KUBE_PING but gave up when the jars weren't available in Keycloak,
and I found docs insufficient for how to build + bundle + configure.

I also tried GOOGLE_PING and, with https://github.com/minio/minio, S3_PING
but ran into access control issues.

The other ping impls may have their advantages, but JDBC_PING worked out of
the box.

/Staffan

On Thu, Nov 10, 2016 at 9:46 AM, Iván Perdomo <ivan at akvo.org> wrote:

> Hi,
>
> I'm also interesting on running Keycloak HA with Kubernetes/Google Cloud
> Platform.
>
> > JGroups members use multicast communication over UDP to broadcast
> > their presence to other instances on a network. Google Cloud
> > Platform, like most cloud providers and enterprise networks, does not
> > support multicast
>
> https://cloudplatform.googleblog.com/2016/02/JGroups-based-clustering-and-
> node-discovery-with-Google-Cloud-Storage.html
>
> Do we need to take the route and use GOOGLE_PING method for node discovery?
>
> Any hints on this topic are quite valuable.
>
> Thanks,
>
> On 11/09/2016 02:54 PM, Staffan wrote:
> > I have verified that `hostname -i` works with Minikube, but not yet a
> > multi-node cluster.
> >
> > Created PR https://github.com/jboss-dockerfiles/keycloak/pull/59 for the
> > official HA docker image.
> >
> > Is the following warning in keycloak logs something that affects
> clustering?
> >
> > WARN  [org.jboss.as.txn] (ServerService Thread Pool -- 49) WFLYTX0013:
> Node
> > identifier property is set to the default value. Please make sure it is
> > unique.
> >
> > It can be remedied using
> > https://github.com/Reposoft/keycloak-ha-kubernetes/commit/
> 413665f0c0827f8fa35379cc1f78098124290cd8
> > but I have avoided config file changes.
> >
> > /Staffan
> >
> > On Tue, Nov 8, 2016 at 3:06 PM, Alan Gibson <alan.gibson at gmail.com>
> wrote:
> >
> >> Hi Staffan,
> >>
> >> We've got 3 clustered Keycloak nodes running in Docker with host (not
> >> bridge) networking and managed by Mesos/Marathon. Cluster communications
> >> run over UDP. We start them with the following command.
> >>
> >> /opt/jboss/docker-entrypoint.sh -Dkeycloak.migration.action={{
> keycloak_migration_action}}
> >> -Dkeycloak.migration.provider={{keycloak_migration_provider}}
> >> -Dkeycloak.migration.file={{keycloak_migration_file}}
> >> -Dkeycloak.migration.strategy={{keycloak_migration_strategy}}
> >> -Djboss.jgroups.stack={{keycloak_jgroups_stack}}
> >> -Djboss.jgroups.udp.port={{keycloak_jgroups_udp_port}}
> >> -Djboss.jgroups.udp.multicast.port={{keycloak_jgroups_udp_
> multicast_port}}
> >> -Djboss.jgroups.udp.fd.port={{keycloak_jgroups_udp_fd_port}}
> >> -Djboss.management.http.port=$PORT1 -Djboss.http.port=$PORT0
> >> -Djboss.bind.address.private=$(hostname -i) -b 0.0.0.0 -bmanagement
> >> 0.0.0.0 --server-config standalone-ha.xml
> >>
> >> keycloak_jgroups_stack: udp
> >> keycloak_jgroups_udp_port: 5520
> >> keycloak_jgroups_udp_multicast_port: 4568
> >> keycloak_jgroups_udp_fd_port: 5420
> >>
> >> The magic ingredient is using getting the jboss.bind.address.private
> >> address from the shell with $(hostname -i). Note that this is definitely
> >> not foolproof, so YMMV.
> >>
> >> Br, Alan
> >>
> >> On Tue, Nov 8, 2016 at 11:59 AM, Staffan <solsson at gmail.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> I've tried in different docker environments (compose, kubernetes,
> >>> standalone) to get a HA setup running using https://hub.docker.com/r/
> >>> jboss/keycloak-ha-postgres/
> >>> <https://hub.docker.com/r/jboss/keycloak-ha-postgres/>.
> >>>
> >>> Keycloak nodes start, but are unaware of each other. I fail to reach
> the
> >>> JGroups ports from any other container or host system. That is
> expected,
> >>> as
> >>> https://keycloak.gitbooks.io/server-installation-and-configu
> >>> ration/content/v/2.3/topics/clustering/multicast.html
> >>> advises you to configure jboss.bind.address.private.
> >>>
> >>> But when I try -Djboss.bind.address.private=0.0.0.0 there's an error
> >>> during
> >>> startup:
> >>>
> >>> MSC000001: Failed to start service jboss.jgroups.channel.ee:
> >>> org.jboss.msc.service.StartException in service
> jboss.jgroups.channel.ee:
> >>> java.security.PrivilegedActionException: java.net.BindException:
> [UDP] /
> >>> 0.0.0.0 is not a valid address on any local network interface
> >>>     at org.wildfly.clustering.jgroups.spi.service.
> ChannelBuilder.start(
> >>> ChannelBuilder.java:80)
> >>> Caused by: java.security.PrivilegedActionException:
> >>> java.net.BindException:
> >>> [UDP] /0.0.0.0 is not a valid address on any local network interface
> >>>     at org.wildfly.security.manager.WildFlySecurityManager.doChecked(
> >>> WildFlySecurityManager.java:640)
> >>> Caused by: java.net.BindException: [UDP] /0.0.0.0 is not a valid
> address
> >>> on
> >>> any local network interface
> >>>     at org.jgroups.util.Util.checkIfValidAddress(Util.java:3522)
> >>>
> >>> ... or if I switch to stack="tcp" in the jgroups subsystem:
> >>>
> >>> MSC000001: Failed to start service jboss.jgroups.channel.ee:
> >>> org.jboss.msc.service.StartException in service
> jboss.jgroups.channel.ee:
> >>> java.security.PrivilegedActionException: java.net.BindException:
> [TCP] /
> >>> 0.0.0.0 is not a valid address on any local network interface
> >>>
> >>> I guess this is a generic Wildfly topic, but I'm curious how the
> official
> >>> Keycloak docker containers are tested. In a docker environment, what
> can
> >>> we
> >>> bind to other than 0.0.0.0 or 127.0.0.1? Is there a way to allow a
> >>> "privileged action"?
> >>>
> >>> regards
> >>> Staffan Olsson
> >>> _______________________________________________
> >>> keycloak-user mailing list
> >>> keycloak-user at lists.jboss.org
> >>> https://lists.jboss.org/mailman/listinfo/keycloak-user
> >>>
> >>
> >>
> > _______________________________________________
> > keycloak-user mailing list
> > keycloak-user at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/keycloak-user
> >
>
> --
> Iván
>