[keycloak-user] Keycloak standalone-ha cluster jgroups without multicast
Meissa M'baye Sakho
msakho at redhat.com
Mon Jul 24 06:00:52 EDT 2017
Christian,
There is a problem with IP that you have (not) defined for the private
interface that is used by jgroup for intracluster communication.
By default, JGroups only binds to the private network interface, which
points to localhost in the default configuration (line 505 in your
configuration file). For security reasons, *JGroups will not bind to the
network interface defined by the -b argument specified during JBoss EAP
startup*, as clustering traffic should not be exposed on a public network
interface.
You can define this value either by setting the
property jboss.bind.address.private at startup or by hard coding it in the
line 505 in you configuration file:
1. <interface name="private">
2. <inet-address value=
"${jboss.bind.address.private:127.0.0.1}"/>
3. <inet-address value="your-private-ip"/>
4. </interface>
5.
Doing so you should have an output like (instead of localhost in your case).
*11:18:25,954 INFO
[org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service
thread 1-7) ISPN000079: Channel hibernate local address is app02.2.xxx.net
<http://app02.2.xxx.net>, physical addresses are [your-private-ip:7600]*
*11:18:25,954 INFO
[org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service
thread 1-4) ISPN000079: Channel ejb local address is app02.2.xxx.net
<http://app02.2.xxx.net>, physical addresses are [your-private-ip:7600]*
*11:18:25,954 INFO
[org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service
thread 1-2) ISPN000079: Channel web local address is app02.2.xxx.net
<http://app02.2.xxx.net>, physical addresses are [your-private-ip:7600]*
*11:18:25,955 INFO
[org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service
thread 1-8) ISPN000079: Channel server local address is app02.2.xxx.net
<http://app02.2.xxx.net>, physical addresses are [your-private-ip:7600]*
*11:18:25,955 INFO
[org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service
thread 1-3) ISPN000079: Channel keycloak local address is app02.2.xxx.net
<http://app02.2.xxx.net>, physical addresses are [1your-private-ip:7600]*
*If your configuration jgroup configuration is ok and if you have defined
the jboss.node.name <http://jboss.node.name> property for each of your
cluster member, you should see in you log the output below that will
confirm you that you members will form a cluster:*
*11:18:45,074 INFO
[org.infinispan.remoting.transport.jgroups.JGroupsTransport]
(thread-2,ee,tcpcluster2) ISPN000094: Received new cluster view for channel
server: [tcpcluster2|1] (2) [tcpcluster2, tcpcluster1]*
*11:18:45,076 INFO
[org.infinispan.remoting.transport.jgroups.JGroupsTransport]
(thread-2,ee,tcpcluster2) ISPN000094: Received new cluster view for channel
keycloak: [tcpcluster2|1] (2) [tcpcluster2, tcpcluster1]*
*11:18:45,078 INFO
[org.infinispan.remoting.transport.jgroups.JGroupsTransport]
(thread-2,ee,tcpcluster2) ISPN000094: Received new cluster view for channel
web: [tcpcluster2|1] (2) [tcpcluster2, tcpcluster1]*
*11:18:45,080 INFO
[org.infinispan.remoting.transport.jgroups.JGroupsTransport]
(thread-2,ee,tcpcluster2) ISPN000094: Received new cluster view for channel
hibernate: [tcpcluster2|1] (2) [tcpcluster2, tcpcluster1]*
*11:18:45,080 INFO
[org.infinispan.remoting.transport.jgroups.JGroupsTransport]
(thread-2,ee,tcpcluster2) ISPN000094: Received new cluster view for channel
ejb: [tcpcluster2|1] (2) [tcpcluster2, tcpcluster1]*
*ISPN000310: Starting cluster-wide rebalance for cache work, topology
CacheTopology{id=1, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns =
60, owners = (1)[tcpcluster2: 60]}, pendingCH=ReplicatedConsistentHash{ns =
60, owners = (2)[tcpcluster2: 30, tcpcluster1: 30]}, unionCH=null,
actualMembers=[tcpcluster2, tcpcluster1]}*
In my case, I've defined the jboss.node.name respectively with tcpcluster1
and tcpcluster2.
Meissa
On Mon, Jul 24, 2017 at 9:26 AM, Marek Posolda <mposolda at redhat.com> wrote:
> From the log, it seem that the servers don't see each other and hence
> don't form cluster. In your configuration, you have
> "10.62.168.51[7600],10.62.168.52[7600]", however in log later you have:
>
> physical addresses are [
> 127.0.0.1:7600
>
> It seems there is the issue with the binding address? It should be
> 10.62.168.51 instead of 127.0.0.1. Did you use the "-b" option when
> starting the server?
>
> I think that JGroups/Infinispan/EAP documentation should have more
> details as it's more related to that rather then to Keycloak.
>
> Marek
>
>
> Dne 23.7.2017 v 12:51 Christian Schneider napsal(a):
> > Hi,
> > I'm trying to setup a keycloak cluster.
> >
> > We copy binaries and configuration through our build pipeline to the
> > servers, so we decided to for the "standalone clustered mode".
> >
> > Since our operations don't enable ip multicast, we wan't to work with ip
> > static ip addresses.
> >
> > The only source for this I found is:
> > https://kb.novaordis.com/index.php/WildFly_Clustering_without_Multicast
> > <https://deref-gmx.net/mail/client/Wiot_0qYjM4/dereferrer/
> ?redirectUrl=https%3A%2F%2Fkb.novaordis.com%2Findex.php%
> 2FWildFly_Clustering_without_Multicast>
> >
> > I configured both servers like this:
> >
> > <subsystem xmlns="urn:jboss:domain:jgroups:4.0">
> > <channels default="ee">
> > <channel name="ee" stack="tcp"/>
> > </channels>
> > <stacks default="tcp">
> > <stack name="tcp">
> > <transfer type="TCP" socket-binding="jgroups-tcp"/>
> > <protocol type="TCPPING">
> > <property
> > name="initial_hosts">10.62.168.51[7600],10.62.168.52[7600]</property>
> > <property name="num_initial_members">2</
> property>
> > <property name="port_range">0</property>
> > <property name="timeout">2000</property>
> > </protocol>
> > <protocol type="MERGE3"/>
> > <protocol type="FD_SOCK"
> > socket-binding="jgroups-tcp-fd"/>
> > <protocol type="FD"/>
> > <protocol type="VERIFY_SUSPECT"/>
> > <protocol type="pbcast.NAKACK2"/>
> > <protocol type="UNICAST3"/>
> > <protocol type="pbcast.STABLE"/>
> > <protocol type="pbcast.GMS"/>
> > <protocol type="MFC"/>
> > <protocol type="FRAG2"/>
> > </stack>
> > </stacks>
> > </subsystem>
> >
> > I can't figure out whether the two nodes paired each other. In the
> logfile
> > I only find this:
> >
> > INFO 2017-07-21 20:35:43 []
> > org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000078:
> > Starting JGroups channel hibernate
> > INFO 2017-07-21 20:35:43 []
> > org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000078:
> > Starting JGroups channel web
> > INFO 2017-07-21 20:35:43 []
> > org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000078:
> > Starting JGroups channel keycloak
> > INFO 2017-07-21 20:35:43 []
> > org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000078:
> > Starting JGroups channel ejb
> > INFO 2017-07-21 20:35:43 []
> > org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000078:
> > Starting JGroups channel server
> > INFO 2017-07-21 20:35:43 []
> > org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000094:
> > Received new cluster view for channel keycloak: [app02.2.xxx.net|0] (1)
> [
> > app02.2.xxx.net]
> > INFO 2017-07-21 20:35:43 []
> > org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000094:
> > Received new cluster view for channel hibernate: [app02.2.xxx.net|0]
> (1) [
> > app02.2.xxx.net]
> > INFO 2017-07-21 20:35:43 []
> > org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000094:
> > Received new cluster view for channel ejb: [app02.2.xxx.net|0] (1) [
> > app02.2.xxx.net]
> > INFO 2017-07-21 20:35:43 []
> > org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000094:
> > Received new cluster view for channel server: [app02.2.xxx.net|0] (1) [
> > app02.2.xxx.net]
> > INFO 2017-07-21 20:35:43 []
> > org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000094:
> > Received new cluster view for channel web: [app02.2.xxx.net|0] (1) [
> > app02.2.xxx.net]
> > INFO 2017-07-21 20:35:43 []
> > org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000079:
> > Channel server local address is app02.2.xxx.net, physical addresses are
> [
> > 127.0.0.1:7600]
> > INFO 2017-07-21 20:35:43 []
> > org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000079:
> > Channel ejb local address is app02.2.xxx.net, physical addresses are [
> > 127.0.0.1:7600]
> > INFO 2017-07-21 20:35:43 []
> > org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000079:
> > Channel web local address is app02.2.xxx.net, physical addresses are [
> > 127.0.0.1:7600]
> > INFO 2017-07-21 20:35:43 []
> > org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000079:
> > Channel keycloak local address is app02.2.xxx.net, physical addresses
> are [
> > 127.0.0.1:7600]
> > INFO 2017-07-21 20:35:43 []
> > org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000079:
> > Channel hibernate local address is app02.2.xxx.net, physical addresses
> are [
> > 127.0.0.1:7600
> >
> > I checked that the ports 7060 are open.
> > When I access just a single node, the setup works fine. With two nodes I
> > get {"error":"invalid_grant","error_description":"Session not active"}
> on
> > one of them.
> > The goal is that all userSessions are synced between the cluster nodes,
> so
> > that the loadbalancer can roundrobbin between all nodes.
> > Since we have around 20k users only, it would be fine to store the users
> in
> > our database. But keycloak doesn't provide this, right?
> > Do you have any idea how to figure out whats wrong?
> >
> > Best Regards,
> > Christian.
> >
> > P.S.: You can find the whole configuration here:
> > https://pastebin.com/WC46pXGp
> > <https://deref-gmx.net/mail/client/UIuwfhWwMZ0/dereferrer/
> ?redirectUrl=https%3A%2F%2Fpastebin.com%2FWC46pXGp>
> > _______________________________________________
> > keycloak-user mailing list
> > keycloak-user at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/keycloak-user
>
>
> _______________________________________________
> keycloak-user mailing list
> keycloak-user at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user
>
More information about the keycloak-user
mailing list