Hi, I'm trying to setup a keycloak cluster. We copy binaries and configuration through our build pipeline to the servers, so we decided to for the "standalone clustered mode". Since our operations don't enable ip multicast, we wan't to work with ip static ip addresses. The only source for this I found is: https://kb.novaordis.com/index.php/WildFly_Clustering_without_Multicast <https://deref-gmx.net/mail/client/Wiot_0qYjM4/dereferrer/?redirectUrl=htt... I configured both servers like this: <subsystem xmlns="urn:jboss:domain:jgroups:4.0"> <channels default="ee"> <channel name="ee" stack="tcp"/> </channels> <stacks default="tcp"> <stack name="tcp"> <transfer type="TCP" socket-binding="jgroups-tcp"/> <protocol type="TCPPING"> <property name="initial_hosts">10.62.168.51[7600],10.62.168.52[7600]</property> <property name="num_initial_members">2</property> <property name="port_range">0</property> <property name="timeout">2000</property> </protocol> <protocol type="MERGE3"/> <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/> <protocol type="FD"/> <protocol type="VERIFY_SUSPECT"/> <protocol type="pbcast.NAKACK2"/> <protocol type="UNICAST3"/> <protocol type="pbcast.STABLE"/> <protocol type="pbcast.GMS"/> <protocol type="MFC"/> <protocol type="FRAG2"/> </stack> </stacks> </subsystem> I can't figure out whether the two nodes paired each other. In the logfile I only find this: INFO 2017-07-21 20:35:43 [] org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000078: Starting JGroups channel hibernate INFO 2017-07-21 20:35:43 [] org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000078: Starting JGroups channel web INFO 2017-07-21 20:35:43 [] org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000078: Starting JGroups channel keycloak INFO 2017-07-21 20:35:43 [] org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000078: Starting JGroups channel ejb INFO 2017-07-21 20:35:43 [] org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000078: Starting JGroups channel server INFO 2017-07-21 20:35:43 [] org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000094: Received new cluster view for channel keycloak: [app02.2.xxx.net|0] (1) [ app02.2.xxx.net] INFO 2017-07-21 20:35:43 [] org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000094: Received new cluster view for channel hibernate: [app02.2.xxx.net|0] (1) [ app02.2.xxx.net] INFO 2017-07-21 20:35:43 [] org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000094: Received new cluster view for channel ejb: [app02.2.xxx.net|0] (1) [ app02.2.xxx.net] INFO 2017-07-21 20:35:43 [] org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000094: Received new cluster view for channel server: [app02.2.xxx.net|0] (1) [ app02.2.xxx.net] INFO 2017-07-21 20:35:43 [] org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000094: Received new cluster view for channel web: [app02.2.xxx.net|0] (1) [ app02.2.xxx.net] INFO 2017-07-21 20:35:43 [] org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000079: Channel server local address is app02.2.xxx.net, physical addresses are [ 127.0.0.1:7600] INFO 2017-07-21 20:35:43 [] org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000079: Channel ejb local address is app02.2.xxx.net, physical addresses are [ 127.0.0.1:7600] INFO 2017-07-21 20:35:43 [] org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000079: Channel web local address is app02.2.xxx.net, physical addresses are [ 127.0.0.1:7600] INFO 2017-07-21 20:35:43 [] org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000079: Channel keycloak local address is app02.2.xxx.net, physical addresses are [ 127.0.0.1:7600] INFO 2017-07-21 20:35:43 [] org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000079: Channel hibernate local address is app02.2.xxx.net, physical addresses are [ 127.0.0.1:7600 I checked that the ports 7060 are open. When I access just a single node, the setup works fine. With two nodes I get {"error":"invalid_grant","error_description":"Session not active"} on one of them. The goal is that all userSessions are synced between the cluster nodes, so that the loadbalancer can roundrobbin between all nodes. Since we have around 20k users only, it would be fine to store the users in our database. But keycloak doesn't provide this, right? Do you have any idea how to figure out whats wrong? Best Regards, Christian. P.S.: You can find the whole configuration here: https://pastebin.com/WC46pXGp <https://deref-gmx.net/mail/client/UIuwfhWwMZ0/dereferrer/?redirectUrl=htt... _______________________________________________ keycloak-user mailing list keycloak-user(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/keycloak-user

Christian, There is a problem with IP that you have (not) defined for the private interface that is used by jgroup for intracluster communication. By default, JGroups only binds to the private network interface, which points to localhost in the default configuration (line 505 in your configuration file). For security reasons, *JGroups will not bind to the network interface defined by the -b argument specified during JBoss EAP startup*, as clustering traffic should not be exposed on a public network interface. You can define this value either by setting the property jboss.bind.address.private at startup or by hard coding it in the line 505 in you configuration file: 1. <interface name="private"> 2. <inet-address value= "${jboss.bind.address.private:127.0.0.1}"/> 3. <inet-address value="your-private-ip"/> 4. </interface> 5. Doing so you should have an output like (instead of localhost in your case). *11:18:25,954 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-7) ISPN000079: Channel hibernate local address is app02.2.xxx.net <http://app02.2.xxx.net>;, physical addresses are [your-private-ip:7600]* *11:18:25,954 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000079: Channel ejb local address is app02.2.xxx.net <http://app02.2.xxx.net>;, physical addresses are [your-private-ip:7600]* *11:18:25,954 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000079: Channel web local address is app02.2.xxx.net <http://app02.2.xxx.net>;, physical addresses are [your-private-ip:7600]* *11:18:25,955 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000079: Channel server local address is app02.2.xxx.net <http://app02.2.xxx.net>;, physical addresses are [your-private-ip:7600]* *11:18:25,955 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000079: Channel keycloak local address is app02.2.xxx.net <http://app02.2.xxx.net>;, physical addresses are [1your-private-ip:7600]* *If your configuration jgroup configuration is ok and if you have defined the jboss.node.name <http://jboss.node.name> property for each of your cluster member, you should see in you log the output below that will confirm you that you members will form a cluster:* *11:18:45,074 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-2,ee,tcpcluster2) ISPN000094: Received new cluster view for channel server: [tcpcluster2|1] (2) [tcpcluster2, tcpcluster1]* *11:18:45,076 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-2,ee,tcpcluster2) ISPN000094: Received new cluster view for channel keycloak: [tcpcluster2|1] (2) [tcpcluster2, tcpcluster1]* *11:18:45,078 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-2,ee,tcpcluster2) ISPN000094: Received new cluster view for channel web: [tcpcluster2|1] (2) [tcpcluster2, tcpcluster1]* *11:18:45,080 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-2,ee,tcpcluster2) ISPN000094: Received new cluster view for channel hibernate: [tcpcluster2|1] (2) [tcpcluster2, tcpcluster1]* *11:18:45,080 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-2,ee,tcpcluster2) ISPN000094: Received new cluster view for channel ejb: [tcpcluster2|1] (2) [tcpcluster2, tcpcluster1]* *ISPN000310: Starting cluster-wide rebalance for cache work, topology CacheTopology{id=1, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 60, owners = (1)[tcpcluster2: 60]}, pendingCH=ReplicatedConsistentHash{ns = 60, owners = (2)[tcpcluster2: 30, tcpcluster1: 30]}, unionCH=null, actualMembers=[tcpcluster2, tcpcluster1]}* In my case, I've defined the jboss.node.name respectively with tcpcluster1 and tcpcluster2. Meissa On Mon, Jul 24, 2017 at 9:26 AM, Marek Posolda <mposolda(a)redhat.com&gt; wrote: > From the log, it seem that the servers don't see each other and hence > don't form cluster. In your configuration, you have > "10.62.168.51[7600],10.62.168.52[7600]", however in log later you have: > > physical addresses are [ > 127.0.0.1:7600 > > It seems there is the issue with the binding address? It should be > 10.62.168.51 instead of 127.0.0.1. Did you use the "-b" option when > starting the server? > > I think that JGroups/Infinispan/EAP documentation should have more > details as it's more related to that rather then to Keycloak. > > Marek > > > Dne 23.7.2017 v 12:51 Christian Schneider napsal(a): > > Hi, > > I'm trying to setup a keycloak cluster. > > > > We copy binaries and configuration through our build pipeline to the > > servers, so we decided to for the "standalone clustered mode". > > > > Since our operations don't enable ip multicast, we wan't to work with ip > > static ip addresses. > > > > The only source for this I found is: > > https://kb.novaordis.com/index.php/WildFly_Clustering_without_Multicast > > < > https://deref-gmx.net/mail/client/Wiot_0qYjM4/dereferrer/?redirectUrl=htt... > > > > > > I configured both servers like this: > > > > <subsystem xmlns="urn:jboss:domain:jgroups:4.0"> > > <channels default="ee"> > > <channel name="ee" stack="tcp"/> > > </channels> > > <stacks default="tcp"> > > <stack name="tcp"> > > <transfer type="TCP" socket-binding="jgroups-tcp"/> > > <protocol type="TCPPING"> > > <property > > name="initial_hosts">10.62.168.51[7600],10.62.168.52[7600]</property> > > <property > name="num_initial_members">2</property> > > <property name="port_range">0</property> > > <property name="timeout">2000</property> > > </protocol> > > <protocol type="MERGE3"/> > > <protocol type="FD_SOCK" > > socket-binding="jgroups-tcp-fd"/> > > <protocol type="FD"/> > > <protocol type="VERIFY_SUSPECT"/> > > <protocol type="pbcast.NAKACK2"/> > > <protocol type="UNICAST3"/> > > <protocol type="pbcast.STABLE"/> > > <protocol type="pbcast.GMS"/> > > <protocol type="MFC"/> > > <protocol type="FRAG2"/> > > </stack> > > </stacks> > > </subsystem> > > > > I can't figure out whether the two nodes paired each other. In the > logfile > > I only find this: > > > > INFO 2017-07-21 20:35:43 [] > > org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000078: > > Starting JGroups channel hibernate > > INFO 2017-07-21 20:35:43 [] > > org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000078: > > Starting JGroups channel web > > INFO 2017-07-21 20:35:43 [] > > org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000078: > > Starting JGroups channel keycloak > > INFO 2017-07-21 20:35:43 [] > > org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000078: > > Starting JGroups channel ejb > > INFO 2017-07-21 20:35:43 [] > > org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000078: > > Starting JGroups channel server > > INFO 2017-07-21 20:35:43 [] > > org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000094: > > Received new cluster view for channel keycloak: [app02.2.xxx.net|0] > (1) [ > > app02.2.xxx.net] > > INFO 2017-07-21 20:35:43 [] > > org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000094: > > Received new cluster view for channel hibernate: [app02.2.xxx.net|0] > (1) [ > > app02.2.xxx.net] > > INFO 2017-07-21 20:35:43 [] > > org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000094: > > Received new cluster view for channel ejb: [app02.2.xxx.net|0] (1) [ > > app02.2.xxx.net] > > INFO 2017-07-21 20:35:43 [] > > org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000094: > > Received new cluster view for channel server: [app02.2.xxx.net|0] (1) [ > > app02.2.xxx.net] > > INFO 2017-07-21 20:35:43 [] > > org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000094: > > Received new cluster view for channel web: [app02.2.xxx.net|0] (1) [ > > app02.2.xxx.net] > > INFO 2017-07-21 20:35:43 [] > > org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000079: > > Channel server local address is app02.2.xxx.net, physical addresses > are [ > > 127.0.0.1:7600] > > INFO 2017-07-21 20:35:43 [] > > org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000079: > > Channel ejb local address is app02.2.xxx.net, physical addresses are [ > > 127.0.0.1:7600] > > INFO 2017-07-21 20:35:43 [] > > org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000079: > > Channel web local address is app02.2.xxx.net, physical addresses are [ > > 127.0.0.1:7600] > > INFO 2017-07-21 20:35:43 [] > > org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000079: > > Channel keycloak local address is app02.2.xxx.net, physical addresses > are [ > > 127.0.0.1:7600] > > INFO 2017-07-21 20:35:43 [] > > org.infinispan.remoting.transport.jgroups.JGroupsTransport ISPN000079: > > Channel hibernate local address is app02.2.xxx.net, physical addresses > are [ > > 127.0.0.1:7600 > > > > I checked that the ports 7060 are open. > > When I access just a single node, the setup works fine. With two nodes I > > get {"error":"invalid_grant","error_description":"Session not active"} > on > > one of them. > > The goal is that all userSessions are synced between the cluster nodes, > so > > that the loadbalancer can roundrobbin between all nodes. > > Since we have around 20k users only, it would be fine to store the > users in > > our database. But keycloak doesn't provide this, right? > > Do you have any idea how to figure out whats wrong? > > > > Best Regards, > > Christian. > > > > P.S.: You can find the whole configuration here: > > https://pastebin.com/WC46pXGp > > < > https://deref-gmx.net/mail/client/UIuwfhWwMZ0/dereferrer/?redirectUrl=htt... > > > > _______________________________________________ > > keycloak-user mailing list > > keycloak-user(a)lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/keycloak-user > > > _______________________________________________ > keycloak-user mailing list > keycloak-user(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-user >