Great write-up! Bookmarked! On Thu, Aug 23, 2018 at 4:36 PM Bela Ban <bban(a)redhat.com&gt; wrote: > Have you checked > https://github.com/belaban/workshop/blob/master/slides/ > admin.adoc#problem-1-members-don-t-find-each-other? > > On 23/08/18 13:53, Sebastian Laskawiec wrote: > > +Bela Ban <mailto:bban@redhat.com> > > > > As I expected, the cluster doesn't form. > > > > I'm not sure where and why those UDP discovery packets are rejected. I > > just stumbled upon this thread [1], which you may find useful. Maybe > > Bela will also have an idea what's going on there. > > > > If you won't manage to get UDP working, you can always fall back into > > TCP (and MPING). > > > > [1] https://serverfault.com/questions/211482/tools-to- > test-multicast-routing > > > > On Thu, Aug 23, 2018 at 1:26 PM Rafael Weingärtner > > <rafaelweingartner(a)gmail.com <mailto:rafaelweingartner@gmail.com>> > wrote: > > > > Thanks for the reply Sebastian! > > > > > > Note, that IP Multicasting is disabled in many data centers (I > > have never found out why they do it, but I've seen it many, many > > times). So make sure your cluster forms correctly (just grep > > logs and look for "view"). > > > > > > I thought about that. Then, I used tcpdump, and I can see the > > multicast packets from both Keycloak replicas. However, it seems > > that these packets are being ignored. > > > > root@Keycloak01:/# tcpdump -i eth0 port 7600 or port 55200 or > > port 45700 or port 45688 or port 23364 or port 4712 or port 4713 > > tcpdump: verbose output suppressed, use -v or -vv for full > > protocol decode > > listening on eth0, link-type EN10MB (Ethernet), capture size > > 262144 bytes > > 11:13:36.540080 IP keycloak02.local.55200 > 230.0.0.4.45688: > > UDP, length 83 > > 11:13:41.288449 IP keycloak02.local.55200 > 230.0.0.4.45688: > > UDP, length 83 > > 11:13:46.342606 IP keycloak02.local.55200 > 230.0.0.4.45688: > > UDP, length 83 > > > > > > root@keycloak02:/# tcpdump -i eth0 port 7600 or port 55200 or > > port 45700 or port 45688 or port 23364 or port 4712 or port 4713 > > tcpdump: verbose output suppressed, use -v or -vv for full > > protocol decode > > listening on eth0, link-type EN10MB (Ethernet), capture size > > 262144 bytes > > 11:12:14.218317 IP Keycloak01.local.55200 > 230.0.0.4.45688: > > UDP, length 83 > > 11:12:23.146798 IP Keycloak01.local.55200 > 230.0.0.4.45688: > > UDP, length 83 > > 11:12:27.201888 IP Keycloak01.local.55200 > 230.0.0.4.45688: > > UDP, length 83 > > > > > > > > Here go the log entries. I filtered by “view”. This is from > Keycloak01. > > > > ^[[0m^[[0m11:16:57,896 INFO > > [org.infinispan.remoting.transport.jgroups.JGroupsTransport] > > (MSC service thread 1-4) ISPN000094: Received new cluster view > > for channel ejb: [keycloak01|0] (1) [keycloak01] > > ^[[0m^[[0m11:16:57,896 INFO > > [org.infinispan.remoting.transport.jgroups.JGroupsTransport] > > (MSC service thread 1-2) ISPN000094: Received new cluster view > > for channel ejb: [keycloak01|0] (1) [keycloak01] > > ^[[0m^[[0m11:16:57,897 INFO > > [org.infinispan.remoting.transport.jgroups.JGroupsTransport] > > (MSC service thread 1-1) ISPN000094: Received new cluster view > > for channel ejb: [keycloak01|0] (1) [keycloak01] > > ^[[0m^[[0m11:16:57,898 INFO > > [org.infinispan.remoting.transport.jgroups.JGroupsTransport] > > (MSC service thread 1-3) ISPN000094: Received new cluster view > > for channel ejb: [keycloak01|0] (1) [keycloak01] > > ^[[0m^[[0m11:16:57,962 INFO > > [org.infinispan.remoting.transport.jgroups.JGroupsTransport] > > (MSC service thread 1-1) ISPN000094: Received new cluster view > > for channel ejb: [keycloak01|0] (1) [keycloak01] > > > > > > I expected it to be only one. I mean, I first started Keycloak01, > > and just then Keycloak02. Next, we have the logs from Keycloak02. > > > > ^[[0m^[[0m11:17:34,950 INFO > > [org.infinispan.remoting.transport.jgroups.JGroupsTransport] > > (MSC service thread 1-3) ISPN000094: Received new cluster view > > for channel ejb: [keycloak02|0] (1) [keycloak02] > > ^[[0m^[[0m11:17:34,952 INFO > > [org.infinispan.remoting.transport.jgroups.JGroupsTransport] > > (MSC service thread 1-4) ISPN000094: Received new cluster view > > for channel ejb: [keycloak02|0] (1) [keycloak02] > > ^[[0m^[[0m11:17:34,957 INFO > > [org.infinispan.remoting.transport.jgroups.JGroupsTransport] > > (MSC service thread 1-1) ISPN000094: Received new cluster view > > for channel ejb: [keycloak02|0] (1) [keycloak02] > > ^[[0m^[[0m11:17:34,957 INFO > > [org.infinispan.remoting.transport.jgroups.JGroupsTransport] > > (MSC service thread 1-2) ISPN000094: Received new cluster view > > for channel ejb: [keycloak02|0] (1) [keycloak02] > > ^[[0m^[[0m11:17:35,052 INFO > > [org.infinispan.remoting.transport.jgroups.JGroupsTransport] > > (MSC service thread 1-1) ISPN000094: Received new cluster view > > for channel ejb: [keycloak02|0] (1) [keycloak02 > > > > > > They are similar. It seems that both applications are not seeing > > each other. At first, I thought that the problem was caused by > > “owners=1” configuration (the lack of data synchronization between > > replicas). I then changed it to “owners=2”, but still, if I log in > > the Keycloak01 and then force my request to go two Keycloak02, my > > session is not there, and I am requested to log in again. > > > > Do you need some other log entries or configuration files? > > > > Again, thanks for your reply and help! > > > > On Thu, Aug 23, 2018 at 5:24 AM, Sebastian Laskawiec > > <slaskawi(a)redhat.com <mailto:slaskawi@redhat.com>> wrote: > > > > > > > > On Wed, Aug 22, 2018 at 10:24 PM Rafael Weingärtner > > <rafaelweingartner(a)gmail.com > > <mailto:rafaelweingartner@gmail.com>> wrote: > > > > Hello Keycloakers, > > > > I have some doubts regarding Keycloak and load balancers. I > > set up two > > keycloak replicas to provide HA. To start them I am using > > “./standalone.sh > > --server-config=standalone-ha.xml”. I am assuming that > they > > will use > > multicast to replicate information between nodes, right? > > > > > > That is correct. It uses PING protocol, which in turn uses IP > > Multicasting for discovery. > > > > Note, that IP Multicasting is disabled in many data centers (I > > have never found out why they do it, but I've seen it many, many > > times). So make sure your cluster forms correctly (just grep > > logs and look for "view"). > > > > Then, I set up a load balancer layer using Apache HTTPD and > > AJP connector > > via 8009 port. To make everything work I needed to use > > sticky session; > > otherwise, the login would never happen. I am fine with the > > sticky session, > > however, if I stop one of the replicas where the user is > > logged in, when > > the user access Keycloak again, he/she is asked to present > > the credentials > > as if he/she was not logged in the other Keycloak replica. > > Is that the > > expected behavior? > > > > > > My intuition tells me that your cluster didn't form correctly > > (as I mentioned before, grep the logs and look for "view" > > generated by JGroups). Therefore, if you enable sticky session, > > all your requests get to the same Keycloak instance, which has > > everything in the local cache. That's why it works fine. > > > > > > Is there some troubleshooting or test that I can perform to > > check if > > replication is being executed? > > > > > > Let's start with investigating the logs. Later on we can check > JMX. > > > > > > -- > > Rafael Weingärtner > > _______________________________________________ > > keycloak-user mailing list > > keycloak-user(a)lists.jboss.org > > <mailto:keycloak-user@lists.jboss.org> > > https://lists.jboss.org/mailman/listinfo/keycloak-user > > > > > > > > > > -- > > Rafael Weingärtner > > > > -- > Bela Ban | http://www.jgroups.org > >