Re: [keycloak-user] Does Keycloak need sticky session at the load balancer?

^[[0m^[[0m13:03:03,800 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000079: Channel ejb local address is keycloak01, physical addresses are [192.168.1.58:55200] ^[[0m^[[0m13:03:03,801 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000094: Received new cluster view for channel ejb: [keycloak02|1] (2) [keycloak02, keycloak01]

An internal server error has occurred

^[[0m^[[31m13:18:04,149 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-24) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: Replication timeout at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:639) ^[[0m^[[31m13:18:15,262 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (expiration-thread--p22-t1) ISPN000136: Error executing command RemoveExpiredCommand, writing keys [468d1940-7293-4824-9e86-4aece6cd6744]: org.infinispan.util.concurrent.TimeoutException: Replication timeout for keycloak02

<distributed-cache name="sessions" mode="SYNC" owners="2"/> <distributed-cache name="authenticationSessions" mode="SYNC" owners="2"/> <distributed-cache name="offlineSessions" mode="SYNC" owners="2"/> <distributed-cache name="clientSessions" mode="SYNC" owners="2"/> <distributed-cache name="offlineClientSessions" mode="SYNC" owners="2"/> <distributed-cache name="loginFailures" mode="SYNC" owners="2"/>

Ah no problem. It was my fault. I forgot to start debugging from the ground up (connectivity, firewalls, applications and so on ) On Wed, Aug 29, 2018 at 9:49 AM, Bela Ban <bban(a)redhat.com&gt; wrote: > Excellent! Unfortunately, JGroups cannot detect this... > > On 29/08/18 14:42, Rafael Weingärtner wrote: > >> Thanks! >> The problem was caused by firewalld blocking Multicast traffic. >> >> On Fri, Aug 24, 2018 at 7:28 AM, Sebastian Laskawiec < >> slaskawi(a)redhat.com <mailto:slaskawi@redhat.com>> wrote: >> >> Great write-up! Bookmarked! >> >> On Thu, Aug 23, 2018 at 4:36 PM Bela Ban <bban(a)redhat.com >> <mailto:bban@redhat.com>> wrote: >> >> Have you checked >> https://github.com/belaban/workshop/blob/master/slides/admin >> .adoc#problem-1-members-don-t-find-each-other >> <https://github.com/belaban/workshop/blob/master/slides/admi >> n.adoc#problem-1-members-don-t-find-each-other>? >> >> On 23/08/18 13:53, Sebastian Laskawiec wrote: >> > +Bela Ban <mailto:bban@redhat.com <mailto:bban@redhat.com>> >> > >> > As I expected, the cluster doesn't form. >> > >> > I'm not sure where and why those UDP discovery packets are >> rejected. I >> > just stumbled upon this thread [1], which you may find >> useful. Maybe >> > Bela will also have an idea what's going on there. >> > >> > If you won't manage to get UDP working, you can always fall >> back into >> > TCP (and MPING). >> > >> > [1] >> https://serverfault.com/questions/211482/tools-to-test-multi >> cast-routing >> <https://serverfault.com/questions/211482/tools-to-test-mult >> icast-routing> >> > >> > On Thu, Aug 23, 2018 at 1:26 PM Rafael Weingärtner >> > <rafaelweingartner(a)gmail.com >> <mailto:rafaelweingartner@gmail.com> >> <mailto:rafaelweingartner@gmail.com >> >> <mailto:rafaelweingartner@gmail.com>>> wrote: >> > >> > Thanks for the reply Sebastian! >> > >> > >> > Note, that IP Multicasting is disabled in many data >> centers (I >> > have never found out why they do it, but I've seen it >> many, many >> > times). So make sure your cluster forms correctly >> (just grep >> > logs and look for "view"). >> > >> > >> > I thought about that. Then, I used tcpdump, and I can see >> the >> > multicast packets from both Keycloak replicas. However, >> it seems >> > that these packets are being ignored. >> > >> > root@Keycloak01:/# tcpdump -i eth0 port 7600 or port >> 55200 or >> > port 45700 or port 45688 or port 23364 or port 4712 >> or port 4713 >> > tcpdump: verbose output suppressed, use -v or -vv for >> full >> > protocol decode >> > listening on eth0, link-type EN10MB (Ethernet), >> capture size >> > 262144 bytes >> > 11:13:36.540080 IP keycloak02.local.55200 > >> 230.0.0.4.45688: >> > UDP, length 83 >> > 11:13:41.288449 IP keycloak02.local.55200 > >> 230.0.0.4.45688: >> > UDP, length 83 >> > 11:13:46.342606 IP keycloak02.local.55200 > >> 230.0.0.4.45688: >> > UDP, length 83 >> > >> > >> > root@keycloak02:/# tcpdump -i eth0 port 7600 or port >> 55200 or >> > port 45700 or port 45688 or port 23364 or port 4712 >> or port 4713 >> > tcpdump: verbose output suppressed, use -v or -vv for >> full >> > protocol decode >> > listening on eth0, link-type EN10MB (Ethernet), >> capture size >> > 262144 bytes >> > 11:12:14.218317 IP Keycloak01.local.55200 > >> 230.0.0.4.45688: >> > UDP, length 83 >> > 11:12:23.146798 IP Keycloak01.local.55200 > >> 230.0.0.4.45688: >> > UDP, length 83 >> > 11:12:27.201888 IP Keycloak01.local.55200 > >> 230.0.0.4.45688: >> > UDP, length 83 >> > >> > >> > >> > Here go the log entries. I filtered by “view”. This is >> from Keycloak01. >> > >> > ^[[0m^[[0m11:16:57,896 INFO >> > [org.infinispan.remoting.tran >> sport.jgroups.JGroupsTransport] >> > (MSC service thread 1-4) ISPN000094: Received new >> cluster view >> > for channel ejb: [keycloak01|0] (1) [keycloak01] >> > ^[[0m^[[0m11:16:57,896 INFO >> > [org.infinispan.remoting.tran >> sport.jgroups.JGroupsTransport] >> > (MSC service thread 1-2) ISPN000094: Received new >> cluster view >> > for channel ejb: [keycloak01|0] (1) [keycloak01] >> > ^[[0m^[[0m11:16:57,897 INFO >> > [org.infinispan.remoting.tran >> sport.jgroups.JGroupsTransport] >> > (MSC service thread 1-1) ISPN000094: Received new >> cluster view >> > for channel ejb: [keycloak01|0] (1) [keycloak01] >> > ^[[0m^[[0m11:16:57,898 INFO >> > [org.infinispan.remoting.tran >> sport.jgroups.JGroupsTransport] >> > (MSC service thread 1-3) ISPN000094: Received new >> cluster view >> > for channel ejb: [keycloak01|0] (1) [keycloak01] >> > ^[[0m^[[0m11:16:57,962 INFO >> > [org.infinispan.remoting.tran >> sport.jgroups.JGroupsTransport] >> > (MSC service thread 1-1) ISPN000094: Received new >> cluster view >> > for channel ejb: [keycloak01|0] (1) [keycloak01] >> > >> > >> > I expected it to be only one. I mean, I first started >> Keycloak01, >> > and just then Keycloak02. Next, we have the logs from >> Keycloak02. >> > >> > ^[[0m^[[0m11:17:34,950 INFO >> > [org.infinispan.remoting.tran >> sport.jgroups.JGroupsTransport] >> > (MSC service thread 1-3) ISPN000094: Received new >> cluster view >> > for channel ejb: [keycloak02|0] (1) [keycloak02] >> > ^[[0m^[[0m11:17:34,952 INFO >> > [org.infinispan.remoting.tran >> sport.jgroups.JGroupsTransport] >> > (MSC service thread 1-4) ISPN000094: Received new >> cluster view >> > for channel ejb: [keycloak02|0] (1) [keycloak02] >> > ^[[0m^[[0m11:17:34,957 INFO >> > [org.infinispan.remoting.tran >> sport.jgroups.JGroupsTransport] >> > (MSC service thread 1-1) ISPN000094: Received new >> cluster view >> > for channel ejb: [keycloak02|0] (1) [keycloak02] >> > ^[[0m^[[0m11:17:34,957 INFO >> > [org.infinispan.remoting.tran >> sport.jgroups.JGroupsTransport] >> > (MSC service thread 1-2) ISPN000094: Received new >> cluster view >> > for channel ejb: [keycloak02|0] (1) [keycloak02] >> > ^[[0m^[[0m11:17:35,052 INFO >> > [org.infinispan.remoting.tran >> sport.jgroups.JGroupsTransport] >> > (MSC service thread 1-1) ISPN000094: Received new >> cluster view >> > for channel ejb: [keycloak02|0] (1) [keycloak02 >> > >> > >> > They are similar. It seems that both applications are not >> seeing >> > each other. At first, I thought that the problem was >> caused by >> > “owners=1” configuration (the lack of data >> synchronization between >> > replicas). I then changed it to “owners=2”, but still, if >> I log in >> > the Keycloak01 and then force my request to go two >> Keycloak02, my >> > session is not there, and I am requested to log in again. >> > >> > Do you need some other log entries or configuration files? >> > >> > Again, thanks for your reply and help! >> > >> > On Thu, Aug 23, 2018 at 5:24 AM, Sebastian Laskawiec >> > <slaskawi(a)redhat.com <mailto:slaskawi@redhat.com> >> <mailto:slaskawi@redhat.com <mailto:slaskawi@redhat.com>>> >> wrote: >> > >> > >> > >> > On Wed, Aug 22, 2018 at 10:24 PM Rafael Weingärtner >> > <rafaelweingartner(a)gmail.com >> <mailto:rafaelweingartner@gmail.com> >> > <mailto:rafaelweingartner@gmail.com >> >> <mailto:rafaelweingartner@gmail.com>>> wrote: >> > >> > Hello Keycloakers, >> > >> > I have some doubts regarding Keycloak and load >> balancers. I >> > set up two >> > keycloak replicas to provide HA. To start them I >> am using >> > “./standalone.sh >> > --server-config=standalone-ha.xml”. I am >> assuming that they >> > will use >> > multicast to replicate information between nodes, >> right? >> > >> > >> > That is correct. It uses PING protocol, which in turn >> uses IP >> > Multicasting for discovery. >> > >> > Note, that IP Multicasting is disabled in many data >> centers (I >> > have never found out why they do it, but I've seen it >> many, many >> > times). So make sure your cluster forms correctly >> (just grep >> > logs and look for "view"). >> > >> > Then, I set up a load balancer layer using Apache >> HTTPD and >> > AJP connector >> > via 8009 port. To make everything work I needed >> to use >> > sticky session; >> > otherwise, the login would never happen. I am >> fine with the >> > sticky session, >> > however, if I stop one of the replicas where the >> user is >> > logged in, when >> > the user access Keycloak again, he/she is asked >> to present >> > the credentials >> > as if he/she was not logged in the other Keycloak >> replica. >> > Is that the >> > expected behavior? >> > >> > >> > My intuition tells me that your cluster didn't form >> correctly >> > (as I mentioned before, grep the logs and look for >> "view" >> > generated by JGroups). Therefore, if you enable >> sticky session, >> > all your requests get to the same Keycloak instance, >> which has >> > everything in the local cache. That's why it works >> fine. >> > >> > >> > Is there some troubleshooting or test that I can >> perform to >> > check if >> > replication is being executed? >> > >> > >> > Let's start with investigating the logs. Later on we >> can check JMX. >> > >> > >> > -- >> > Rafael Weingärtner >> > _______________________________________________ >> > keycloak-user mailing list >> > keycloak-user(a)lists.jboss.org >> <mailto:keycloak-user@lists.jboss.org> >> > <mailto:keycloak-user@lists.jboss.org >> <mailto:keycloak-user@lists.jboss.org>> >> > https://lists.jboss.org/mailman/listinfo/keycloak-user >> <https://lists.jboss.org/mailman/listinfo/keycloak-user> >> > >> > >> > >> > >> > -- >> > Rafael Weingärtner >> > >> >> -- Bela Ban | http://www.jgroups.org >> >> >> >> >> -- >> Rafael Weingärtner >> > > -- > Bela Ban | http://www.jgroups.org > > -- Rafael Weingärtner