Re: [keycloak-user] Cross-DC Replication not working for `sessions` cache

Hi, I didn't check everything, but one thing I noted is, that in your keycloak-standalone-ha.xml, you don't have "alternative" providers configured. For Keycloak to work with the infinispan 9.2.X server or newer, it was needed to configure providers like this: https://github.com/rhdemo/rh-sso/blob/master/standalone-openshift-cfg/con... . There is also a need to add the userStorage to your realm, which can be done through admin console or by importing the realm. See: https://github.com/rhdemo/rh-sso/blob/master/realm-summit.json#L1051 Marek On 08/08/18 15:07, Sebastian Laskawiec wrote: > On Tue, Aug 7, 2018 at 3:28 PM Hayden Fuss <hfuss(a)bandwidth.com&gt; wrote: > >> Hello, >> >> We are attempting to run Keycloak on two OpenShift clusters using remote >> ISPNs and a single MariaDB instance. We're hacking together the Keycloak on >> Openshift blogpost, the JDG-as-a-service demo from Summit, RH SSO demo from >> Summit, and following the Keycloak/RH SSO basic setup guide to Cross-DC >> replication. The hope is do an initial evaluation of Keycloak's >> availability. >> >> We were able to create a new user on master (site1), disable the user on >> master2 (site2), and see the user was disabled on master. So ISPN >> replication seems to be working because the work cache was replicated to >> invalidate the local caches. However, the sessions cache does not seem to >> be replicated because when logged in as the same user on the two different >> Keycloaks (in Incognito mode) there is only one active session shown on >> both UIs and the timestamp/IP/etc is different for the listed session. >> > So at this point the Infinispan cluster within a single DC works correctly > [1] (the one that is formed by KUBE_PING). The Cross-DC cluster (also known > as the Global Cluster) also works correctly [2]. Users cache replicates > fine but sessions don't. > > If I understood everything correctly, there might be two issues there. > > The first one is Infinispan misconfiguration (I briefly looked through the > configuration and can not spot any mistake but there might be some typo or > anything like that). That one is easy to be verified, just put an entry on > one node (e.g. using REST [3]) and see if it's available on the other one > (again, using REST for example [4]). > > If this test works fine, you can check if Keycloak forwards traffic to the > Infinispan cluster. The easiest way is to set a breakpoint somewhere > in org.keycloak.models.sessions.infinispan.changes.sessions.LastSessionRefreshChecker#shouldSaveClientSessionToRemoteCache > and org.keycloak.models.sessions.infinispan.changes.sessions.LastSessionRefreshChecker#shouldSaveUserSessionToRemoteCache. > > [1] can be verified by calling `oc logs infinispan-app | grep view` > [2] can be verified by calling `oc logs infinispan-app | grep "x-site"` > [3] curl -d test ISPN_IP:8080/rest/sessions/test > [4] curl ISPN_IP2:8080/rest/sessions/test > > >> We are using the latest, stable Keycloak image, version 4.1.0.Final, and >> the latest, stable Infinispan image for to act as our data grid, version >> 9.3.1.Final, which we know differs from the 8.2.8 version Keycloak uses for >> its local caches. >> >> We were trying one Keycloak node and two ISPN nodes in each cluster, but >> for simplicity we've attached logs where we only ran one Keycloak and one >> ISPN in each cluster. >> We were connecting to the two different Keycloaks via two different >> OpenShift Routes without a load balancer to fake sticky sessions for now. >> Keycloak connects to ISPN via a "HotRod" Service. ISPN connects to other >> nodes within the same cluster via KUBE_PING, and discovers the other >> cluster via TCPPING hitting a particular OpenShift app node from that >> cluster that exposes the "discovery" Service with a NodePort. The Keycloaks >> share the single MariaDB through a NodePort Service in one of the clusters >> as well. >> >> The logs didn't seem to contain any of the messages in the trouble shooting >> guide. We had trouble using JMX to check the ISPNs because they were >> running in containers, but we've using the CLI tool and the Infinispan >> management console to try to troubleshoot but any key we pulled from the >> logs that we thought was a session ID was not in the caches and we could >> not find a way to simply list all keys in the caches. >> >> Below is a viewable link to a zip containing logs from the scenario >> described in the second paragraph, and our config files. >> >> >> https://drive.google.com/open?id=0B_OCdNCEtoCYOU12T3dEUFplS193VFNFbEFYclB... >> >> Thanks for your time and help! >> >> Best, >> Hayden >> _______________________________________________ >> keycloak-user mailing list >> keycloak-user(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/keycloak-user >> > _______________________________________________ > keycloak-user mailing list > keycloak-user(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-user