[keycloak-user] Cross-DC Replication not working for `sessions` cache

[Marek Posolda]

Hi,

I didn't check everything, but one thing I noted is, that in your 
keycloak-standalone-ha.xml, you don't have "alternative" providers 
configured.

For Keycloak to work with the infinispan 9.2.X server or newer, it was 
needed to configure providers like this: 
https://github.com/rhdemo/rh-sso/blob/master/standalone-openshift-cfg/configuration/standalone-openshift-jdg.xml#L676-L681 
.

There is also a need to add the userStorage to your realm, which can be 
done through admin console or by importing the realm. See: 
https://github.com/rhdemo/rh-sso/blob/master/realm-summit.json#L1051

Marek


On 08/08/18 15:07, Sebastian Laskawiec wrote:
> On Tue, Aug 7, 2018 at 3:28 PM Hayden Fuss <hfuss at bandwidth.com> wrote:
>
>> Hello,
>>
>> We are attempting to run Keycloak on two OpenShift clusters using remote
>> ISPNs and a single MariaDB instance. We're hacking together the Keycloak on
>> Openshift blogpost, the JDG-as-a-service demo from Summit, RH SSO demo from
>> Summit, and following the Keycloak/RH SSO basic setup guide to Cross-DC
>> replication. The hope is do an initial evaluation of Keycloak's
>> availability.
>>
>> We were able to create a new user on master (site1), disable the user on
>> master2 (site2), and see the user was disabled on master. So ISPN
>> replication seems to be working because the work cache was replicated to
>> invalidate the local caches. However, the sessions cache does not seem to
>> be replicated because when logged in as the same user on the two different
>> Keycloaks (in Incognito mode) there is only one active session shown on
>> both UIs and the timestamp/IP/etc is different for the listed session.
>>
> So at this point the Infinispan cluster within a single DC works correctly
> [1] (the one that is formed by KUBE_PING). The Cross-DC cluster (also known
> as the Global Cluster) also works correctly [2]. Users cache replicates
> fine but sessions don't.
>
> If I understood everything correctly, there might be two issues there.
>
> The first one is Infinispan misconfiguration (I briefly looked through the
> configuration and can not spot any mistake but there might be some typo or
> anything like that). That one is easy to be verified, just put an entry on
> one node (e.g. using REST [3]) and see if it's available on the other one
> (again, using REST for example [4]).
>
> If this test works fine, you can check if Keycloak forwards traffic to the
> Infinispan cluster. The easiest way is to set a breakpoint somewhere
> in org.keycloak.models.sessions.infinispan.changes.sessions.LastSessionRefreshChecker#shouldSaveClientSessionToRemoteCache
> and org.keycloak.models.sessions.infinispan.changes.sessions.LastSessionRefreshChecker#shouldSaveUserSessionToRemoteCache.
>
> [1] can be verified by calling `oc logs infinispan-app | grep view`
> [2] can be verified by calling `oc logs infinispan-app | grep "x-site"`
> [3] curl -d test ISPN_IP:8080/rest/sessions/test
> [4] curl ISPN_IP2:8080/rest/sessions/test
>
>
>> We are using the latest, stable Keycloak image, version 4.1.0.Final, and
>> the latest, stable Infinispan image for to act as our data grid, version
>> 9.3.1.Final, which we know differs from the 8.2.8 version Keycloak uses for
>> its local caches.
>>
>> We were trying one Keycloak node and two ISPN nodes in each cluster, but
>> for simplicity we've attached logs where we only ran one Keycloak and one
>> ISPN in each cluster.
>> We were connecting to the two different Keycloaks via two different
>> OpenShift Routes without a load balancer to fake sticky sessions for now.
>> Keycloak connects to ISPN via a "HotRod" Service. ISPN connects to other
>> nodes within the same cluster via KUBE_PING, and discovers the other
>> cluster via TCPPING hitting a particular OpenShift app node from that
>> cluster that exposes the "discovery" Service with a NodePort. The Keycloaks
>> share the single MariaDB through a NodePort Service in one of the clusters
>> as well.
>>
>> The logs didn't seem to contain any of the messages in the trouble shooting
>> guide. We had trouble using JMX to check the ISPNs because they were
>> running in containers, but we've using the CLI tool and the Infinispan
>> management console to try to troubleshoot but any key we pulled from the
>> logs that we thought was a session ID was not in the caches and we could
>> not find a way to simply list all keys in the caches.
>>
>> Below is a viewable link to a zip containing logs from the scenario
>> described in the second paragraph, and our config files.
>>
>>
>> https://drive.google.com/open?id=0B_OCdNCEtoCYOU12T3dEUFplS193VFNFbEFYclB4Tm5WR0o4
>>
>> Thanks for your time and help!
>>
>> Best,
>> Hayden
>> _______________________________________________
>> keycloak-user mailing list
>> keycloak-user at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/keycloak-user
>>
> _______________________________________________
> keycloak-user mailing list
> keycloak-user at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user