Re: [keycloak-user] Galera Replication and Caching

On 17/10/16 20:54, Jared Blashka wrote: Both of our keycloak nodes are living in the same physical datacenter+networking space and are the only two nodes in an infinispan cluster; they're just each using a different Galera DB (and these are clustered synchronously together along with a 3rd Galera node). We were trying to validate that DB replication wouldn't break like it did for a similar configuration we were using earlier (using asynchronous DB replication). So the DB replication isn't breaking and appears to be functioning as expected, but it looks like there's data cached by each Keycloak node that doesn't get refreshed from the DB nor corrected by infinispan. So far the only thing we've noticed are changes not appearing in the Admin UI e.g. Realm/Client changes performed on Keycloak01 don't appear in the UI for Keycloak02 but *do* appear in Galera02. The issue doesn't seem to extend to client sessions; we haven't heard any issues of people being asked to log in multiple times. I'd be happy to run any specific tests in our set up if you want additional info. Could you try this simple test like: - Create user in admin console on keycloak node1 - Verify that user is visible on keycloak node2 - Then update this user on node2 (For example change his firstName) - Go back to node1 and see if firstName was changed This is the similar test, which I've tried with 2 keycloak cluster nodes configured against 2 MariaDB Galera cluster nodes and it worked fine for me. The automated test is here if you want to take a look : https://github.com/mposolda/keycloak-mariadb/blob/master/ mariadb-cluster-test/src/test/java/org/keycloak/test/UsersClusterTest.java . If this scenario works fine for you, then it's maybe just listing clients, which is somehow broken. Then it's high probability that I will reproduce in my environment too. Otherwise if user's scenario is broken for you as well, then it's probably something related to your environment setup though... Marek Jared On Mon, Oct 17, 2016 at 2:34 PM, Stian Thorgersen <sthorger(a)redhat.com&gt; wrote: > Just to point out the maybe not so obvious - all realm configuration > including clients are cached in an Infinispan invalidation cache. I've got > no idea how to setup the Infinispan invalidation caches cross data centers, > but that would be required for entries to be re-loaded in one DC when > updated in another DC. > > On 13 October 2016 at 17:08, Marek Posolda <mposolda(a)redhat.com&gt; wrote: > >> And are also both Keycloak nodes in the same infinispan cluster? >> >> Marek >> >> Dne 12.10.2016 v 23:27 Jared Blashka napsal(a): >> > We've got synchronous replication enabled. I've looked in the DB >> > tables for both galera nodes and the data is there. e.g. both DB nodes >> > have client 'myclient' but the UI for Keycloak node 2 doesn't list a >> > 'myclient'. But Keycloak will error if you try to add 'myclient' >> > saying it already exists. >> > >> > On Wed, Oct 12, 2016 at 4:42 PM, Marek Posolda <mposolda(a)redhat.com >> > <mailto:mposolda@redhat.com>> wrote: >> > >> > Then it's probably related to the Galera cluster rather then to >> > caching... >> > >> > Do you have DB configured with synchronous replication (eg. >> > inserting some record on DB1 is successfully finished after the >> > record is successfully replicated to DB2 too) ? >> > >> > You can maybe compare with the configuration in my docker image >> > https://github.com/mposolda/keycloak-mariadb >> > <https://github.com/mposolda/keycloak-mariadb> . I can't recall to >> > see any issue like this, but not sure about other aspects of my >> > configuration (performance etc). >> > >> > Marek >> > >> > >> > On 12/10/16 19:08, Jared Blashka wrote: >> >> We're already running 1.9.8.Final. Our previous configuration was >> >> using 2 clustered nodes configured against the same DB node and >> >> we didn't run into this issue. >> >> >> >> On Wed, Oct 12, 2016 at 2:45 AM, Marek Posolda >> >> <mposolda(a)redhat.com <mailto:mposolda@redhat.com>> wrote: >> >> >> >> Which Keycloak version are you using? If it's older than >> >> 1.9.8.Final, >> >> then it's suggested to upgrade as there were caching fixes >> >> meanwhile. >> >> >> >> There is also possibility to disable caching in >> >> keycloak-server.json (or >> >> in standalone.xml in latest version). It's mentioned in the >> >> docs how to >> >> do it. >> >> >> >> Finally it may also help if you have opportunity to try with >> >> 2 Keycloak >> >> cluster nodes configured against same DB node. This may help >> >> to better >> >> isolate the problem and see if it's related to caching or to >> >> MariaDB >> >> cluster. >> >> >> >> Marek >> >> >> >> On 10/10/16 22:31, Josh Cain wrote: >> >> > Hi all, >> >> > >> >> > We're running into a problem with a couple of MariaDB >> >> instances + >> >> > Galera. When I go to add a client on the first Keycloak >> >> node/DB (we'll >> >> > call it DB01), it add successfully. I can then go to the >> >> second >> >> > Keycloak Node/DB (call this one DB02) and do not see the >> >> client on the >> >> > 'clients' list. However, if I were to add the same client >> >> on DB02, I >> >> > get the expected 'client with ID already exists' message. >> >> What's more, >> >> > if I bounce the Keycloak node that talks to DB02, the >> >> client list >> >> > populates with the new entry added at DB01. >> >> > >> >> > Was guessing it's some kind of caching issue - is there a >> >> setting where >> >> > I can alter this behavior? >> >> > >> >> >> >> _______________________________________________ >> >> keycloak-user mailing list >> >> keycloak-user(a)lists.jboss.org >> >> <mailto:keycloak-user@lists.jboss.org> >> >> https://lists.jboss.org/mailman/listinfo/keycloak-user >> >> <https://lists.jboss.org/mailman/listinfo/keycloak-user> >> >> >> >> >> > >> > >> >> _______________________________________________ >> keycloak-user mailing list >> keycloak-user(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/keycloak-user >> > >