Hi list, I'm trying to cluster several Keycloak 4.0.0 nodes running in docker containers. I'm seeing "interesting" (unexpected) behaviour when requesting new OIDC tokens with "grant_type=password" (Direct Grant/ROPC) and then attempting to get a new set with "grant_type=refresh_token" . After I start two nodes (containers), if I issue a "grant_type=password" request to the node that started first, requests for "grant_type=refresh_token" on the newer node fail. If I issue the "grant_type=password" request to the node that started last, requests for "grant_type=refresh_token" succeed on any node AND all future password/refresh_token requests work correctly no matter which node handles the request. So, let's say node1 starts first and node2 starts second: 1. Password auth on node1: OK 2. Refresh token auth on node2 with token from previous step: Error: invalid_grant (Invalid refresh token) 3. Refresh token auth on node1 with token from step 1: OK (new set of refresh+access tokens) BUT! 4. Password auth on node2: OK 5. Refresh token auth on node1 with token from previous step: OK! (new set of refresh+access tokens) 6. Refresh token auth on node2 with token from step 4: OK 7. Password auth sequence from steps 1-3: also OK! It's as though the node that starts most recently needs a password auth request to "wake up" and start communicating with the rest of the cluster. Once it does, everything's in sync. Some facts that are hopefully relevant: * Keycloak 4.0.0 docker image base * standalone-ha.xml from distribution with changes in JGroups subsystem. I'm using JDBC PING configured for the same DB as Keycloak itself, which is MySQL 5.7. See the subsystem config below. * Custom org.keycloak.storage.UserStorageProviderFactory SPI, which creates a provider that makes an HTTP call to an external authentication service to validate username/password credentials. * A couple of custom themes. * One realm with a handful of clients provisioned via a shell script that just calls kcadm.sh and jboss-cli.sh * There's a simple LB in front of both instances JGroups subsystem config: <subsystem xmlns="urn:jboss:domain:jgroups:5.0"> <channels default="ee"> <channel name="ee" stack="tcp" cluster="ejb"/> </channels> <stacks> <stack name="tcp"> <transport type="TCP" socket-binding="jgroups-tcp"> <property name="external_addr">${env.HOST}</property> <property name="external_port">${env.PORT_7600}</property> </transport> <protocol type="org.jgroups.protocols.JDBC_PING"> <property name="datasource_jndi_name"> java:jboss/datasources/KeycloakDS </property> </protocol> <protocol type="MERGE3"/> <protocol type="FD_SOCK"/> <protocol type="FD_ALL"/> <protocol type="VERIFY_SUSPECT"/> <protocol type="pbcast.NAKACK2"/> <protocol type="UNICAST3"/> <protocol type="pbcast.STABLE"/> <protocol type="pbcast.GMS"/> <protocol type="MFC"/> <protocol type="FRAG2"/> </stack> </stacks> </subsystem> $HOST and $PORT_7600 are set to external host:port combination that allows the two instances to communicate. There's also a socket-binding to a public interface: <socket-binding name="jgroups-tcp" port="7600"/> In the JGroups and Infinispan log entries I can see the two nodes do find each other and are able to communicate. I haven't been able to get ispn-cli.sh to connect to the internal Infinispan instances running in containers, so I can't confirm that they have the same entries, but as described in flows above they do eventually work together. Is there a configuration change I'm missing somewhere to make the new node joining the cluster become aware of the other one? Thanks for any help, DV _______________________________________________ keycloak-user mailing list keycloak-user(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/keycloak-user

Weird indeed. Yes, the logs indicate two nodes. I've uploaded the full start-up logs here: https://drive.google.com/drive/folders/1AiyLtTXu2AxEbVBdR-5kfJLxqoYladBn?... . I started node 1, let it settle, then started node 2. You can see that node1 starts with just itself, but later node2 joins the cluster and caches are rebalanced. As for the experiment, I tried waiting for a few minutes after both nodes started in case there's some synchronization delay somewhere, but it didn't change the outcome. Thanks, DV On Wed, Sep 12, 2018 at 3:22 AM Sebastian Laskawiec <slaskawi(a)redhat.com&gt; wrote: > Hmmm this sounds a bit weird... like there was some delay in the > communication path. > > Could you please look through your logs and look for lines including > "view" keyword? Are there two nodes, as expected? How the timestamps relate > to your experiment? >

Hmm ... maybe the lb is pinging the port? I'm running dockercloud/haproxy, which autodetects open ports. However, I'm excluding port 7600 so that it doesn't try to route application requests to JGroups ports. The SocketTimeoutException only happens once at startup, though. I don't see it later when I start running auth tests. Thanks for the pointer to Infinispan statistics query. I ran it for both nodes and saved the results in the "ispn-cluster-query" subdirectory in (previous shared) https://drive.google.com/drive/folders/1AiyLtTXu2AxEbVBdR-5kfJLxqoYladBn?... : - "start" prefix is for the output of the query right after starting the nodes. Node1 starts first, then node2. - "first-auth" is the initial grant_type=password auth. In this set of tests it was done on node1. - "refresh-auth" is the subsequent failing grant_type=refresh_token . It's successful on node1 and failing on node2. - "post-node2-auth" is after grant_type=password auth is executed on node2 (which brings the cluster in-sync). I couldn't spot any issues in the output with my untrained eyes. I wonder, should the statistics be pulled from the sessions distributed cache as well? Is that the one that would be consulted during grant_type=refresh_token auth? Thanks, DV On Mon, Sep 17, 2018 at 4:27 AM Sebastian Laskawiec <slaskawi(a)redhat.com&gt; wrote: > So the only thing that look suspicious is this: > JGRP000006: failed accepting connection from peer: > java.net.SocketTimeoutException: Read timed out > > It might indicate that some other application tried to connect to > Keycloak on port 7600 and immediately disconnected. That leads to a > question on your environment, are you sure you are looking into proper > applications servers? Perhaps some other applications (Wildfly for example, > since Keycloak is built on Wildfly) are trying to join the cluster. > > However, if the answer is yes, the next thing to check are Infinispan > statistics over JMX or JBoss CLI. Here's a sample query you may use: > /subsystem=infinispan/cache-container=keycloak/replicated-cache=*:query > And then have a look at number of entries and number of entries in the > cluster. > > @Marek Posolda <mposolda(a)redhat.com&gt;. perhaps this ring you any bell? > ISPN seems fine here (at least from the logs and symptoms DV's describing. > > On Thu, Sep 13, 2018 at 6:53 PM D V <dv(a)glyphy.com&gt; wrote: > >> Weird indeed. Yes, the logs indicate two nodes. I've uploaded the full >> start-up logs here: >> https://drive.google.com/drive/folders/1AiyLtTXu2AxEbVBdR-5kfJLxqoYladBn?... >> . I started node 1, let it settle, then started node 2. You can see that >> node1 starts with just itself, but later node2 joins the cluster and caches >> are rebalanced. >> >> As for the experiment, I tried waiting for a few minutes after both >> nodes started in case there's some synchronization delay somewhere, but it >> didn't change the outcome. >> >> Thanks, >> DV >> >> On Wed, Sep 12, 2018 at 3:22 AM Sebastian Laskawiec <slaskawi(a)redhat.com&gt; >> wrote: >> >>> Hmmm this sounds a bit weird... like there was some delay in the >>> communication path. >>> >>> Could you please look through your logs and look for lines including >>> "view" keyword? Are there two nodes, as expected? How the timestamps relate >>> to your experiment? >>> >>

The issue was resolved in a somewhat unexpected way. I had a custom org.keycloak.storage.UserStorageProviderFactory SPI registered that returned providers implementing org.keycloak.storage.user.UserLookupProvider, but org.keycloak.storage.user.UserLookupProvider#getUserById method wasn't fully filled out. I just had it return null. It wasn't obvious to me that it was required (or under what circumstances). Once I implemented it, the experiments in my original message passed. I did have to set owners to 2 for the "sessions" and "clientSessions" distributed cache infinispan configs. One thing I noticed is that node2 (the one that doesn't get hit on the initial password auth) has to do a lookup via getUserById the first time it handles a grant_type=refresh_token auth. Is the data it needs not shared across the cluster? It seems to be cached only locally on the node. Just as a test I tried to set all configured non-local caches to be replicated and it didn't help. Any thoughts about this? Thanks, DV >

Makes sense re: replicated caches. Here's my infinispan subsystem config right now: <subsystem xmlns="urn:jboss:domain:infinispan:4.0"> <cache-container name="keycloak" jndi-name="infinispan/Keycloak" statistics-enabled="true"> <transport lock-timeout="60000"/> <local-cache name="realms" statistics-enabled="true"> <eviction max-entries="10000" strategy="LRU"/> </local-cache> <local-cache name="users" statistics-enabled="true"> <eviction max-entries="10000" strategy="LRU"/> </local-cache> <!-- These two need to be replicated or the node that didn't issue the initial refresh token will return "invalid_grant" errors when attempting to auth with that refresh token. --> <replicated-cache name="sessions" statistics-enabled="true"/> <replicated-cache name="clientSessions" statistics-enabled="true"/> <distributed-cache name="authenticationSessions" mode="SYNC" owners="1" statistics-enabled="true"/> <distributed-cache name="offlineSessions" mode="SYNC" owners="1" statistics-enabled="true"/> <distributed-cache name="offlineClientSessions" mode="SYNC" owners="1" statistics-enabled="true"/> <distributed-cache name="loginFailures" mode="SYNC" owners="1" statistics-enabled="true"/> <local-cache name="authorization" statistics-enabled="true"> <eviction max-entries="10000" strategy="LRU"/> </local-cache> <replicated-cache name="work" mode="SYNC" statistics-enabled="true"/> <local-cache name="keys" statistics-enabled="true"> <eviction max-entries="1000" strategy="LRU"/> <expiration max-idle="3600000"/> </local-cache> <distributed-cache name="actionTokens" mode="SYNC" owners="2" statistics-enabled="true"> <eviction max-entries="-1" strategy="NONE"/> <expiration max-idle="-1" interval="300000"/> </distributed-cache> </cache-container> <cache-container name="server" aliases="singleton cluster" default-cache="default" module="org.wildfly.clustering.server"> <transport lock-timeout="60000"/> <replicated-cache name="default"> <transaction mode="BATCH"/> </replicated-cache> </cache-container> <cache-container name="web" default-cache="dist" module="org.wildfly.clustering.web.infinispan"> <transport lock-timeout="60000"/> <distributed-cache name="dist"> <locking isolation="REPEATABLE_READ"/> <transaction mode="BATCH"/> <file-store/> </distributed-cache> </cache-container> <cache-container name="ejb" aliases="sfsb" default-cache="dist" module="org.wildfly.clustering.ejb.infinispan"> <transport lock-timeout="60000"/> <distributed-cache name="dist"> <locking isolation="REPEATABLE_READ"/> <transaction mode="BATCH"/> <file-store/> </distributed-cache> </cache-container> <cache-container name="hibernate" default-cache="local-query" module="org.hibernate.infinispan"> <transport lock-timeout="60000"/> <local-cache name="local-query"> <eviction strategy="LRU" max-entries="10000"/> <expiration max-idle="100000"/> </local-cache> <invalidation-cache name="entity"> <transaction mode="NON_XA"/> <eviction strategy="LRU" max-entries="10000"/> <expiration max-idle="100000"/> </invalidation-cache> <replicated-cache name="timestamps" mode="ASYNC"/> </cache-container> </subsystem> The scenario I'm testing: 1. Auth with grant_type=password on node1. 2. Shut down node1. 3. Auth with grant_type=refresh_token on node2. When client_sessions is not replicated (distributed, with owners=1, as in the distribution's standalone-ha.xml), I get this on node2: { "error": "invalid_grant", "error_description": "Session doesn't have required client" } When sessions is not replicated: { "error": "invalid_grant", "error_description": "Session not active" } On Wed, Sep 19, 2018 at 6:56 AM Sebastian Laskawiec <slaskawi(a)redhat.com&gt; wrote: > Thanks for letting us know DV! > > Setting the number of owners equal to the cluster size doesn't make any > sense. You might use a replicated cache in that scenarios (which works the > same way apart from some Infinispan internal behavior, which can be omitted > in your case). Could you please paste your Infinispan configuration? Maybe > there's some hint there... > > Thanks, > Seb > > On Tue, Sep 18, 2018 at 11:02 PM D V <dv(a)glyphy.com&gt; wrote: > >> The issue was resolved in a somewhat unexpected way. I had a custom >> org.keycloak.storage.UserStorageProviderFactory SPI registered that >> returned providers >> implementing org.keycloak.storage.user.UserLookupProvider, >> but org.keycloak.storage.user.UserLookupProvider#getUserById method wasn't >> fully filled out. I just had it return null. It wasn't obvious to me that >> it was required (or under what circumstances). Once I implemented it, the >> experiments in my original message passed. I did have to set owners to 2 >> for the "sessions" and "clientSessions" distributed cache infinispan >> configs. >> >> One thing I noticed is that node2 (the one that doesn't get hit on the >> initial password auth) has to do a lookup via getUserById the first time it >> handles a grant_type=refresh_token auth. Is the data it needs not shared >> across the cluster? It seems to be cached only locally on the node. Just as >> a test I tried to set all configured non-local caches to be replicated and >> it didn't help. Any thoughts about this? >> >> Thanks, >> DV >> >>>

OK. So, with all caches being replicated, there's an error on startup: 2018-09-20 14:03:38,307 ERROR [org.infinispan.remoting.rpc.RpcManagerImpl] (ServerService Thread Pool -- 62) ISPN000073: Unexpected error while replicating: org.infinispan.commons.marshall.NotSerializableException: org.keycloak.models.PasswordPolicy$Builder Caused by: an exception which occurred: in field org.keycloak.models.PasswordPolicy.builder in object org.keycloak.models.PasswordPolicy@6ab5350d in field org.keycloak.models.cache.infinispan.entities.CachedRealm.passwordPolicy in object org.keycloak.models.cache.infinispan.entities.CachedRealm@7864be21 in object org.keycloak.models.cache.infinispan.entities.CachedRealm@7864be21 in object org.infinispan.commands.write.PutKeyValueCommand@fec4dc5e in object org.infinispan.commands.remote.SingleRpcCommand@3f2e5d1a If I make the realms cache local but leave the rest replicated, I observe the same behaviour: the node that didn't issue the original set of refresh/access tokens does a getUserById lookup, which in my case results in a network call against a remote service. I noticed there are caches running that aren't mentioned in the config, like userRevisions. These are local and adding them to the config as replicated doesn't actually make them as such. On Thu, Sep 20, 2018 at 7:36 AM Sebastian Laskawiec <slaskawi(a)redhat.com&gt; wrote: > Could you please try to unify the caches? Please replace all local-cache > and distributed-cache with replicated-cache. > > Even though using distributed caches over replicated ones should be the > cause, I think those local caches might cause the behavior you're > describing. > > On Wed, Sep 19, 2018 at 3:21 PM D V <dv(a)glyphy.com&gt; wrote: > >> Makes sense re: replicated caches. Here's my infinispan subsystem config >> right now: >> >> <subsystem xmlns="urn:jboss:domain:infinispan:4.0"> >> <cache-container name="keycloak" >> jndi-name="infinispan/Keycloak" statistics-enabled="true"> >> <transport lock-timeout="60000"/> >> <local-cache name="realms" statistics-enabled="true"> >> <eviction max-entries="10000" strategy="LRU"/> >> </local-cache> >> <local-cache name="users" statistics-enabled="true"> >> <eviction max-entries="10000" strategy="LRU"/> >> </local-cache> >> >> <!-- >> These two need to be replicated or the node that didn't >> issue the initial refresh token >> will return "invalid_grant" errors when attempting to >> auth with that refresh token. >> --> >> <replicated-cache name="sessions" >> statistics-enabled="true"/> >> <replicated-cache name="clientSessions" >> statistics-enabled="true"/> >> >> <distributed-cache name="authenticationSessions" >> mode="SYNC" owners="1" statistics-enabled="true"/> >> <distributed-cache name="offlineSessions" mode="SYNC" >> owners="1" statistics-enabled="true"/> >> <distributed-cache name="offlineClientSessions" >> mode="SYNC" owners="1" statistics-enabled="true"/> >> <distributed-cache name="loginFailures" mode="SYNC" >> owners="1" statistics-enabled="true"/> >> <local-cache name="authorization" >> statistics-enabled="true"> >> <eviction max-entries="10000" strategy="LRU"/> >> </local-cache> >> <replicated-cache name="work" mode="SYNC" >> statistics-enabled="true"/> >> <local-cache name="keys" statistics-enabled="true"> >> <eviction max-entries="1000" strategy="LRU"/> >> <expiration max-idle="3600000"/> >> </local-cache> >> <distributed-cache name="actionTokens" mode="SYNC" >> owners="2" statistics-enabled="true"> >> <eviction max-entries="-1" strategy="NONE"/> >> <expiration max-idle="-1" interval="300000"/> >> </distributed-cache> >> </cache-container> >> <cache-container name="server" aliases="singleton cluster" >> default-cache="default" module="org.wildfly.clustering.server"> >> <transport lock-timeout="60000"/> >> <replicated-cache name="default"> >> <transaction mode="BATCH"/> >> </replicated-cache> >> </cache-container> >> <cache-container name="web" default-cache="dist" >> module="org.wildfly.clustering.web.infinispan"> >> <transport lock-timeout="60000"/> >> <distributed-cache name="dist"> >> <locking isolation="REPEATABLE_READ"/> >> <transaction mode="BATCH"/> >> <file-store/> >> </distributed-cache> >> </cache-container> >> <cache-container name="ejb" aliases="sfsb" >> default-cache="dist" module="org.wildfly.clustering.ejb.infinispan"> >> <transport lock-timeout="60000"/> >> <distributed-cache name="dist"> >> <locking isolation="REPEATABLE_READ"/> >> <transaction mode="BATCH"/> >> <file-store/> >> </distributed-cache> >> </cache-container> >> <cache-container name="hibernate" >> default-cache="local-query" module="org.hibernate.infinispan"> >> <transport lock-timeout="60000"/> >> <local-cache name="local-query"> >> <eviction strategy="LRU" max-entries="10000"/> >> <expiration max-idle="100000"/> >> </local-cache> >> <invalidation-cache name="entity"> >> <transaction mode="NON_XA"/> >> <eviction strategy="LRU" max-entries="10000"/> >> <expiration max-idle="100000"/> >> </invalidation-cache> >> <replicated-cache name="timestamps" mode="ASYNC"/> >> </cache-container> >> </subsystem> >> >> The scenario I'm testing: >> 1. Auth with grant_type=password on node1. >> 2. Shut down node1. >> 3. Auth with grant_type=refresh_token on node2. >> >> When client_sessions is not replicated (distributed, with owners=1, as >> in the distribution's standalone-ha.xml), I get this on node2: >> { >> "error": "invalid_grant", >> "error_description": "Session doesn't have required client" >> } >> >> When sessions is not replicated: >> { >> "error": "invalid_grant", >> "error_description": "Session not active" >> } >> >> On Wed, Sep 19, 2018 at 6:56 AM Sebastian Laskawiec <slaskawi(a)redhat.com&gt; >> wrote: >> >>> Thanks for letting us know DV! >>> >>> Setting the number of owners equal to the cluster size doesn't make any >>> sense. You might use a replicated cache in that scenarios (which works the >>> same way apart from some Infinispan internal behavior, which can be omitted >>> in your case). Could you please paste your Infinispan configuration? Maybe >>> there's some hint there... >>> >>> Thanks, >>> Seb >>> >>> On Tue, Sep 18, 2018 at 11:02 PM D V <dv(a)glyphy.com&gt; wrote: >>> >>>> The issue was resolved in a somewhat unexpected way. I had a custom >>>> org.keycloak.storage.UserStorageProviderFactory SPI registered that >>>> returned providers >>>> implementing org.keycloak.storage.user.UserLookupProvider, >>>> but org.keycloak.storage.user.UserLookupProvider#getUserById method wasn't >>>> fully filled out. I just had it return null. It wasn't obvious to me that >>>> it was required (or under what circumstances). Once I implemented it, the >>>> experiments in my original message passed. I did have to set owners to 2 >>>> for the "sessions" and "clientSessions" distributed cache infinispan >>>> configs. >>>> >>>> One thing I noticed is that node2 (the one that doesn't get hit on the >>>> initial password auth) has to do a lookup via getUserById the first time it >>>> handles a grant_type=refresh_token auth. Is the data it needs not shared >>>> across the cluster? It seems to be cached only locally on the node. Just as >>>> a test I tried to set all configured non-local caches to be replicated and >>>> it didn't help. Any thoughts about this? >>>> >>>> Thanks, >>>> DV >>>> >>>>>

Sorry, I did not read whole thread. Just a quick note, that caches "realms", "users", "keys" and "authorization" are supposed to be local caches. The pattern, we're using ATM is, that every cluster node caches it's data (realms, users etc) locally. In case that some objects are updated (EG. realm or users), there is separate cache "work", which make sure to notify other cluster nodes (or even nodes on all the other DCs), so all the nodes can invalidate particular cached object from their caches. Caches "realms", "users", "keys" and "authorization" are not meant to be replicated/distributed, but local. So this NotSerializableException doesn't look like a bug to me. Marek On 25/09/18 12:06, Sebastian Laskawiec wrote: > Thanks a lot for checking this. > > This seems like a bug to me, so I filled > https://issues.jboss.org/browse/KEYCLOAK-8415. Unfortunately, we are > preparing for some urgent work on the product side and I can't > promise you when we will be able to look into this. I highly > encourage you to contribute a fix if you are in hurry or just > subscribe to the ticket and wait till we find a free slot to get it > fixed. > > Thanks, > Sebastian > > On Thu, Sep 20, 2018 at 4:27 PM D V <dv(a)glyphy.com > <mailto:dv@glyphy.com>> wrote: > > OK. So, with all caches being replicated, there's an error on > startup: > > 2018-09-20 14:03:38,307 ERROR > [org.infinispan.remoting.rpc.RpcManagerImpl] (ServerService > Thread Pool -- 62) ISPN000073: Unexpected error while > replicating: > org.infinispan.commons.marshall.NotSerializableException: > org.keycloak.models.PasswordPolicy$Builder > Caused by: an exception which occurred: > in field org.keycloak.models.PasswordPolicy.builder > in object org.keycloak.models.PasswordPolicy@6ab5350d > in field > org.keycloak.models.cache.infinispan.entities.CachedRealm.passwordPolicy > in object > org.keycloak.models.cache.infinispan.entities.CachedRealm@7864be21 > in object > org.keycloak.models.cache.infinispan.entities.CachedRealm@7864be21 > in object org.infinispan.commands.write.PutKeyValueCommand@fec4dc5e > in object org.infinispan.commands.remote.SingleRpcCommand@3f2e5d1a > > If I make the realms cache local but leave the rest replicated, I > observe the same behaviour: the node that didn't issue the > original set of refresh/access tokens does a getUserById lookup, > which in my case results in a network call against a remote service. > > I noticed there are caches running that aren't mentioned in the > config, like userRevisions. These are local and adding them to > the config as replicated doesn't actually make them as such. > > On Thu, Sep 20, 2018 at 7:36 AM Sebastian Laskawiec > <slaskawi(a)redhat.com <mailto:slaskawi@redhat.com>> wrote: > > Could you please try to unify the caches? Please replace > all local-cache and distributed-cache with replicated-cache. > > Even though using distributed caches over replicated ones > should be the cause, I think those local caches might cause > the behavior you're describing. > > On Wed, Sep 19, 2018 at 3:21 PM D V <dv(a)glyphy.com > <mailto:dv@glyphy.com>> wrote: > > Makes sense re: replicated caches. Here's my infinispan > subsystem config right now: > >         <subsystem xmlns="urn:jboss:domain:infinispan:4.0"> >             <cache-container name="keycloak" > jndi-name="infinispan/Keycloak" statistics-enabled="true"> >                 <transport lock-timeout="60000"/> >                 <local-cache name="realms" > statistics-enabled="true"> >                     <eviction max-entries="10000" > strategy="LRU"/> >                 </local-cache> >                 <local-cache name="users" > statistics-enabled="true"> >                     <eviction max-entries="10000" > strategy="LRU"/> >                 </local-cache> > >                 <!-- >                 These two need to be replicated or the > node that didn't issue the initial refresh token >                 will return "invalid_grant" errors when > attempting to auth with that refresh token. >                 --> >                 <replicated-cache name="sessions" > statistics-enabled="true"/> >                 <replicated-cache name="clientSessions" > statistics-enabled="true"/> > >                 <distributed-cache > name="authenticationSessions" mode="SYNC" owners="1" > statistics-enabled="true"/> >                 <distributed-cache name="offlineSessions" > mode="SYNC" owners="1" statistics-enabled="true"/> >                 <distributed-cache > name="offlineClientSessions" mode="SYNC" owners="1" > statistics-enabled="true"/> >                 <distributed-cache name="loginFailures" > mode="SYNC" owners="1" statistics-enabled="true"/> >                 <local-cache name="authorization" > statistics-enabled="true"> >                     <eviction max-entries="10000" > strategy="LRU"/> >                 </local-cache> >                 <replicated-cache name="work" mode="SYNC" > statistics-enabled="true"/> >                 <local-cache name="keys" > statistics-enabled="true"> >                     <eviction max-entries="1000" > strategy="LRU"/> >                     <expiration max-idle="3600000"/> >                 </local-cache> >                 <distributed-cache name="actionTokens" > mode="SYNC" owners="2" statistics-enabled="true"> >                     <eviction max-entries="-1" > strategy="NONE"/> >                     <expiration max-idle="-1" > interval="300000"/> > </distributed-cache> >             </cache-container> >             <cache-container name="server" > aliases="singleton cluster" default-cache="default" > module="org.wildfly.clustering.server"> >                 <transport lock-timeout="60000"/> >                 <replicated-cache name="default"> >                     <transaction mode="BATCH"/> > </replicated-cache> >             </cache-container> >             <cache-container name="web" > default-cache="dist" > module="org.wildfly.clustering.web.infinispan"> >                 <transport lock-timeout="60000"/> >                 <distributed-cache name="dist"> >                     <locking isolation="REPEATABLE_READ"/> >                     <transaction mode="BATCH"/> > <file-store/> > </distributed-cache> >             </cache-container> >             <cache-container name="ejb" aliases="sfsb" > default-cache="dist" > module="org.wildfly.clustering.ejb.infinispan"> >                 <transport lock-timeout="60000"/> >                 <distributed-cache name="dist"> >                     <locking isolation="REPEATABLE_READ"/> >                     <transaction mode="BATCH"/> > <file-store/> > </distributed-cache> >             </cache-container> >             <cache-container name="hibernate" > default-cache="local-query" > module="org.hibernate.infinispan"> >                 <transport lock-timeout="60000"/> >                 <local-cache name="local-query"> >                     <eviction strategy="LRU" > max-entries="10000"/> >                     <expiration max-idle="100000"/> >                 </local-cache> > <invalidation-cache name="entity"> >                     <transaction mode="NON_XA"/> >                     <eviction strategy="LRU" > max-entries="10000"/> >                     <expiration max-idle="100000"/> > </invalidation-cache> >                 <replicated-cache name="timestamps" > mode="ASYNC"/> >             </cache-container> >         </subsystem> > > The scenario I'm testing: > 1. Auth with grant_type=password on node1. > 2. Shut down node1. > 3. Auth with grant_type=refresh_token on node2. > > When client_sessions is not replicated (distributed, with > owners=1, as in the distribution's standalone-ha.xml), I > get this on node2: > { >     "error": "invalid_grant", >     "error_description": "Session doesn't have required > client" > } > > When sessions is not replicated: > { >     "error": "invalid_grant", >     "error_description": "Session not active" > } > > On Wed, Sep 19, 2018 at 6:56 AM Sebastian Laskawiec > <slaskawi(a)redhat.com <mailto:slaskawi@redhat.com>> wrote: > > Thanks for letting us know DV! > > Setting the number of owners equal to the cluster > size doesn't make any sense. You might use a > replicated cache in that scenarios (which works the > same way apart from some Infinispan internal > behavior, which can be omitted in your case). Could > you please paste your Infinispan configuration? Maybe > there's some hint there... > > Thanks, > Seb > > On Tue, Sep 18, 2018 at 11:02 PM D V <dv(a)glyphy.com > <mailto:dv@glyphy.com>> wrote: > > The issue was resolved in a somewhat unexpected > way. I had a custom > org.keycloak.storage.UserStorageProviderFactory > SPI registered that returned providers > implementing org.keycloak.storage.user.UserLookupProvider, > but org.keycloak.storage.user.UserLookupProvider#getUserById > method wasn't fully filled out. I just had it > return null. It wasn't obvious to me that it was > required (or under what circumstances). Once I > implemented it, the experiments in my original > message passed. I did have to set owners to 2 for > the "sessions" and "clientSessions" distributed > cache infinispan configs. > > One thing I noticed is that node2 (the one that > doesn't get hit on the initial password auth) has > to do a lookup via getUserById the first time it > handles a grant_type=refresh_token auth. Is the > data it needs not shared across the cluster? It > seems to be cached only locally on the node. Just > as a test I tried to set all configured non-local > caches to be replicated and it didn't help. Any > thoughts about this? > > Thanks, > DV >

Thanks for the responses, folks. The issue now isn't the inability to set all caches to replicated. It's that a get-user-by-id is called whenever a node has to process a an authentication via a refresh token that wasn't also issued by that same node. See the last paragraph of http://lists.jboss.org/pipermail/keycloak-user/2018-September/015549.html . The results are cached, but only on the original issuing node. I was expecting the user-by-id information to be shared between keycloak nodes to avoid external service calls, but perhaps this is by design? If so, could you explain why? On Tue, Sep 25, 2018 at 9:14 AM Marek Posolda <mposolda(a)redhat.com <mailto:mposolda@redhat.com>> wrote: Some more info about our caches: https://www.keycloak.org/docs/latest/server_installation/index.html#cache... Not sure if this info should be updated and some more things to be clearified? Marek On 25/09/18 15:12, Marek Posolda wrote: > Sorry, I did not read whole thread. > > Just a quick note, that caches "realms", "users", "keys" and > "authorization" are supposed to be local caches. The pattern, > we're using ATM is, that every cluster node caches it's data > (realms, users etc) locally. In case that some objects are > updated (EG. realm or users), there is separate cache "work", > which make sure to notify other cluster nodes (or even nodes on > all the other DCs), so all the nodes can invalidate particular > cached object from their caches. > > Caches "realms", "users", "keys" and "authorization" are not > meant to be replicated/distributed, but local. So this > NotSerializableException doesn't look like a bug to me. > > Marek > > On 25/09/18 12:06, Sebastian Laskawiec wrote: >> Thanks a lot for checking this. >> >> This seems like a bug to me, so I filled >> https://issues.jboss.org/browse/KEYCLOAK-8415. Unfortunately, we >> are preparing for some urgent work on the product side and I >> can't promise you when we will be able to look into this. I >> highly encourage you to contribute a fix if you are in hurry or >> just subscribe to the ticket and wait till we find a free slot >> to get it fixed. >> >> Thanks, >> Sebastian >> >> On Thu, Sep 20, 2018 at 4:27 PM D V <dv(a)glyphy.com >> <mailto:dv@glyphy.com>> wrote: >> >> OK. So, with all caches being replicated, there's an error >> on startup: >> >> 2018-09-20 14:03:38,307 ERROR >> [org.infinispan.remoting.rpc.RpcManagerImpl] (ServerService >> Thread Pool -- 62) ISPN000073: Unexpected error while >> replicating: >> org.infinispan.commons.marshall.NotSerializableException: >> org.keycloak.models.PasswordPolicy$Builder >> Caused by: an exception which occurred: >> in field org.keycloak.models.PasswordPolicy.builder >> in object org.keycloak.models.PasswordPolicy@6ab5350d >> in field >> org.keycloak.models.cache.infinispan.entities.CachedRealm.passwordPolicy >> in object >> org.keycloak.models.cache.infinispan.entities.CachedRealm@7864be21 >> in object >> org.keycloak.models.cache.infinispan.entities.CachedRealm@7864be21 >> in object >> org.infinispan.commands.write.PutKeyValueCommand@fec4dc5e >> in object >> org.infinispan.commands.remote.SingleRpcCommand@3f2e5d1a >> >> If I make the realms cache local but leave the rest >> replicated, I observe the same behaviour: the node that >> didn't issue the original set of refresh/access tokens does >> a getUserById lookup, which in my case results in a network >> call against a remote service. >> >> I noticed there are caches running that aren't mentioned in >> the config, like userRevisions. These are local and adding >> them to the config as replicated doesn't actually make them >> as such. >> >> On Thu, Sep 20, 2018 at 7:36 AM Sebastian Laskawiec >> <slaskawi(a)redhat.com <mailto:slaskawi@redhat.com>> wrote: >> >> Could you please try to unify the caches? Please replace >> all local-cache and distributed-cache with replicated-cache. >> >> Even though using distributed caches over replicated >> ones should be the cause, I think those local caches >> might cause the behavior you're describing. >> >> On Wed, Sep 19, 2018 at 3:21 PM D V <dv(a)glyphy.com >> <mailto:dv@glyphy.com>> wrote: >> >> Makes sense re: replicated caches. Here's my >> infinispan subsystem config right now: >> >>         <subsystem >> xmlns="urn:jboss:domain:infinispan:4.0"> >> <cache-container name="keycloak" >> jndi-name="infinispan/Keycloak" >> statistics-enabled="true"> >>                 <transport lock-timeout="60000"/> >> <local-cache name="realms" statistics-enabled="true"> >> <eviction max-entries="10000" strategy="LRU"/> >> </local-cache> >> <local-cache name="users" statistics-enabled="true"> >> <eviction max-entries="10000" strategy="LRU"/> >> </local-cache> >> >>                 <!-- >>                 These two need to be replicated or >> the node that didn't issue the initial refresh token >>                 will return "invalid_grant" errors >> when attempting to auth with that refresh token. >>                 --> >> <replicated-cache name="sessions" >> statistics-enabled="true"/> >> <replicated-cache name="clientSessions" >> statistics-enabled="true"/> >> >> <distributed-cache name="authenticationSessions" >> mode="SYNC" owners="1" statistics-enabled="true"/> >> <distributed-cache name="offlineSessions" >> mode="SYNC" owners="1" statistics-enabled="true"/> >> <distributed-cache name="offlineClientSessions" >> mode="SYNC" owners="1" statistics-enabled="true"/> >> <distributed-cache name="loginFailures" mode="SYNC" >> owners="1" statistics-enabled="true"/> >> <local-cache name="authorization" >> statistics-enabled="true"> >> <eviction max-entries="10000" strategy="LRU"/> >> </local-cache> >> <replicated-cache name="work" mode="SYNC" >> statistics-enabled="true"/> >> <local-cache name="keys" statistics-enabled="true"> >> <eviction max-entries="1000" strategy="LRU"/> >> <expiration max-idle="3600000"/> >> </local-cache> >> <distributed-cache name="actionTokens" mode="SYNC" >> owners="2" statistics-enabled="true"> >> <eviction max-entries="-1" strategy="NONE"/> >> <expiration max-idle="-1" interval="300000"/> >> </distributed-cache> >> </cache-container> >> <cache-container name="server" aliases="singleton >> cluster" default-cache="default" >> module="org.wildfly.clustering.server"> >>                 <transport lock-timeout="60000"/> >> <replicated-cache name="default"> >> <transaction mode="BATCH"/> >> </replicated-cache> >> </cache-container> >> <cache-container name="web" default-cache="dist" >> module="org.wildfly.clustering.web.infinispan"> >>                 <transport lock-timeout="60000"/> >> <distributed-cache name="dist"> >> <locking isolation="REPEATABLE_READ"/> >> <transaction mode="BATCH"/> >> <file-store/> >> </distributed-cache> >> </cache-container> >> <cache-container name="ejb" aliases="sfsb" >> default-cache="dist" >> module="org.wildfly.clustering.ejb.infinispan"> >>                 <transport lock-timeout="60000"/> >> <distributed-cache name="dist"> >> <locking isolation="REPEATABLE_READ"/> >> <transaction mode="BATCH"/> >> <file-store/> >> </distributed-cache> >> </cache-container> >> <cache-container name="hibernate" >> default-cache="local-query" >> module="org.hibernate.infinispan"> >>                 <transport lock-timeout="60000"/> >> <local-cache name="local-query"> >> <eviction strategy="LRU" max-entries="10000"/> >> <expiration max-idle="100000"/> >> </local-cache> >> <invalidation-cache name="entity"> >> <transaction mode="NON_XA"/> >> <eviction strategy="LRU" max-entries="10000"/> >> <expiration max-idle="100000"/> >> </invalidation-cache> >> <replicated-cache name="timestamps" mode="ASYNC"/> >> </cache-container> >>         </subsystem> >> >> The scenario I'm testing: >> 1. Auth with grant_type=password on node1. >> 2. Shut down node1. >> 3. Auth with grant_type=refresh_token on node2. >> >> When client_sessions is not replicated (distributed, >> with owners=1, as in the distribution's >> standalone-ha.xml), I get this on node2: >> { >>     "error": "invalid_grant", >>     "error_description": "Session doesn't have >> required client" >> } >> >> When sessions is not replicated: >> { >>     "error": "invalid_grant", >>     "error_description": "Session not active" >> } >> >> On Wed, Sep 19, 2018 at 6:56 AM Sebastian Laskawiec >> <slaskawi(a)redhat.com <mailto:slaskawi@redhat.com>> >> wrote: >> >> Thanks for letting us know DV! >> >> Setting the number of owners equal to the >> cluster size doesn't make any sense. You might >> use a replicated cache in that scenarios (which >> works the same way apart from some Infinispan >> internal behavior, which can be omitted in your >> case). Could you please paste your Infinispan >> configuration? Maybe there's some hint there... >> >> Thanks, >> Seb >> >> On Tue, Sep 18, 2018 at 11:02 PM D V >> <dv(a)glyphy.com <mailto:dv@glyphy.com>> wrote: >> >> The issue was resolved in a somewhat >> unexpected way. I had a custom >> org.keycloak.storage.UserStorageProviderFactory >> SPI registered that returned providers >> implementing org.keycloak.storage.user.UserLookupProvider, >> but org.keycloak.storage.user.UserLookupProvider#getUserById >> method wasn't fully filled out. I just had >> it return null. It wasn't obvious to me that >> it was required (or under what >> circumstances). Once I implemented it, the >> experiments in my original message passed. I >> did have to set owners to 2 for the >> "sessions" and "clientSessions" distributed >> cache infinispan configs. >> >> One thing I noticed is that node2 (the one >> that doesn't get hit on the initial password >> auth) has to do a lookup via getUserById the >> first time it handles a >> grant_type=refresh_token auth. Is the data >> it needs not shared across the cluster? It >> seems to be cached only locally on the node. >> Just as a test I tried to set all configured >> non-local caches to be replicated and it >> didn't help. Any thoughts about this? >> >> Thanks, >> DV >> >