[keycloak-user] Trouble with Keycloak Cluster Mode and Service Accounts

[Chris Savory]

We are currently doing some load testing of our application.  I have Keycloak configured to run in Standalone Clustered mode.  We are running Keycloak 5 in docker containers on AWS ECS.  We are using JDBC_PING for jgroups.  I have Sticky Sessions enabled on the front end, so logins and token retrievals through our Angular app are working fine. 

The problem I am running into right now is that when I go to create users via the service account on our backend API the TokenManager (inside the keycloak-admin-client) has to refresh it's token every 5 minutes.  I see a lot of these errors in the logs:

23:04:03,349 WARN [org.keycloak.events] (default task-29) type=REFRESH_TOKEN_ERROR, realmId=platform, clientId=elrc, userId=b33ec381-4e8b-425e-81e2-c526859ec7f2, ipAddress=52.4.47.98, error=invalid_token, grant_type=refresh_token, refresh_token_type=Refresh, refresh_token_id=9e6bf90c-aeba-4479-8d25-9b7b954bcb12, client_auth_method=client-secret

All this works fine when we use only one or two keycloaks in the cluster, but as soon as I try to scale to 3 or 4 keycloaks we see all kinds of errors trying refresh tokens.  I think this is because when our backend secret clients go to refresh their tokens, they do not have the session affinity to go back to the same keycloak instance where their token was originally generated, whereas front end users do get pinned to the same keycloak instance.  

Any ideas how I might solve this problem for our backend apis?  

--
Christopher Savory