[keycloak-user] Keycloak in HA mode on Kubernetes fails with "invalid_code" when requesting tokens

[Jody H]

Hi,

we have some trouble to generate tokens with the authentication code flow
in our Keycloak 5.0.0 cluster.
Some information about the cluster:
1) We have a cluster with 3 instances in Kubernetes, deployed by the
Keycloak Helm Chart (
https://github.com/helm/charts/tree/master/stable/keycloak)
2) I can see that some Infinispan stuff is going on in the logs when the
cluster is starting up. I have checked that the shell script that is
executed on startup contains the " -c standalone-ha.xml" switch. I can not
find any mentions of the string "standalone-ha.xml" in the log output
though.
3) Our cluster is loadbalanced with a HAProxy
4) The webservice we want to access is secured by Keycloak Gatekeeper (
https://github.com/keycloak/keycloak-gatekeeper)

When using a browser to log in to keycloak-secured websites (i.e. websites
that use the keycloak cluster to perform the OIDC authentication code flow
and authenticate our users), we did not see problems so far. The keycloak
gatekeeper "proxy" is redirecting to keycloak when no cookie is present for
login, trading in the code for id, access and refresh tokens and passing
the access_token to the reverse-proxied website after successful login.

To test our APIs we would like to use Postman.
However, when using Postman with its built-in OAuth 2.0 authentication, we
see a problem that is reproducible on 4 laptops which are in the same LAN
as the keycloak cluster. Postman can request access tokens by using the
authentication code flow in its GUI. In Postmans "Get New Access Token"
window, we use these settings:
1. callback url: the same redirect_uri that is pointing to the Keycloak
gatekeeper callback endpoint (/oauth/callback endpoint)
2. auth url:
https://keycloak.domain/auth/realms/our-realm/protocol/openid-connect/auth
3. access token url:
https://keycloak.domain/auth/realms/our-realm/protocol/openid-connect/token
4. client-id: client-id from Keycloak
5. client-secet: client-secret from Keycloak
6. scope: openid
7. Client Authentication: "Send as Basic Auth header"

When clicking the "Request Token" button in Postman, we receive the error
"invalid_code" in roughly 9 out of 10 tries. Basically, if we spam the
button, sometimes it works but most of the time it does not. For another
laptop which is connected via VPN and thus has a higher latency, the
requests work just fine.

I am thinking about the following:
Is it possible that the initial request is sent to keycloak-0, then
returned to the client (postman) and then immediately sent back to the
loadbalancer-url to trade in the code for tokens... and then hits another
instance due to loadbalancing, for example keycloak-1, which has no
information about the authentication process that was initiated on
keycloak-0? The invalid_code error is returned after just 4 milliseconds,
which is rather fast. Maybe the cluster is not properly synchronizing in
time? Any idea on how to fix this?

Thanks
Jody