[keycloak-user] Scalability with a lot of clients

[El-Himoud Nils (INST/ECS2)]

Hi everyone,

In our project we are creating lots of clients in Keycloak. In our loadtests with ~6000 clients we found very slow response times.

For example average response times during load tests:
23.9 sec / admin/realms/{realm}/users/{id}/role-mappings (GET)
28.2 sec /admin/realms/{realm}/clients (POST)
20.2 sec /admin/realms/{realm}/clients/{id} (DELETE)

By debugging Keycloak we found that the server is iterating over all clients in the realm. For this finding we opened ticket https://issues.jboss.org/browse/KEYCLOAK-9553. Initially after Keycloak startup this could take up to 5 minutes but was much faster for subsequent requests. We assume due to local caches. The variance of the response times is very high. They range from <1s to timeouts after 5 minutes.

What we´ve tried so far:
First we scaled up the Keycloak instances because we thought it might be a load problem. Turned out that it doesn´t need load to reproduce the slow responses, just enough clients.

Then we tried to warmup the caches by running the loadtests for a longer time but couldn´t see improvements.

We found that there are configuration options for the caches and tried to gain some insight on the runtime behavior via jboss cli by enabling the cache statistics

/subsystem=infinispan/cache-container=keycloak/local-cache=realms:write-attribute(name=statistics-enabled,value=true)
/subsystem=infinispan/cache-container=keycloak:write-attribute(name=statistics-enabled, value=true)
:reload

With no success. Statistics keep showing only zeros.

[standalone at localhost:9990 /] ls subsystem=infinispan/cache-container=keycloak/local-cache=realms
component                                 elapsed-time=0                            module=undefined                          stores=0
memory                                    hit-ratio=0.0                             number-of-entries=0                       time-since-reset=0
store                                     hits=0                                    passivations=0                            eviction={"EVICTION" => undefined}
activations=0                             indexing=NONE                             read-write-ratio=0.0                      expiration={"EXPIRATION" => undefined}
average-read-time=0                       indexing-properties=undefined             remove-hits=0                             locking={"LOCKING" => undefined}
average-write-time=0                      invalidations=0                           remove-misses=0                           transaction={"TRANSACTION" => undefined}
batching=false                            jndi-name=undefined                       start=LAZY
cache-status=RUNNING                      misses=0                                  statistics-enabled=false

At last we tried to manipulate the cache settings for the realms cache

# default 10000
/subsystem=infinispan/cache-container=keycloak/local-cache=realms/memory=object:write-attribute(name=max-entries, value=80000)
# default 10000
/subsystem=infinispan/cache-container=keycloak/local-cache=realms/memory=object:write-attribute(name=size, value=80000)

This also had no noticeable effect on the response times.

Ah! The connection pool size for the db was also something that we tried to increase.

Setup
Keycloak is running in standalone HA mode with jgroups on Kubernetes (3 replicas). The database is AWS RDS.

Next we want to test the scalability of Keycloak with respect to the number of clients.

Do we miss something about the cache configuration. Is the realm cache the correct one to optimize the problematic endpoints? How can we get the cache statistics working?

resources:
  request:
    mem: 4Gi
    cpu: 1
  limit:
    mem: 6Gi
    cpu: 3

Java memory parameters are set
/usr/lib/jvm/java/bin/java -D[Standalone] -server -Xms3276m -Xmx4914m -javaagent:/opt/jboss/newrelic/newrelic.jar -Djboss.modules.system.pkgs


Mit freundlichen Grüßen / Best regards

Nils El-Himoud
INST-IOT/ESW-Imb