9:53 a.m.
Hi everyone,
In our project we are creating lots of clients in Keycloak. In our loadtests with ~6000
clients we found very slow response times.
For example average response times during load tests:
23.9 sec / admin/realms/{realm}/users/{id}/role-mappings (GET)
28.2 sec /admin/realms/{realm}/clients (POST)
20.2 sec /admin/realms/{realm}/clients/{id} (DELETE)
By debugging Keycloak we found that the server is iterating over all clients in the realm.
For this finding we opened ticket https://issues.jboss.org/browse/KEYCLOAK-9553. Initially
after Keycloak startup this could take up to 5 minutes but was much faster for subsequent
requests. We assume due to local caches. The variance of the response times is very high.
They range from <1s to timeouts after 5 minutes.
What we´ve tried so far:
First we scaled up the Keycloak instances because we thought it might be a load problem.
Turned out that it doesn´t need load to reproduce the slow responses, just enough
clients.
Then we tried to warmup the caches by running the loadtests for a longer time but couldn´t
see improvements.
We found that there are configuration options for the caches and tried to gain some
insight on the runtime behavior via jboss cli by enabling the cache statistics
/subsystem=infinispan/cache-container=keycloak/local-cache=realms:write-attribute(name=statistics-enabled,value=true)
/subsystem=infinispan/cache-container=keycloak:write-attribute(name=statistics-enabled,
value=true)
:reload
With no success. Statistics keep showing only zeros.
[standalone@localhost:9990 /] ls
subsystem=infinispan/cache-container=keycloak/local-cache=realms
component elapsed-time=0
module=undefined stores=0
memory hit-ratio=0.0
number-of-entries=0 time-since-reset=0
store hits=0
passivations=0 eviction={"EVICTION" => undefined}
activations=0 indexing=NONE
read-write-ratio=0.0 expiration={"EXPIRATION" =>
undefined}
average-read-time=0 indexing-properties=undefined
remove-hits=0 locking={"LOCKING" => undefined}
average-write-time=0 invalidations=0
remove-misses=0 transaction={"TRANSACTION" =>
undefined}
batching=false jndi-name=undefined
start=LAZY
cache-status=RUNNING misses=0
statistics-enabled=false
At last we tried to manipulate the cache settings for the realms cache
# default 10000
/subsystem=infinispan/cache-container=keycloak/local-cache=realms/memory=object:write-attribute(name=max-entries,
value=80000)
# default 10000
/subsystem=infinispan/cache-container=keycloak/local-cache=realms/memory=object:write-attribute(name=size,
value=80000)
This also had no noticeable effect on the response times.
Ah! The connection pool size for the db was also something that we tried to increase.
Setup
Keycloak is running in standalone HA mode with jgroups on Kubernetes (3 replicas). The
database is AWS RDS.
Next we want to test the scalability of Keycloak with respect to the number of clients.
Do we miss something about the cache configuration. Is the realm cache the correct one to
optimize the problematic endpoints? How can we get the cache statistics working?
resources:
request:
mem: 4Gi
cpu: 1
limit:
mem: 6Gi
cpu: 3
Java memory parameters are set
/usr/lib/jvm/java/bin/java -D[Standalone] -server -Xms3276m -Xmx4914m
-javaagent:/opt/jboss/newrelic/newrelic.jar -Djboss.modules.system.pkgs
Mit freundlichen Grüßen / Best regards
Nils El-Himoud
INST-IOT/ESW-Imb