[keycloak-dev] Performance improvements with large number of realms

[Gabriel Lavoie]

Hi,
     following
http://lists.jboss.org/pipermail/keycloak-user/2016-October/008032.html
that was discussed in the keycloak-user mailing list, I've been working in
the last few weeks on fixing performance issues for Keycloak setup with a
very large number of realms (tested with 500-600). As suggested by Stian, I
have changes to submit as pull requests that I want to discuss here first
to make sure they are acceptable.

With this improvements, the following issues were solved in our setup:
- Startup time reduced from more than 15 minute to under a minute.
- Login time to the admin console reduced from ~5 minutes to ~20 secs when
cache is being filled, to under 5 when cache is filled.
- Realm creation time reduced from ~1 minute to under 10 secs.

1 - Too many autoFlush checks by Hibernate and explicit em.flush():
>From what I see, the JPA entity model is very disconnected and many
entities are retrieved through their ID using NamedQuery. I'm not sure why
the mapping was done this way.

This causes Hibernate to make a autoFlush check for dirty entities on every
query, which is very CPU intensive. There are a few entities that can
benefit from having a Collection/Set to their child entities and use it
rather than a query to retrieve the required information. Also, with
explicit mapping, I foresee multiple explicit em.flush() that could be
removed when changing the data.

In this changeset I've covered only what was necessary for us, but there
are still a few areas that could be improved in my opinion:

https://github.com/glavoie/keycloak/commit/93c8056498e38ed075523f5b9b78dbde2814d783


2 - Many missing index for FKs:
We are using an Oracle database and indexes are not created automatically
for FKs. I've added many but I'm not 100% sure if you want a new Liquibase
changeset for every changes that could go in a version or only one per
version. I still have one index to fix so don't take this as-is for now:

https://github.com/glavoie/keycloak/commit/7bb44b7dfeea31e663695218524a0da4ce48a331

3 - Using Set rather than Collection with JPA entities:
When using a Collection, Hibernate will delete all rows and re-insert them
in the join table to update the content. This behavior was very slow and
visible with the "admin" composite role that points to roles for all realms
(through the different clients). This was slowing down a lot new realm
creation. When using a Set, Hibernate does a better job at tracking changes.

https://github.com/glavoie/keycloak/commit/9eb1bf013e80395affeec09625d5ee6157a7084a


4 - "session" cache when testing for "hasRole" with a composite role:
The recursive search done KeycloakModelUtils.searchFor() can generate quite
a lot of hits on the Infinispan cache when trying to test for role. With
500 realms, we saw up to 1.5 millions hits on the realmsRevision cache just
to login to the admin console. I've added a in-memory cache of the
retrieved composite RoleModel list in the Infinispan RoleAdapter. As I
understand, every instance of a RoleAdapter is bound to a KeycloakSession
that can be seen as a transaction/request. Still relying on the delegate
object if the RealmAdapter is being invalidated.

This is the change that reduced the most the login time to the admin
console or the time to obtain a Bearer token for the admin API:

https://github.com/glavoie/keycloak/commit/0ab39670525315148761627e7d0f808d4ef9c3c0


5 - realmRevisions cache size:
In https://issues.jboss.org/browse/KEYCLOAK-3202, the realmRevisions cache
creation was moved from standalone.xml to Java code. With our current
Keycloak cluster we already added the eviction policy mentioned in the
ticket, but had to increase the size to 50000 to support our number of
realms. Having this size hardcoded in the Java code becomes a bit
inconvenient with the latest versions.

I didn't really fix the code as Stian mentioned that this part is being
heavily reworked. I would guess this has something to do with
http://lists.jboss.org/pipermail/keycloak-dev/2016-November/008397.html?

https://github.com/glavoie/keycloak/commit/6e3e7e8400da3440a743941980bad3dc83be7965


Thank you!

Gabriel

-- 
Gabriel Lavoie
glavoie at gmail.com