[keycloak-dev] Performance improvements with large number of realms

Hi, following http://lists.jboss.org/pipermail/keycloak-user/2016-October/008032.html that was discussed in the keycloak-user mailing list, I've been working in the last few weeks on fixing performance issues for Keycloak setup with a very large number of realms (tested with 500-600). As suggested by Stian, I have changes to submit as pull requests that I want to discuss here first to make sure they are acceptable. With this improvements, the following issues were solved in our setup: - Startup time reduced from more than 15 minute to under a minute. - Login time to the admin console reduced from ~5 minutes to ~20 secs when cache is being filled, to under 5 when cache is filled. - Realm creation time reduced from ~1 minute to under 10 secs. 1 - Too many autoFlush checks by Hibernate and explicit em.flush(): entities are retrieved through their ID using NamedQuery. I'm not sure why the mapping was done this way. This causes Hibernate to make a autoFlush check for dirty entities on every query, which is very CPU intensive. There are a few entities that can benefit from having a Collection/Set to their child entities and use it rather than a query to retrieve the required information. Also, with explicit mapping, I foresee multiple explicit em.flush() that could be removed when changing the data. In this changeset I've covered only what was necessary for us, but there are still a few areas that could be improved in my opinion: https://github.com/glavoie/keycloak/commit/93c8056498e38ed075523f5b9b78db... 2 - Many missing index for FKs: We are using an Oracle database and indexes are not created automatically for FKs. I've added many but I'm not 100% sure if you want a new Liquibase changeset for every changes that could go in a version or only one per version. I still have one index to fix so don't take this as-is for now: https://github.com/glavoie/keycloak/commit/7bb44b7dfeea31e663695218524a0d... 3 - Using Set rather than Collection with JPA entities: When using a Collection, Hibernate will delete all rows and re-insert them in the join table to update the content. This behavior was very slow and visible with the "admin" composite role that points to roles for all realms (through the different clients). This was slowing down a lot new realm creation. When using a Set, Hibernate does a better job at tracking changes. https://github.com/glavoie/keycloak/commit/9eb1bf013e80395affeec09625d5ee... 4 - "session" cache when testing for "hasRole" with a composite role: The recursive search done KeycloakModelUtils.searchFor() can generate quite a lot of hits on the Infinispan cache when trying to test for role. With 500 realms, we saw up to 1.5 millions hits on the realmsRevision cache just to login to the admin console. I've added a in-memory cache of the retrieved composite RoleModel list in the Infinispan RoleAdapter. As I understand, every instance of a RoleAdapter is bound to a KeycloakSession that can be seen as a transaction/request. Still relying on the delegate object if the RealmAdapter is being invalidated. This is the change that reduced the most the login time to the admin console or the time to obtain a Bearer token for the admin API: https://github.com/glavoie/keycloak/commit/0ab39670525315148761627e7d0f80... 5 - realmRevisions cache size: In https://issues.jboss.org/browse/KEYCLOAK-3202, the realmRevisions cache creation was moved from standalone.xml to Java code. With our current Keycloak cluster we already added the eviction policy mentioned in the ticket, but had to increase the size to 50000 to support our number of realms. Having this size hardcoded in the Java code becomes a bit inconvenient with the latest versions. I didn't really fix the code as Stian mentioned that this part is being heavily reworked. I would guess this has something to do with http://lists.jboss.org/pipermail/keycloak-dev/2016-November/008397.html? https://github.com/glavoie/keycloak/commit/6e3e7e8400da3440a743941980bad3... Thank you! Gabriel -- Gabriel Lavoie glavoie(a)gmail.com