[keycloak-dev] Performance improvements with large number of realms

[Hynek Mlnarik]

Hi,

thank you for you changes. Could you please turn them into PRs so that we
can review and discuss them individually? I'm currently in favor of merging
changes 2 (the new version) and 3. The rest of changes need more time for
review.

Thanks

--Hynek

On Wed, Nov 23, 2016 at 2:35 PM, Gabriel Lavoie <glavoie at gmail.com> wrote:

> Hi,
>      I received some feedback on the Liquibase changeset for the missing
> indexes. Any comment on the other suggested changes?
>
> Thank you,
>
> Gabriel
>
> 2016-11-16 19:43 GMT-05:00 Gabriel Lavoie <glavoie at gmail.com>:
>
> > Hi,
> >      following http://lists.jboss.org/pipermail/keycloak-user/
> > 2016-October/008032.html that was discussed in the keycloak-user mailing
> > list, I've been working in the last few weeks on fixing performance
> issues
> > for Keycloak setup with a very large number of realms (tested with
> > 500-600). As suggested by Stian, I have changes to submit as pull
> requests
> > that I want to discuss here first to make sure they are acceptable.
> >
> > With this improvements, the following issues were solved in our setup:
> > - Startup time reduced from more than 15 minute to under a minute.
> > - Login time to the admin console reduced from ~5 minutes to ~20 secs
> when
> > cache is being filled, to under 5 when cache is filled.
> > - Realm creation time reduced from ~1 minute to under 10 secs.
> >
> > 1 - Too many autoFlush checks by Hibernate and explicit em.flush():
> > From what I see, the JPA entity model is very disconnected and many
> > entities are retrieved through their ID using NamedQuery. I'm not sure
> why
> > the mapping was done this way.
> >
> > This causes Hibernate to make a autoFlush check for dirty entities on
> > every query, which is very CPU intensive. There are a few entities that
> can
> > benefit from having a Collection/Set to their child entities and use it
> > rather than a query to retrieve the required information. Also, with
> > explicit mapping, I foresee multiple explicit em.flush() that could be
> > removed when changing the data.
> >
> > In this changeset I've covered only what was necessary for us, but there
> > are still a few areas that could be improved in my opinion:
> >
> > https://github.com/glavoie/keycloak/commit/
> 93c8056498e38ed075523f5b9b78db
> > de2814d783
> >
> >
> > 2 - Many missing index for FKs:
> > We are using an Oracle database and indexes are not created automatically
> > for FKs. I've added many but I'm not 100% sure if you want a new
> Liquibase
> > changeset for every changes that could go in a version or only one per
> > version. I still have one index to fix so don't take this as-is for now:
> >
> > https://github.com/glavoie/keycloak/commit/
> 7bb44b7dfeea31e663695218524a0d
> > a4ce48a331
> >
> > 3 - Using Set rather than Collection with JPA entities:
> > When using a Collection, Hibernate will delete all rows and re-insert
> them
> > in the join table to update the content. This behavior was very slow and
> > visible with the "admin" composite role that points to roles for all
> realms
> > (through the different clients). This was slowing down a lot new realm
> > creation. When using a Set, Hibernate does a better job at tracking
> changes.
> >
> > https://github.com/glavoie/keycloak/commit/
> 9eb1bf013e80395affeec09625d5ee
> > 6157a7084a
> >
> >
> > 4 - "session" cache when testing for "hasRole" with a composite role:
> > The recursive search done KeycloakModelUtils.searchFor() can generate
> > quite a lot of hits on the Infinispan cache when trying to test for role.
> > With 500 realms, we saw up to 1.5 millions hits on the realmsRevision
> cache
> > just to login to the admin console. I've added a in-memory cache of the
> > retrieved composite RoleModel list in the Infinispan RoleAdapter. As I
> > understand, every instance of a RoleAdapter is bound to a KeycloakSession
> > that can be seen as a transaction/request. Still relying on the delegate
> > object if the RealmAdapter is being invalidated.
> >
> > This is the change that reduced the most the login time to the admin
> > console or the time to obtain a Bearer token for the admin API:
> >
> > https://github.com/glavoie/keycloak/commit/
> 0ab39670525315148761627e7d0f80
> > 8d4ef9c3c0
> >
> >
> > 5 - realmRevisions cache size:
> > In https://issues.jboss.org/browse/KEYCLOAK-3202, the realmRevisions
> > cache creation was moved from standalone.xml to Java code. With our
> current
> > Keycloak cluster we already added the eviction policy mentioned in the
> > ticket, but had to increase the size to 50000 to support our number of
> > realms. Having this size hardcoded in the Java code becomes a bit
> > inconvenient with the latest versions.
> >
> > I didn't really fix the code as Stian mentioned that this part is being
> > heavily reworked. I would guess this has something to do with
> > http://lists.jboss.org/pipermail/keycloak-dev/2016-November/008397.html?
> >
> > https://github.com/glavoie/keycloak/commit/
> 6e3e7e8400da3440a743941980bad3
> > dc83be7965
> >
> >
> > Thank you!
> >
> > Gabriel
> >
> > --
> > Gabriel Lavoie
> > glavoie at gmail.com
> >
>
>
>
> --
> Gabriel Lavoie
> glavoie at gmail.com
> _______________________________________________
> keycloak-dev mailing list
> keycloak-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>



-- 

--Hynek