[keycloak-dev] Potential performance regression between 3.4.3.Final and 4.3.0.Final

[Thomas Darimont]

Hello,

FYI I created https://issues.jboss.org/browse/KEYCLOAK-8150

Cheers,
Thomas

Am Mo., 27. Aug. 2018 um 13:33 Uhr schrieb Stian Thorgersen <
sthorger at redhat.com>:

> Can you open a bug for this in JIRA please and include as much details as
> you can.
>
> On Thu, 23 Aug 2018 at 15:45, Thomas Darimont <
> thomas.darimont at googlemail.com> wrote:
>
>> Hi Sebastian,
>>
>> thanks for investigating! Unfortunately, I didn't have time yet to look
>> further into the problem, but I also observed the slow attributes handling.
>> Btw. we have around 100k users and ~700k attributes. The
>> Entity-Attribute-Value data model takes it's toll here...
>>
>> @Stian so far I've only observed this when scrolling through the user
>> overview in the admin-console, which also triggers
>> the mentioned spike in heap usage.
>>
>> Cheers,
>> Thomas
>>
>> Am Do., 23. Aug. 2018 um 15:22 Uhr schrieb Schuster Sebastian (INST/ESY1)
>> <Sebastian.Schuster at bosch-si.com>:
>>
>>> Btw. we also have 40K user attributes...
>>>
>>> Best regards,
>>> Sebastian
>>>
>>> Mit freundlichen Grüßen / Best regards
>>>
>>> Dr.-Ing.  Sebastian Schuster
>>>
>>> Engineering and Support (INST/ESY1)
>>> Bosch Software Innovations GmbH | Ullsteinstr. 128 | 12109 Berlin |
>>> GERMANY | www.bosch-si.com
>>> Tel. +49 30 726112-485 | Fax +49 30 726112-100 |
>>> Sebastian.Schuster at bosch-si.com
>>>
>>> Sitz: Berlin, Registergericht: Amtsgericht Charlottenburg; HRB 148411 B
>>> Aufsichtsratsvorsitzender: Dr.-Ing. Thorsten Lücke; Geschäftsführung:
>>> Dr. Stefan Ferber, Michael Hahn
>>>
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: keycloak-dev-bounces at lists.jboss.org <
>>> keycloak-dev-bounces at lists.jboss.org> On Behalf Of Stian Thorgersen
>>> Sent: Mittwoch, 22. August 2018 21:25
>>> To: Thomas Darimont <thomas.darimont at googlemail.com>
>>> Cc: keycloak-dev <keycloak-dev at lists.jboss.org>
>>> Subject: Re: [keycloak-dev] Potential performance regression between
>>> 3.4.3.Final and 4.3.0.Final
>>>
>>> That's quite worrying. Is it limited to browsing users through the admin
>>> console or are you seeing bad behaviour elsewhere? Same question applies to
>>> the heap. My best bet here is that it has that something has changed around
>>> user querying/caching.
>>>
>>> On Mon, 20 Aug 2018 at 19:18, Thomas Darimont <
>>> thomas.darimont at googlemail.com> wrote:
>>>
>>> > Hello Keycloak team,
>>> >
>>> > has anyone encountered some performance issues after upgrading 3.4.3
>>> > to 4.x (4.3.0)?
>>> >
>>> > Today I noticed a performance regression while preparing an upgrade
>>> > from Keycloak 3.4.3.Final to 4.3.0.Final in our staging environment.
>>> >
>>> > In our test environment, we have around ~100k test users stored in a
>>> > postgres-backed database. When we started the server with the new
>>> > Keycloak version, the migration went through, and everything looked
>>> > fine at first, but when we tried to browse the list of users via the
>>> > admin-console, we noticed that the CPU and memory consumption of the
>>> > server increased significantly, up to a point where Keycloak crashed
>>> with an OOME.
>>> >
>>> > All previous Keycloak versions including 3.4.3 were very modest with
>>> > their memory requirements and quite happy with ~1g heap.
>>> > However, that seems to have changed in Keycloak 4.3.0 - there we
>>> > needed at least 4g to prevent Keycloak from crashing with an OOME.
>>> > Furthermore, we noticed that the response times for browsing the
>>> > paginated user view increased significantly as well:
>>> > In Keycloak 3.4.3 the average time to load a user page is ~80ms. In
>>> > Keycloak 4.3.0 (and older versions >= 4.0.0.Beta1) the same operation
>>> > takes
>>> > ~7 seconds for a test realm with just 10k users.
>>> >
>>> > In the test realm with 100k users, the time to load a single page in
>>> > the users listing was 66 seconds for version 4.3.0, on average -
>>> > compared to quite stable 80ms in 3.4.3.
>>> >
>>> > The database query that is executed by Keycloak 4.3.0 runs in ~1.5
>>> > seconds for 100k users, so I assume the processing logic in Keycloak
>>> > is the culprit.
>>> >
>>> > The problem of long load-times can be reproduced with the Keycloak
>>> > docker images and the in-memory database. I also created a small
>>> > example project that creates some users with just a few attributes in
>>> > a docker based 3.4.3 and 4.3.0 Keycloak environment with a Postgres
>>> > database to reproduce the problem.
>>> > https://github.com/thomasdarimont/kc-user-regression-tester
>>> >
>>> > Cheers,
>>> > Thomas
>>> > _______________________________________________
>>> > keycloak-dev mailing list
>>> > keycloak-dev at lists.jboss.org
>>> > https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>> >
>>> _______________________________________________
>>> keycloak-dev mailing list
>>> keycloak-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>
>>