[keycloak-user] Major Memory Leak/Consumption When Creating Many Users

[Chris Hairfield]

There seems to be a leak when enabling the user cache, even when providing
Keycloak with all the memory it needs to perform admirably (initially). My
heap drops to 3.4 GB when I perform a manual GC.

I'm now seeing a number of PessimisticLockExceptions as it fails to lock
the USER_ENTITY table. I expect Keycloak to have ground to a halt by the
morning.[image: test8-graph.png]

On Thu, Jun 23, 2016 at 4:52 PM Chris Hairfield <chairfield at gmail.com>
wrote:

> I am only testing creating users. With the user cache disabled, I see no
> evidence of a memory leak at many JVM heap-size settings.
>
> Things get interesting when re-enabling the user cache. Performance seems
> to take a major hit in lower-memory scenarios, with some very worrisome
> scenarios where the rate at which I ingest continuously decreases.
>
> I am able to create users at a high rate of speed with a max heap size of
> 4g or 8g. Interestingly, Keycloak seems to prefer an Xms setting lower than
> the Xmx setting, which goes against official JBoss GC performance tuning
> documentation
> <http://www.mastertheboss.com/jboss-server/jboss-performance/jboss-performance-tuning-part-1>
> .
>
> Though speeds are high with a large heap, I *do* see the possibility of a
> memory leak with the user cache enabled and no authentication happening. I
> will be running a test overnight to attempt to confirm or deny.
>
> Thanks for your help so far, Stian. I will be on vacation tomorrow through
> Sunday, so I will pick this back up on Monday.
>
> P.S. I will be thinking of ways to better the documentation around
> performance tuning, as my tests indicate that standard JVM options cause
> Keycloak to run in a very sub-optimal state.
>
> On Wed, Jun 22, 2016 at 11:59 PM Stian Thorgersen <sthorger at redhat.com>
> wrote:
>
>> Are you only creating users or are you also authenticating users? User
>> sessions are kept purely in memory so obviously the more you create the
>> more memory is used. Only creating users should not continue to increase,
>> but will do so for a while at least due to the way Java garbage collection
>> works.
>>
>> I would only have the user cache disabled for testing memory leak.
>> Re-enable it and retest with it before you eventually go into production as
>> it will have a significant impact on performance.
>>
>> On 23 June 2016 at 01:10, Chris Hairfield <chairfield at gmail.com> wrote:
>>
>>> Scratch the results of the graph I posted. I was running the test
>>> incorrectly. I'll post back with the results of the test run properly.
>>>
>>> On Wed, Jun 22, 2016 at 12:38 PM Chris Hairfield <chairfield at gmail.com>
>>> wrote:
>>>
>>>> Thomas, this test is run with whatever local database Keycloak defaults
>>>> to. We're using Postgres generally, and we will have more information
>>>> pertaining to tests against Postgres soon.
>>>>
>>>> Stian, thanks for the tips. I am currently running a test to ingest
>>>> about 50m users into the default database with the user cache disabled, 8gb
>>>> mem (Xmx and Xms), and parallel GC threads == processor count.
>>>>
>>>> Though my test is young (430k users ingested), I'm noticing memory
>>>> allocation increasing in lockstep with the number of ingested users. Is it
>>>> expected to continue in this fashion, or is Keycloak designed to level off
>>>> in its memory usage?
>>>>
>>>> [image: increasing-heap.png]
>>>>
>>>>
>>>> On Tue, Jun 21, 2016 at 4:29 PM Stian Thorgersen <sthorger at redhat.com>
>>>> wrote:
>>>>
>>>>> Keycloak by default caches users in-memory, by default it will keep up
>>>>> to 10000 entries cached. You can verify that there's no leak by disabling
>>>>> the user cache provider. See
>>>>> http://keycloak.github.io/docs/userguide/keycloak-server/html/server_cache.html#d4e3187
>>>>>
>>>>> If you're planning on having millions of users I suggest you increase
>>>>> the allocated memory for the JVM (512MB which it seems you have is not
>>>>> sufficient).
>>>>>
>>>>> On 22 June 2016 at 00:29, Chris Hairfield <chairfield at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> When testing Keycloak 1.9.8 by ingesting a few million users, we find
>>>>>> that Keycloak leaks memory until it is rendered unresponsive (see graph).
>>>>>> Increasing JVM memory only increases the time it takes to encounter this
>>>>>> issue.
>>>>>>
>>>>>> We have put together a test project here
>>>>>> <https://github.com/anaerobic/keycloak-leakage> and opened an issue
>>>>>> here <https://issues.jboss.org/browse/KEYCLOAK-3146> as we continue
>>>>>> to investigate. As we are relying on Keycloak as a central infrastructural
>>>>>> component, any help would be greatly appreciated.
>>>>>>
>>>>>> We'll update with more information as we find it.
>>>>>>
>>>>>> Thanks,
>>>>>> Chris
>>>>>>
>>>>>> [image: mem-cpu.png]
>>>>>>
>>>>>> _______________________________________________
>>>>>> keycloak-user mailing list
>>>>>> keycloak-user at lists.jboss.org
>>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-user
>>>>>>
>>>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/keycloak-user/attachments/20160624/2e7e215f/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mem-cpu.png
Type: image/png
Size: 100050 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/keycloak-user/attachments/20160624/2e7e215f/attachment-0003.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: increasing-heap.png
Type: image/png
Size: 11436 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/keycloak-user/attachments/20160624/2e7e215f/attachment-0004.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test8-graph.png
Type: image/png
Size: 46063 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/keycloak-user/attachments/20160624/2e7e215f/attachment-0005.png