[keycloak-user] Major Memory Leak/Consumption When Creating Many Users

[Chris Hairfield]

I am only testing creating users. With the user cache disabled, I see no
evidence of a memory leak at many JVM heap-size settings.

Things get interesting when re-enabling the user cache. Performance seems
to take a major hit in lower-memory scenarios, with some very worrisome
scenarios where the rate at which I ingest continuously decreases.

I am able to create users at a high rate of speed with a max heap size of
4g or 8g. Interestingly, Keycloak seems to prefer an Xms setting lower than
the Xmx setting, which goes against official JBoss GC performance tuning
documentation
<http://www.mastertheboss.com/jboss-server/jboss-performance/jboss-performance-tuning-part-1>
.

Though speeds are high with a large heap, I *do* see the possibility of a
memory leak with the user cache enabled and no authentication happening. I
will be running a test overnight to attempt to confirm or deny.

Thanks for your help so far, Stian. I will be on vacation tomorrow through
Sunday, so I will pick this back up on Monday.

P.S. I will be thinking of ways to better the documentation around
performance tuning, as my tests indicate that standard JVM options cause
Keycloak to run in a very sub-optimal state.

On Wed, Jun 22, 2016 at 11:59 PM Stian Thorgersen <sthorger at redhat.com>
wrote:

> Are you only creating users or are you also authenticating users? User
> sessions are kept purely in memory so obviously the more you create the
> more memory is used. Only creating users should not continue to increase,
> but will do so for a while at least due to the way Java garbage collection
> works.
>
> I would only have the user cache disabled for testing memory leak.
> Re-enable it and retest with it before you eventually go into production as
> it will have a significant impact on performance.
>
> On 23 June 2016 at 01:10, Chris Hairfield <chairfield at gmail.com> wrote:
>
>> Scratch the results of the graph I posted. I was running the test
>> incorrectly. I'll post back with the results of the test run properly.
>>
>> On Wed, Jun 22, 2016 at 12:38 PM Chris Hairfield <chairfield at gmail.com>
>> wrote:
>>
>>> Thomas, this test is run with whatever local database Keycloak defaults
>>> to. We're using Postgres generally, and we will have more information
>>> pertaining to tests against Postgres soon.
>>>
>>> Stian, thanks for the tips. I am currently running a test to ingest
>>> about 50m users into the default database with the user cache disabled, 8gb
>>> mem (Xmx and Xms), and parallel GC threads == processor count.
>>>
>>> Though my test is young (430k users ingested), I'm noticing memory
>>> allocation increasing in lockstep with the number of ingested users. Is it
>>> expected to continue in this fashion, or is Keycloak designed to level off
>>> in its memory usage?
>>>
>>> [image: increasing-heap.png]
>>>
>>>
>>> On Tue, Jun 21, 2016 at 4:29 PM Stian Thorgersen <sthorger at redhat.com>
>>> wrote:
>>>
>>>> Keycloak by default caches users in-memory, by default it will keep up
>>>> to 10000 entries cached. You can verify that there's no leak by disabling
>>>> the user cache provider. See
>>>> http://keycloak.github.io/docs/userguide/keycloak-server/html/server_cache.html#d4e3187
>>>>
>>>> If you're planning on having millions of users I suggest you increase
>>>> the allocated memory for the JVM (512MB which it seems you have is not
>>>> sufficient).
>>>>
>>>> On 22 June 2016 at 00:29, Chris Hairfield <chairfield at gmail.com> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> When testing Keycloak 1.9.8 by ingesting a few million users, we find
>>>>> that Keycloak leaks memory until it is rendered unresponsive (see graph).
>>>>> Increasing JVM memory only increases the time it takes to encounter this
>>>>> issue.
>>>>>
>>>>> We have put together a test project here
>>>>> <https://github.com/anaerobic/keycloak-leakage> and opened an issue
>>>>> here <https://issues.jboss.org/browse/KEYCLOAK-3146> as we continue
>>>>> to investigate. As we are relying on Keycloak as a central infrastructural
>>>>> component, any help would be greatly appreciated.
>>>>>
>>>>> We'll update with more information as we find it.
>>>>>
>>>>> Thanks,
>>>>> Chris
>>>>>
>>>>> [image: mem-cpu.png]
>>>>>
>>>>> _______________________________________________
>>>>> keycloak-user mailing list
>>>>> keycloak-user at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-user
>>>>>
>>>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/keycloak-user/attachments/20160623/367b13ae/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mem-cpu.png
Type: image/png
Size: 100050 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/keycloak-user/attachments/20160623/367b13ae/attachment-0002.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: increasing-heap.png
Type: image/png
Size: 11436 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/keycloak-user/attachments/20160623/367b13ae/attachment-0003.png