[keycloak-user] Major Memory Leak/Consumption When Creating Many Users

Mon Jun 27 12:03:36 EDT 2016

One more important note: that memory cannot be GC'd even *after* I manually
clear the user and realm caches.

To be specific, upon adding 71k users, I noticed an extra 50mb was
allocated. Clearing the user and realm caches allowed me to GC 3mb of that.
The only way I know how to clear the remaining 47mb is by restarting
Keycloak.

On Mon, Jun 27, 2016 at 9:44 AM Chris Hairfield <chairfield at gmail.com>
wrote:

> Stian,
>
> I've noticed some interesting things:
>
>    - When the user cache in enabled, creating a user allocates a block on
>    memory that cannot be GC'd
>    - When I restart Keycloak after creating those users, that memory is
>    not re-allocated
>
> This begs the question: what is the purpose of the memory allocation on
> user creation if it doesn't need to stick around upon restart?
>
> It seems to me that the user cache is keeping more than 10k users on user
> creation. Does this hypothesis agree with your understanding of Keycloak's
> design?
>
> Thanks,
> Chris
>
> P.S. I have yet to test authentication load.
>
> On Thu, Jun 23, 2016 at 10:52 PM Chris Hairfield <chairfield at gmail.com>
> wrote:
>
>> There seems to be a leak when enabling the user cache, even when
>> providing Keycloak with all the memory it needs to perform admirably
>> (initially). My heap drops to 3.4 GB when I perform a manual GC.
>>
>> I'm now seeing a number of PessimisticLockExceptions as it fails to lock
>> the USER_ENTITY table. I expect Keycloak to have ground to a halt by the
>> morning.[image: test8-graph.png]
>>
>>
>> On Thu, Jun 23, 2016 at 4:52 PM Chris Hairfield <chairfield at gmail.com>
>> wrote:
>>
>>> I am only testing creating users. With the user cache disabled, I see no
>>> evidence of a memory leak at many JVM heap-size settings.
>>>
>>> Things get interesting when re-enabling the user cache. Performance
>>> seems to take a major hit in lower-memory scenarios, with some very
>>> worrisome scenarios where the rate at which I ingest continuously decreases.
>>>
>>> I am able to create users at a high rate of speed with a max heap size
>>> of 4g or 8g. Interestingly, Keycloak seems to prefer an Xms setting lower
>>> than the Xmx setting, which goes against official JBoss GC performance
>>> tuning documentation
>>> <http://www.mastertheboss.com/jboss-server/jboss-performance/jboss-performance-tuning-part-1>
>>> .
>>>
>>> Though speeds are high with a large heap, I *do* see the possibility of
>>> a memory leak with the user cache enabled and no authentication happening.
>>> I will be running a test overnight to attempt to confirm or deny.
>>>
>>> Thanks for your help so far, Stian. I will be on vacation tomorrow
>>> through Sunday, so I will pick this back up on Monday.
>>>
>>> P.S. I will be thinking of ways to better the documentation around
>>> performance tuning, as my tests indicate that standard JVM options cause
>>> Keycloak to run in a very sub-optimal state.
>>>
>>> On Wed, Jun 22, 2016 at 11:59 PM Stian Thorgersen <sthorger at redhat.com>
>>> wrote:
>>>
>>>> Are you only creating users or are you also authenticating users? User
>>>> sessions are kept purely in memory so obviously the more you create the
>>>> more memory is used. Only creating users should not continue to increase,
>>>> but will do so for a while at least due to the way Java garbage collection
>>>> works.
>>>>
>>>> I would only have the user cache disabled for testing memory leak.
>>>> Re-enable it and retest with it before you eventually go into production as
>>>> it will have a significant impact on performance.
>>>>
>>>> On 23 June 2016 at 01:10, Chris Hairfield <chairfield at gmail.com> wrote:
>>>>
>>>>> Scratch the results of the graph I posted. I was running the test
>>>>> incorrectly. I'll post back with the results of the test run properly.
>>>>>
>>>>> On Wed, Jun 22, 2016 at 12:38 PM Chris Hairfield <chairfield at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Thomas, this test is run with whatever local database Keycloak
>>>>>> defaults to. We're using Postgres generally, and we will have more
>>>>>> information pertaining to tests against Postgres soon.
>>>>>>
>>>>>> Stian, thanks for the tips. I am currently running a test to ingest
>>>>>> about 50m users into the default database with the user cache disabled, 8gb
>>>>>> mem (Xmx and Xms), and parallel GC threads == processor count.
>>>>>>
>>>>>> Though my test is young (430k users ingested), I'm noticing memory
>>>>>> allocation increasing in lockstep with the number of ingested users. Is it
>>>>>> expected to continue in this fashion, or is Keycloak designed to level off
>>>>>> in its memory usage?
>>>>>>
>>>>>> [image: increasing-heap.png]
>>>>>>
>>>>>>
>>>>>> On Tue, Jun 21, 2016 at 4:29 PM Stian Thorgersen <sthorger at redhat.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Keycloak by default caches users in-memory, by default it will keep
>>>>>>> up to 10000 entries cached. You can verify that there's no leak by
>>>>>>> disabling the user cache provider. See
>>>>>>> http://keycloak.github.io/docs/userguide/keycloak-server/html/server_cache.html#d4e3187
>>>>>>>
>>>>>>> If you're planning on having millions of users I suggest you
>>>>>>> increase the allocated memory for the JVM (512MB which it seems you have is
>>>>>>> not sufficient).
>>>>>>>
>>>>>>> On 22 June 2016 at 00:29, Chris Hairfield <chairfield at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> When testing Keycloak 1.9.8 by ingesting a few million users, we
>>>>>>>> find that Keycloak leaks memory until it is rendered unresponsive (see
>>>>>>>> graph). Increasing JVM memory only increases the time it takes to encounter
>>>>>>>> this issue.
>>>>>>>>
>>>>>>>> We have put together a test project here
>>>>>>>> <https://github.com/anaerobic/keycloak-leakage> and opened an
>>>>>>>> issue here <https://issues.jboss.org/browse/KEYCLOAK-3146> as we
>>>>>>>> continue to investigate. As we are relying on Keycloak as a central
>>>>>>>> infrastructural component, any help would be greatly appreciated.
>>>>>>>>
>>>>>>>> We'll update with more information as we find it.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Chris
>>>>>>>>
>>>>>>>> [image: mem-cpu.png]
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> keycloak-user mailing list
>>>>>>>> keycloak-user at lists.jboss.org
>>>>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-user
>>>>>>>>
>>>>>>>
>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/keycloak-user/attachments/20160627/0421952a/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mem-cpu.png
Type: image/png
Size: 100050 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/keycloak-user/attachments/20160627/0421952a/attachment-0003.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: increasing-heap.png
Type: image/png
Size: 11436 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/keycloak-user/attachments/20160627/0421952a/attachment-0004.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test8-graph.png
Type: image/png
Size: 46063 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/keycloak-user/attachments/20160627/0421952a/attachment-0005.png 