[keycloak-user] Major Memory Leak/Consumption When Creating Many Users

Tue Jun 28 04:13:14 EDT 2016

I've confirmed the issue now and found the cause:
https://issues.jboss.org/browse/KEYCLOAK-3202

The leak is in realmVersion cache. A workaround is to set a eviction policy
on this cache, for example LRU 10000.

On 28 June 2016 at 08:47, Stian Thorgersen <sthorger at redhat.com> wrote:

> We have had our QE guys confirm this issue and it looks like
> something ominous is going on.
>
> On 27 June 2016 at 17:45, Chris Hairfield <chairfield at gmail.com> wrote:
>
>> Stian,
>>
>> I've noticed some interesting things:
>>
>>    - When the user cache in enabled, creating a user allocates a block
>>    on memory that cannot be GC'd
>>    - When I restart Keycloak after creating those users, that memory is
>>    not re-allocated
>>
>> This begs the question: what is the purpose of the memory allocation on
>> user creation if it doesn't need to stick around upon restart?
>>
>
> Creating a new user is expected to consume memory that can't be GC'd due
> to the caching. Upon restart the users are loaded on demand into the cache
> (user logs in or through admin console/endpoints).
>
>
>>
>> It seems to me that the user cache is keeping more than 10k users on user
>> creation. Does this hypothesis agree with your understanding of Keycloak's
>> design?
>>
>
> By default it should only keep 10K users in memory. Once it hits 10K the
> least recently used should entries should be evicted from the cache.
> However, it seems this is not working properly.
>
>
>>
>> Thanks,
>> Chris
>>
>> P.S. I have yet to test authentication load.
>>
>> On Thu, Jun 23, 2016 at 10:52 PM Chris Hairfield <chairfield at gmail.com>
>> wrote:
>>
>>> There seems to be a leak when enabling the user cache, even when
>>> providing Keycloak with all the memory it needs to perform admirably
>>> (initially). My heap drops to 3.4 GB when I perform a manual GC.
>>>
>>> I'm now seeing a number of PessimisticLockExceptions as it fails to lock
>>> the USER_ENTITY table. I expect Keycloak to have ground to a halt by the
>>> morning.[image: test8-graph.png]
>>>
>>>
>>> On Thu, Jun 23, 2016 at 4:52 PM Chris Hairfield <chairfield at gmail.com>
>>> wrote:
>>>
>>>> I am only testing creating users. With the user cache disabled, I see
>>>> no evidence of a memory leak at many JVM heap-size settings.
>>>>
>>>> Things get interesting when re-enabling the user cache. Performance
>>>> seems to take a major hit in lower-memory scenarios, with some very
>>>> worrisome scenarios where the rate at which I ingest continuously decreases.
>>>>
>>>> I am able to create users at a high rate of speed with a max heap size
>>>> of 4g or 8g. Interestingly, Keycloak seems to prefer an Xms setting lower
>>>> than the Xmx setting, which goes against official JBoss GC performance
>>>> tuning documentation
>>>> <http://www.mastertheboss.com/jboss-server/jboss-performance/jboss-performance-tuning-part-1>
>>>> .
>>>>
>>>> Though speeds are high with a large heap, I *do* see the possibility
>>>> of a memory leak with the user cache enabled and no authentication
>>>> happening. I will be running a test overnight to attempt to confirm or deny.
>>>>
>>>> Thanks for your help so far, Stian. I will be on vacation tomorrow
>>>> through Sunday, so I will pick this back up on Monday.
>>>>
>>>> P.S. I will be thinking of ways to better the documentation around
>>>> performance tuning, as my tests indicate that standard JVM options cause
>>>> Keycloak to run in a very sub-optimal state.
>>>>
>>>> On Wed, Jun 22, 2016 at 11:59 PM Stian Thorgersen <sthorger at redhat.com>
>>>> wrote:
>>>>
>>>>> Are you only creating users or are you also authenticating users? User
>>>>> sessions are kept purely in memory so obviously the more you create the
>>>>> more memory is used. Only creating users should not continue to increase,
>>>>> but will do so for a while at least due to the way Java garbage collection
>>>>> works.
>>>>>
>>>>> I would only have the user cache disabled for testing memory leak.
>>>>> Re-enable it and retest with it before you eventually go into production as
>>>>> it will have a significant impact on performance.
>>>>>
>>>>> On 23 June 2016 at 01:10, Chris Hairfield <chairfield at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Scratch the results of the graph I posted. I was running the test
>>>>>> incorrectly. I'll post back with the results of the test run properly.
>>>>>>
>>>>>> On Wed, Jun 22, 2016 at 12:38 PM Chris Hairfield <
>>>>>> chairfield at gmail.com> wrote:
>>>>>>
>>>>>>> Thomas, this test is run with whatever local database Keycloak
>>>>>>> defaults to. We're using Postgres generally, and we will have more
>>>>>>> information pertaining to tests against Postgres soon.
>>>>>>>
>>>>>>> Stian, thanks for the tips. I am currently running a test to ingest
>>>>>>> about 50m users into the default database with the user cache disabled, 8gb
>>>>>>> mem (Xmx and Xms), and parallel GC threads == processor count.
>>>>>>>
>>>>>>> Though my test is young (430k users ingested), I'm noticing memory
>>>>>>> allocation increasing in lockstep with the number of ingested users. Is it
>>>>>>> expected to continue in this fashion, or is Keycloak designed to level off
>>>>>>> in its memory usage?
>>>>>>>
>>>>>>> [image: increasing-heap.png]
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Jun 21, 2016 at 4:29 PM Stian Thorgersen <
>>>>>>> sthorger at redhat.com> wrote:
>>>>>>>
>>>>>>>> Keycloak by default caches users in-memory, by default it will keep
>>>>>>>> up to 10000 entries cached. You can verify that there's no leak by
>>>>>>>> disabling the user cache provider. See
>>>>>>>> http://keycloak.github.io/docs/userguide/keycloak-server/html/server_cache.html#d4e3187
>>>>>>>>
>>>>>>>> If you're planning on having millions of users I suggest you
>>>>>>>> increase the allocated memory for the JVM (512MB which it seems you have is
>>>>>>>> not sufficient).
>>>>>>>>
>>>>>>>> On 22 June 2016 at 00:29, Chris Hairfield <chairfield at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> When testing Keycloak 1.9.8 by ingesting a few million users, we
>>>>>>>>> find that Keycloak leaks memory until it is rendered unresponsive (see
>>>>>>>>> graph). Increasing JVM memory only increases the time it takes to encounter
>>>>>>>>> this issue.
>>>>>>>>>
>>>>>>>>> We have put together a test project here
>>>>>>>>> <https://github.com/anaerobic/keycloak-leakage> and opened an
>>>>>>>>> issue here <https://issues.jboss.org/browse/KEYCLOAK-3146> as we
>>>>>>>>> continue to investigate. As we are relying on Keycloak as a central
>>>>>>>>> infrastructural component, any help would be greatly appreciated.
>>>>>>>>>
>>>>>>>>> We'll update with more information as we find it.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Chris
>>>>>>>>>
>>>>>>>>> [image: mem-cpu.png]
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> keycloak-user mailing list
>>>>>>>>> keycloak-user at lists.jboss.org
>>>>>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-user
>>>>>>>>>
>>>>>>>>
>>>>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/keycloak-user/attachments/20160628/1590b4c4/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mem-cpu.png
Type: image/png
Size: 100050 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/keycloak-user/attachments/20160628/1590b4c4/attachment-0003.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: increasing-heap.png
Type: image/png
Size: 11436 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/keycloak-user/attachments/20160628/1590b4c4/attachment-0004.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test8-graph.png
Type: image/png
Size: 46063 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/keycloak-user/attachments/20160628/1590b4c4/attachment-0005.png 