[keycloak-user] Major Memory Leak/Consumption When Creating Many Users

Wed Jun 29 02:18:01 EDT 2016

I was worried that the workaround would have some impact on performance,
but would never have thought it would be anywhere near this bad.

We're still investigating a proper solution. Afraid it's not going to be
resolved for 2.0.0.Final, but most likely will be in 2.1.0.CR1.

On 29 June 2016 at 05:31, Chris Hairfield <chairfield at gmail.com> wrote:

> Stian,
>
> We don't seem to be out of the woods yet. The leak is resolved with your
> workaround, but performance suffers mightily when an eviction policy is
> set. Perhaps you have suggestions on how to get similar performance while
> employing this workaround?
>
> Details of test run:
>
>    - Eviction policy set at 1 million entries with LRU strategy for
>    realmVersions cache (no eviction policy on other caches)
>    - No statistics enabled (to simplify test)
>    - 1g Xms, 4g Xmx, parallel GC at 4 cores
>    - No authentication, only user creation
>
> Observations:
>
>    - Account creation speeds start out at about half of what I am able to
>    get without the workaround (~170/s rather than ~350/s)
>    - I have only been able to create 132k users over a span of ~6 hours
>    - Memory seems to thrash at a much higher rate than I've seen in
>    previous tests
>
> Thoughts?
>
> [image: test15-term2.png]
>
> On Tue, Jun 28, 2016 at 10:40 AM Chris Hairfield <chairfield at gmail.com>
> wrote:
>
>> I can confirm that adding the eviction policy clears the issue. Thank you
>> so much for your assistance, Stian. This clears the major hurdle in our
>> path to using Keycloak as our sole identity management store :)
>>
>> On Tue, Jun 28, 2016 at 2:13 AM Stian Thorgersen <sthorger at redhat.com>
>> wrote:
>>
>>> I've confirmed the issue now and found the cause:
>>> https://issues.jboss.org/browse/KEYCLOAK-3202
>>>
>>> The leak is in realmVersion cache. A workaround is to set a eviction
>>> policy on this cache, for example LRU 10000.
>>>
>>> On 28 June 216 at 08:47, Stian Thorgersen <sthorger at redhat.com> wrote:
>>>
>>>> We have had our QE guys confirm this issue and it looks like
>>>> something ominous is going on.
>>>>
>>>> On 27 June 2016 at 17:45, Chris Hairfield <chairfield at gmail.com> wrote:
>>>>
>>>>> Stian,
>>>>>
>>>>> I've noticed some interesting things:
>>>>>
>>>>>    - When the user cache in enabled, creating a user allocates a
>>>>>    block on memory that cannot be GC'd
>>>>>    - When I restart Keycloak after creating those users, that memory
>>>>>    is not re-allocated
>>>>>
>>>>> This begs the question: what is the purpose of the memory allocation
>>>>> on user creation if it doesn't need to stick around upon restart?
>>>>>
>>>>
>>>> Creating a new user is expected to consume memory that can't be GC'd
>>>> due to the caching. Upon restart the users are loaded on demand into the
>>>> cache (user logs in or through admin console/endpoints).
>>>>
>>>>
>>>>>
>>>>> It seems to me that the user cache is keeping more than 10k users on
>>>>> user creation. Does this hypothesis agree with your understanding of
>>>>> Keycloak's design?
>>>>>
>>>>
>>>> By default it should only keep 10K users in memory. Once it hits 10K
>>>> the least recently used should entries should be evicted from the cache.
>>>> However, it seems this is not working properly.
>>>>
>>>>
>>>>>
>>>>> Thanks,
>>>>> Chris
>>>>>
>>>>> P.S. I have yet to test authentication load.
>>>>>
>>>>> On Thu, Jun 23, 2016 at 10:52 PM Chris Hairfield <chairfield at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> There seems to be a leak when enabling the user cache, even when
>>>>>> providing Keycloak with all the memory it needs to perform admirably
>>>>>> (initially). My heap drops to 3.4 GB when I perform a manual GC.
>>>>>>
>>>>>> I'm now seeing a number of PessimisticLockExceptions as it fails to
>>>>>> lock the USER_ENTITY table. I expect Keycloak to have ground to a halt by
>>>>>> the morning.[image: test8-graph.png]
>>>>>>
>>>>>>
>>>>>> On Thu, Jun 23, 2016 at 4:52 PM Chris Hairfield <chairfield at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I am only testing creating users. With the user cache disabled, I
>>>>>>> see no evidence of a memory leak at many JVM heap-size settings.
>>>>>>>
>>>>>>> Things get interesting when re-enabling the user cache. Performance
>>>>>>> seems to take a major hit in lower-memory scenarios, with some very
>>>>>>> worrisome scenarios where the rate at which I ingest continuously decreases.
>>>>>>>
>>>>>>> I am able to create users at a high rate of speed with a max heap
>>>>>>> size of 4g or 8g. Interestingly, Keycloak seems to prefer an Xms setting
>>>>>>> lower than the Xmx setting, which goes against official JBoss GC
>>>>>>> performance tuning documentation
>>>>>>> <http://www.mastertheboss.com/jboss-server/jboss-performance/jboss-performance-tuning-part-1>
>>>>>>> .
>>>>>>>
>>>>>>> Though speeds are high with a large heap, I *do* see the
>>>>>>> possibility of a memory leak with the user cache enabled and no
>>>>>>> authentication happening. I will be running a test overnight to attempt to
>>>>>>> confirm or deny.
>>>>>>>
>>>>>>> Thanks for your help so far, Stian. I will be on vacation tomorrow
>>>>>>> through Sunday, so I will pick this back up on Monday.
>>>>>>>
>>>>>>> P.S. I will be thinking of ways to better the documentation around
>>>>>>> performance tuning, as my tests indicate that standard JVM options cause
>>>>>>> Keycloak to run in a very sub-optimal state.
>>>>>>>
>>>>>>> On Wed, Jun 22, 2016 at 11:59 PM Stian Thorgersen <
>>>>>>> sthorger at redhat.com> wrote:
>>>>>>>
>>>>>>>> Are you only creating users or are you also authenticating users?
>>>>>>>> User sessions are kept purely in memory so obviously the more you create
>>>>>>>> the more memory is used. Only creating users should not continue to
>>>>>>>> increase, but will do so for a while at least due to the way Java garbage
>>>>>>>> collection works.
>>>>>>>>
>>>>>>>> I would only have the user cache disabled for testing memory leak.
>>>>>>>> Re-enable it and retest with it before you eventually go into production as
>>>>>>>> it will have a significant impact on performance.
>>>>>>>>
>>>>>>>> On 23 June 2016 at 01:10, Chris Hairfield <chairfield at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Scratch the results of the graph I posted. I was running the test
>>>>>>>>> incorrectly. I'll post back with the results of the test run properly.
>>>>>>>>>
>>>>>>>>> On Wed, Jun 22, 2016 at 12:38 PM Chris Hairfield <
>>>>>>>>> chairfield at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Thomas, this test is run with whatever local database Keycloak
>>>>>>>>>> defaults to. We're using Postgres generally, and we will have more
>>>>>>>>>> information pertaining to tests against Postgres soon.
>>>>>>>>>>
>>>>>>>>>> Stian, thanks for the tips. I am currently running a test to
>>>>>>>>>> ingest about 50m users into the default database with the user cache
>>>>>>>>>> disabled, 8gb mem (Xmx and Xms), and parallel GC threads == processor count.
>>>>>>>>>>
>>>>>>>>>> Though my test is young (430k users ingested), I'm noticing
>>>>>>>>>> memory allocation increasing in lockstep with the number of ingested users.
>>>>>>>>>> Is it expected to continue in this fashion, or is Keycloak designed to
>>>>>>>>>> level off in its memory usage?
>>>>>>>>>>
>>>>>>>>>> [image: increasing-heap.png]
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Jun 21, 2016 at 4:29 PM Stian Thorgersen <
>>>>>>>>>> sthorger at redhat.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Keycloak by default caches users in-memory, by default it will
>>>>>>>>>>> keep up to 10000 entries cached. You can verify that there's no leak by
>>>>>>>>>>> disabling the user cache provider. See
>>>>>>>>>>> http://keycloak.github.io/docs/userguide/keycloak-server/html/server_cache.html#d4e3187
>>>>>>>>>>>
>>>>>>>>>>> If you're planning on having millions of users I suggest you
>>>>>>>>>>> increase the allocated memory for the JVM (512MB which it seems you have is
>>>>>>>>>>> not sufficient).
>>>>>>>>>>>
>>>>>>>>>>> On 22 June 2016 at 00:29, Chris Hairfield <chairfield at gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hello,
>>>>>>>>>>>>
>>>>>>>>>>>> When testing Keycloak 1.9.8 by ingesting a few million users,
>>>>>>>>>>>> we find that Keycloak leaks memory until it is rendered unresponsive (see
>>>>>>>>>>>> graph). Increasing JVM memory only increases the time it takes to encounter
>>>>>>>>>>>> this issue.
>>>>>>>>>>>>
>>>>>>>>>>>> We have put together a test project here
>>>>>>>>>>>> <https://github.com/anaerobic/keycloak-leakage> and opened an
>>>>>>>>>>>> issue here <https://issues.jboss.org/browse/KEYCLOAK-3146> as
>>>>>>>>>>>> we continue to investigate. As we are relying on Keycloak as a central
>>>>>>>>>>>> infrastructural component, any help would be greatly appreciated.
>>>>>>>>>>>>
>>>>>>>>>>>> We'll update with more information as we find it.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Chris
>>>>>>>>>>>>
>>>>>>>>>>>> [image: mem-cpu.png]
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> keycloak-user mailing list
>>>>>>>>>>>> keycloak-user at lists.jboss.org
>>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-user
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>
>>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/keycloak-user/attachments/20160629/9fcf5df6/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: increasing-heap.png
Type: image/png
Size: 11436 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/keycloak-user/attachments/20160629/9fcf5df6/attachment-0004.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test8-graph.png
Type: image/png
Size: 46063 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/keycloak-user/attachments/20160629/9fcf5df6/attachment-0005.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mem-cpu.png
Type: image/png
Size: 100050 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/keycloak-user/attachments/20160629/9fcf5df6/attachment-0006.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test15-term2.png
Type: image/png
Size: 25646 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/keycloak-user/attachments/20160629/9fcf5df6/attachment-0007.png 