I am only testing creating users. With the user cache disabled, I see no
evidence of a memory leak at many JVM heap-size settings.
Things get interesting when re-enabling the user cache. Performance seems
to take a major hit in lower-memory scenarios, with some very worrisome
scenarios where the rate at which I ingest continuously decreases.
I am able to create users at a high rate of speed with a max heap size of
4g or 8g. Interestingly, Keycloak seems to prefer an Xms setting lower than
the Xmx setting, which goes against official JBoss GC performance tuning
documentation
<
http://www.mastertheboss.com/jboss-server/jboss-performance/jboss-perform...
.
Though speeds are high with a large heap, I *do* see the possibility of a
memory leak with the user cache enabled and no authentication happening. I
will be running a test overnight to attempt to confirm or deny.
Thanks for your help so far, Stian. I will be on vacation tomorrow through
Sunday, so I will pick this back up on Monday.
P.S. I will be thinking of ways to better the documentation around
performance tuning, as my tests indicate that standard JVM options cause
Keycloak to run in a very sub-optimal state.
On Wed, Jun 22, 2016 at 11:59 PM Stian Thorgersen <sthorger(a)redhat.com>
wrote:
Are you only creating users or are you also authenticating users?
User
sessions are kept purely in memory so obviously the more you create the
more memory is used. Only creating users should not continue to increase,
but will do so for a while at least due to the way Java garbage collection
works.
I would only have the user cache disabled for testing memory leak.
Re-enable it and retest with it before you eventually go into production as
it will have a significant impact on performance.
On 23 June 2016 at 01:10, Chris Hairfield <chairfield(a)gmail.com> wrote:
> Scratch the results of the graph I posted. I was running the test
> incorrectly. I'll post back with the results of the test run properly.
>
> On Wed, Jun 22, 2016 at 12:38 PM Chris Hairfield <chairfield(a)gmail.com>
> wrote:
>
>> Thomas, this test is run with whatever local database Keycloak defaults
>> to. We're using Postgres generally, and we will have more information
>> pertaining to tests against Postgres soon.
>>
>> Stian, thanks for the tips. I am currently running a test to ingest
>> about 50m users into the default database with the user cache disabled, 8gb
>> mem (Xmx and Xms), and parallel GC threads == processor count.
>>
>> Though my test is young (430k users ingested), I'm noticing memory
>> allocation increasing in lockstep with the number of ingested users. Is it
>> expected to continue in this fashion, or is Keycloak designed to level off
>> in its memory usage?
>>
>> [image: increasing-heap.png]
>>
>>
>> On Tue, Jun 21, 2016 at 4:29 PM Stian Thorgersen <sthorger(a)redhat.com>
>> wrote:
>>
>>> Keycloak by default caches users in-memory, by default it will keep up
>>> to 10000 entries cached. You can verify that there's no leak by
disabling
>>> the user cache provider. See
>>>
http://keycloak.github.io/docs/userguide/keycloak-server/html/server_cach...
>>>
>>> If you're planning on having millions of users I suggest you increase
>>> the allocated memory for the JVM (512MB which it seems you have is not
>>> sufficient).
>>>
>>> On 22 June 2016 at 00:29, Chris Hairfield <chairfield(a)gmail.com>
wrote:
>>>
>>>> Hello,
>>>>
>>>> When testing Keycloak 1.9.8 by ingesting a few million users, we find
>>>> that Keycloak leaks memory until it is rendered unresponsive (see
graph).
>>>> Increasing JVM memory only increases the time it takes to encounter this
>>>> issue.
>>>>
>>>> We have put together a test project here
>>>> <
https://github.com/anaerobic/keycloak-leakage> and opened an
issue
>>>> here <
https://issues.jboss.org/browse/KEYCLOAK-3146> as we
continue
>>>> to investigate. As we are relying on Keycloak as a central
infrastructural
>>>> component, any help would be greatly appreciated.
>>>>
>>>> We'll update with more information as we find it.
>>>>
>>>> Thanks,
>>>> Chris
>>>>
>>>> [image: mem-cpu.png]
>>>>
>>>> _______________________________________________
>>>> keycloak-user mailing list
>>>> keycloak-user(a)lists.jboss.org
>>>>
https://lists.jboss.org/mailman/listinfo/keycloak-user
>>>>
>>>