Hello, When testing Keycloak 1.9.8 by ingesting a few million users, we find that Keycloak leaks memory until it is rendered unresponsive (see graph). Increasing JVM memory only increases the time it takes to encounter this issue. We have put together a test project here <https://github.com/anaerobic/keycloak-leakage> and opened an issue here <https://issues.jboss.org/browse/KEYCLOAK-3146> as we continue to investigate. As we are relying on Keycloak as a central infrastructural component, any help would be greatly appreciated. We'll update with more information as we find it. Thanks, Chris [image: mem-cpu.png] _______________________________________________ keycloak-user mailing list keycloak-user(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/keycloak-user

Thomas, this test is run with whatever local database Keycloak defaults to. We're using Postgres generally, and we will have more information pertaining to tests against Postgres soon. Stian, thanks for the tips. I am currently running a test to ingest about 50m users into the default database with the user cache disabled, 8gb mem (Xmx and Xms), and parallel GC threads == processor count. Though my test is young (430k users ingested), I'm noticing memory allocation increasing in lockstep with the number of ingested users. Is it expected to continue in this fashion, or is Keycloak designed to level off in its memory usage? [image: increasing-heap.png] On Tue, Jun 21, 2016 at 4:29 PM Stian Thorgersen <sthorger(a)redhat.com&gt; wrote: > Keycloak by default caches users in-memory, by default it will keep up to > 10000 entries cached. You can verify that there's no leak by disabling the > user cache provider. See > http://keycloak.github.io/docs/userguide/keycloak-server/html/server_cach... > > If you're planning on having millions of users I suggest you increase the > allocated memory for the JVM (512MB which it seems you have is not > sufficient). > > On 22 June 2016 at 00:29, Chris Hairfield <chairfield(a)gmail.com&gt; wrote: > >> Hello, >> >> When testing Keycloak 1.9.8 by ingesting a few million users, we find >> that Keycloak leaks memory until it is rendered unresponsive (see graph). >> Increasing JVM memory only increases the time it takes to encounter this >> issue. >> >> We have put together a test project here >> <https://github.com/anaerobic/keycloak-leakage> and opened an issue here >> <https://issues.jboss.org/browse/KEYCLOAK-3146> as we continue to >> investigate. As we are relying on Keycloak as a central infrastructural >> component, any help would be greatly appreciated. >> >> We'll update with more information as we find it. >> >> Thanks, >> Chris >> >> [image: mem-cpu.png] >> >> _______________________________________________ >> keycloak-user mailing list >> keycloak-user(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/keycloak-user >> >

Are you only creating users or are you also authenticating users? User sessions are kept purely in memory so obviously the more you create the more memory is used. Only creating users should not continue to increase, but will do so for a while at least due to the way Java garbage collection works. I would only have the user cache disabled for testing memory leak. Re-enable it and retest with it before you eventually go into production as it will have a significant impact on performance. On 23 June 2016 at 01:10, Chris Hairfield <chairfield(a)gmail.com&gt; wrote: > Scratch the results of the graph I posted. I was running the test > incorrectly. I'll post back with the results of the test run properly. > > On Wed, Jun 22, 2016 at 12:38 PM Chris Hairfield <chairfield(a)gmail.com&gt; > wrote: > >> Thomas, this test is run with whatever local database Keycloak defaults >> to. We're using Postgres generally, and we will have more information >> pertaining to tests against Postgres soon. >> >> Stian, thanks for the tips. I am currently running a test to ingest >> about 50m users into the default database with the user cache disabled, 8gb >> mem (Xmx and Xms), and parallel GC threads == processor count. >> >> Though my test is young (430k users ingested), I'm noticing memory >> allocation increasing in lockstep with the number of ingested users. Is it >> expected to continue in this fashion, or is Keycloak designed to level off >> in its memory usage? >> >> [image: increasing-heap.png] >> >> >> On Tue, Jun 21, 2016 at 4:29 PM Stian Thorgersen <sthorger(a)redhat.com&gt; >> wrote: >> >>> Keycloak by default caches users in-memory, by default it will keep up >>> to 10000 entries cached. You can verify that there's no leak by disabling >>> the user cache provider. See >>> http://keycloak.github.io/docs/userguide/keycloak-server/html/server_cach... >>> >>> If you're planning on having millions of users I suggest you increase >>> the allocated memory for the JVM (512MB which it seems you have is not >>> sufficient). >>> >>> On 22 June 2016 at 00:29, Chris Hairfield <chairfield(a)gmail.com&gt; wrote: >>> >>>> Hello, >>>> >>>> When testing Keycloak 1.9.8 by ingesting a few million users, we find >>>> that Keycloak leaks memory until it is rendered unresponsive (see graph). >>>> Increasing JVM memory only increases the time it takes to encounter this >>>> issue. >>>> >>>> We have put together a test project here >>>> <https://github.com/anaerobic/keycloak-leakage> and opened an issue >>>> here <https://issues.jboss.org/browse/KEYCLOAK-3146> as we continue >>>> to investigate. As we are relying on Keycloak as a central infrastructural >>>> component, any help would be greatly appreciated. >>>> >>>> We'll update with more information as we find it. >>>> >>>> Thanks, >>>> Chris >>>> >>>> [image: mem-cpu.png] >>>> >>>> _______________________________________________ >>>> keycloak-user mailing list >>>> keycloak-user(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/keycloak-user >>>> >>>

There seems to be a leak when enabling the user cache, even when providing Keycloak with all the memory it needs to perform admirably (initially). My heap drops to 3.4 GB when I perform a manual GC. I'm now seeing a number of PessimisticLockExceptions as it fails to lock the USER_ENTITY table. I expect Keycloak to have ground to a halt by the morning.[image: test8-graph.png] On Thu, Jun 23, 2016 at 4:52 PM Chris Hairfield <chairfield(a)gmail.com&gt; wrote: > I am only testing creating users. With the user cache disabled, I see no > evidence of a memory leak at many JVM heap-size settings. > > Things get interesting when re-enabling the user cache. Performance seems > to take a major hit in lower-memory scenarios, with some very worrisome > scenarios where the rate at which I ingest continuously decreases. > > I am able to create users at a high rate of speed with a max heap size of > 4g or 8g. Interestingly, Keycloak seems to prefer an Xms setting lower than > the Xmx setting, which goes against official JBoss GC performance tuning > documentation > <http://www.mastertheboss.com/jboss-server/jboss-performance/jboss-perform... > . > > Though speeds are high with a large heap, I *do* see the possibility of > a memory leak with the user cache enabled and no authentication happening. > I will be running a test overnight to attempt to confirm or deny. > > Thanks for your help so far, Stian. I will be on vacation tomorrow > through Sunday, so I will pick this back up on Monday. > > P.S. I will be thinking of ways to better the documentation around > performance tuning, as my tests indicate that standard JVM options cause > Keycloak to run in a very sub-optimal state. > > On Wed, Jun 22, 2016 at 11:59 PM Stian Thorgersen <sthorger(a)redhat.com&gt; > wrote: > >> Are you only creating users or are you also authenticating users? User >> sessions are kept purely in memory so obviously the more you create the >> more memory is used. Only creating users should not continue to increase, >> but will do so for a while at least due to the way Java garbage collection >> works. >> >> I would only have the user cache disabled for testing memory leak. >> Re-enable it and retest with it before you eventually go into production as >> it will have a significant impact on performance. >> >> On 23 June 2016 at 01:10, Chris Hairfield <chairfield(a)gmail.com&gt; wrote: >> >>> Scratch the results of the graph I posted. I was running the test >>> incorrectly. I'll post back with the results of the test run properly. >>> >>> On Wed, Jun 22, 2016 at 12:38 PM Chris Hairfield <chairfield(a)gmail.com&gt; >>> wrote: >>> >>>> Thomas, this test is run with whatever local database Keycloak >>>> defaults to. We're using Postgres generally, and we will have more >>>> information pertaining to tests against Postgres soon. >>>> >>>> Stian, thanks for the tips. I am currently running a test to ingest >>>> about 50m users into the default database with the user cache disabled, 8gb >>>> mem (Xmx and Xms), and parallel GC threads == processor count. >>>> >>>> Though my test is young (430k users ingested), I'm noticing memory >>>> allocation increasing in lockstep with the number of ingested users. Is it >>>> expected to continue in this fashion, or is Keycloak designed to level off >>>> in its memory usage? >>>> >>>> [image: increasing-heap.png] >>>> >>>> >>>> On Tue, Jun 21, 2016 at 4:29 PM Stian Thorgersen <sthorger(a)redhat.com&gt; >>>> wrote: >>>> >>>>> Keycloak by default caches users in-memory, by default it will keep >>>>> up to 10000 entries cached. You can verify that there's no leak by >>>>> disabling the user cache provider. See >>>>> http://keycloak.github.io/docs/userguide/keycloak-server/html/server_cach... >>>>> >>>>> If you're planning on having millions of users I suggest you increase >>>>> the allocated memory for the JVM (512MB which it seems you have is not >>>>> sufficient). >>>>> >>>>> On 22 June 2016 at 00:29, Chris Hairfield <chairfield(a)gmail.com&gt; >>>>> wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> When testing Keycloak 1.9.8 by ingesting a few million users, we >>>>>> find that Keycloak leaks memory until it is rendered unresponsive (see >>>>>> graph). Increasing JVM memory only increases the time it takes to encounter >>>>>> this issue. >>>>>> >>>>>> We have put together a test project here >>>>>> <https://github.com/anaerobic/keycloak-leakage> and opened an issue >>>>>> here <https://issues.jboss.org/browse/KEYCLOAK-3146> as we continue >>>>>> to investigate. As we are relying on Keycloak as a central infrastructural >>>>>> component, any help would be greatly appreciated. >>>>>> >>>>>> We'll update with more information as we find it. >>>>>> >>>>>> Thanks, >>>>>> Chris >>>>>> >>>>>> [image: mem-cpu.png] >>>>>> >>>>>> _______________________________________________ >>>>>> keycloak-user mailing list >>>>>> keycloak-user(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-user >>>>>> >>>>> >>

Stian, I've noticed some interesting things: - When the user cache in enabled, creating a user allocates a block on memory that cannot be GC'd - When I restart Keycloak after creating those users, that memory is not re-allocated This begs the question: what is the purpose of the memory allocation on user creation if it doesn't need to stick around upon restart?

It seems to me that the user cache is keeping more than 10k users on user creation. Does this hypothesis agree with your understanding of Keycloak's design?

Thanks, Chris P.S. I have yet to test authentication load. On Thu, Jun 23, 2016 at 10:52 PM Chris Hairfield <chairfield(a)gmail.com&gt; wrote: > There seems to be a leak when enabling the user cache, even when > providing Keycloak with all the memory it needs to perform admirably > (initially). My heap drops to 3.4 GB when I perform a manual GC. > > I'm now seeing a number of PessimisticLockExceptions as it fails to lock > the USER_ENTITY table. I expect Keycloak to have ground to a halt by the > morning.[image: test8-graph.png] > > > On Thu, Jun 23, 2016 at 4:52 PM Chris Hairfield <chairfield(a)gmail.com&gt; > wrote: > >> I am only testing creating users. With the user cache disabled, I see no >> evidence of a memory leak at many JVM heap-size settings. >> >> Things get interesting when re-enabling the user cache. Performance >> seems to take a major hit in lower-memory scenarios, with some very >> worrisome scenarios where the rate at which I ingest continuously decreases. >> >> I am able to create users at a high rate of speed with a max heap size >> of 4g or 8g. Interestingly, Keycloak seems to prefer an Xms setting lower >> than the Xmx setting, which goes against official JBoss GC performance >> tuning documentation >> <http://www.mastertheboss.com/jboss-server/jboss-performance/jboss-perform... >> . >> >> Though speeds are high with a large heap, I *do* see the possibility of >> a memory leak with the user cache enabled and no authentication happening. >> I will be running a test overnight to attempt to confirm or deny. >> >> Thanks for your help so far, Stian. I will be on vacation tomorrow >> through Sunday, so I will pick this back up on Monday. >> >> P.S. I will be thinking of ways to better the documentation around >> performance tuning, as my tests indicate that standard JVM options cause >> Keycloak to run in a very sub-optimal state. >> >> On Wed, Jun 22, 2016 at 11:59 PM Stian Thorgersen <sthorger(a)redhat.com&gt; >> wrote: >> >>> Are you only creating users or are you also authenticating users? User >>> sessions are kept purely in memory so obviously the more you create the >>> more memory is used. Only creating users should not continue to increase, >>> but will do so for a while at least due to the way Java garbage collection >>> works. >>> >>> I would only have the user cache disabled for testing memory leak. >>> Re-enable it and retest with it before you eventually go into production as >>> it will have a significant impact on performance. >>> >>> On 23 June 2016 at 01:10, Chris Hairfield <chairfield(a)gmail.com&gt; wrote: >>> >>>> Scratch the results of the graph I posted. I was running the test >>>> incorrectly. I'll post back with the results of the test run properly. >>>> >>>> On Wed, Jun 22, 2016 at 12:38 PM Chris Hairfield <chairfield(a)gmail.com&gt; >>>> wrote: >>>> >>>>> Thomas, this test is run with whatever local database Keycloak >>>>> defaults to. We're using Postgres generally, and we will have more >>>>> information pertaining to tests against Postgres soon. >>>>> >>>>> Stian, thanks for the tips. I am currently running a test to ingest >>>>> about 50m users into the default database with the user cache disabled, 8gb >>>>> mem (Xmx and Xms), and parallel GC threads == processor count. >>>>> >>>>> Though my test is young (430k users ingested), I'm noticing memory >>>>> allocation increasing in lockstep with the number of ingested users. Is it >>>>> expected to continue in this fashion, or is Keycloak designed to level off >>>>> in its memory usage? >>>>> >>>>> [image: increasing-heap.png] >>>>> >>>>> >>>>> On Tue, Jun 21, 2016 at 4:29 PM Stian Thorgersen <sthorger(a)redhat.com&gt; >>>>> wrote: >>>>> >>>>>> Keycloak by default caches users in-memory, by default it will keep >>>>>> up to 10000 entries cached. You can verify that there's no leak by >>>>>> disabling the user cache provider. See >>>>>> http://keycloak.github.io/docs/userguide/keycloak-server/html/server_cach... >>>>>> >>>>>> If you're planning on having millions of users I suggest you >>>>>> increase the allocated memory for the JVM (512MB which it seems you have is >>>>>> not sufficient). >>>>>> >>>>>> On 22 June 2016 at 00:29, Chris Hairfield <chairfield(a)gmail.com&gt; >>>>>> wrote: >>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> When testing Keycloak 1.9.8 by ingesting a few million users, we >>>>>>> find that Keycloak leaks memory until it is rendered unresponsive (see >>>>>>> graph). Increasing JVM memory only increases the time it takes to encounter >>>>>>> this issue. >>>>>>> >>>>>>> We have put together a test project here >>>>>>> <https://github.com/anaerobic/keycloak-leakage> and opened an >>>>>>> issue here <https://issues.jboss.org/browse/KEYCLOAK-3146> as we >>>>>>> continue to investigate. As we are relying on Keycloak as a central >>>>>>> infrastructural component, any help would be greatly appreciated. >>>>>>> >>>>>>> We'll update with more information as we find it. >>>>>>> >>>>>>> Thanks, >>>>>>> Chris >>>>>>> >>>>>>> [image: mem-cpu.png] >>>>>>> >>>>>>> _______________________________________________ >>>>>>> keycloak-user mailing list >>>>>>> keycloak-user(a)lists.jboss.org >>>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-user >>>>>>> >>>>>> >>>

I've confirmed the issue now and found the cause: https://issues.jboss.org/browse/KEYCLOAK-3202 The leak is in realmVersion cache. A workaround is to set a eviction policy on this cache, for example LRU 10000. On 28 June 2016 at 08:47, Stian Thorgersen <sthorger(a)redhat.com&gt; wrote: > We have had our QE guys confirm this issue and it looks like > something ominous is going on. > > On 27 June 2016 at 17:45, Chris Hairfield <chairfield(a)gmail.com&gt; wrote: > >> Stian, >> >> I've noticed some interesting things: >> >> - When the user cache in enabled, creating a user allocates a block >> on memory that cannot be GC'd >> - When I restart Keycloak after creating those users, that memory is >> not re-allocated >> >> This begs the question: what is the purpose of the memory allocation on >> user creation if it doesn't need to stick around upon restart? >> > > Creating a new user is expected to consume memory that can't be GC'd due > to the caching. Upon restart the users are loaded on demand into the cache > (user logs in or through admin console/endpoints). > > >> >> It seems to me that the user cache is keeping more than 10k users on >> user creation. Does this hypothesis agree with your understanding of >> Keycloak's design? >> > > By default it should only keep 10K users in memory. Once it hits 10K the > least recently used should entries should be evicted from the cache. > However, it seems this is not working properly. > > >> >> Thanks, >> Chris >> >> P.S. I have yet to test authentication load. >> >> On Thu, Jun 23, 2016 at 10:52 PM Chris Hairfield <chairfield(a)gmail.com&gt; >> wrote: >> >>> There seems to be a leak when enabling the user cache, even when >>> providing Keycloak with all the memory it needs to perform admirably >>> (initially). My heap drops to 3.4 GB when I perform a manual GC. >>> >>> I'm now seeing a number of PessimisticLockExceptions as it fails to >>> lock the USER_ENTITY table. I expect Keycloak to have ground to a halt by >>> the morning.[image: test8-graph.png] >>> >>> >>> On Thu, Jun 23, 2016 at 4:52 PM Chris Hairfield <chairfield(a)gmail.com&gt; >>> wrote: >>> >>>> I am only testing creating users. With the user cache disabled, I see >>>> no evidence of a memory leak at many JVM heap-size settings. >>>> >>>> Things get interesting when re-enabling the user cache. Performance >>>> seems to take a major hit in lower-memory scenarios, with some very >>>> worrisome scenarios where the rate at which I ingest continuously decreases. >>>> >>>> I am able to create users at a high rate of speed with a max heap size >>>> of 4g or 8g. Interestingly, Keycloak seems to prefer an Xms setting lower >>>> than the Xmx setting, which goes against official JBoss GC performance >>>> tuning documentation >>>> <http://www.mastertheboss.com/jboss-server/jboss-performance/jboss-perform... >>>> . >>>> >>>> Though speeds are high with a large heap, I *do* see the possibility >>>> of a memory leak with the user cache enabled and no authentication >>>> happening. I will be running a test overnight to attempt to confirm or deny. >>>> >>>> Thanks for your help so far, Stian. I will be on vacation tomorrow >>>> through Sunday, so I will pick this back up on Monday. >>>> >>>> P.S. I will be thinking of ways to better the documentation around >>>> performance tuning, as my tests indicate that standard JVM options cause >>>> Keycloak to run in a very sub-optimal state. >>>> >>>> On Wed, Jun 22, 2016 at 11:59 PM Stian Thorgersen <sthorger(a)redhat.com&gt; >>>> wrote: >>>> >>>>> Are you only creating users or are you also authenticating users? >>>>> User sessions are kept purely in memory so obviously the more you create >>>>> the more memory is used. Only creating users should not continue to >>>>> increase, but will do so for a while at least due to the way Java garbage >>>>> collection works. >>>>> >>>>> I would only have the user cache disabled for testing memory leak. >>>>> Re-enable it and retest with it before you eventually go into production as >>>>> it will have a significant impact on performance. >>>>> >>>>> On 23 June 2016 at 01:10, Chris Hairfield <chairfield(a)gmail.com&gt; >>>>> wrote: >>>>> >>>>>> Scratch the results of the graph I posted. I was running the test >>>>>> incorrectly. I'll post back with the results of the test run properly. >>>>>> >>>>>> On Wed, Jun 22, 2016 at 12:38 PM Chris Hairfield < >>>>>> chairfield(a)gmail.com&gt; wrote: >>>>>> >>>>>>> Thomas, this test is run with whatever local database Keycloak >>>>>>> defaults to. We're using Postgres generally, and we will have more >>>>>>> information pertaining to tests against Postgres soon. >>>>>>> >>>>>>> Stian, thanks for the tips. I am currently running a test to ingest >>>>>>> about 50m users into the default database with the user cache disabled, 8gb >>>>>>> mem (Xmx and Xms), and parallel GC threads == processor count. >>>>>>> >>>>>>> Though my test is young (430k users ingested), I'm noticing memory >>>>>>> allocation increasing in lockstep with the number of ingested users. Is it >>>>>>> expected to continue in this fashion, or is Keycloak designed to level off >>>>>>> in its memory usage? >>>>>>> >>>>>>> [image: increasing-heap.png] >>>>>>> >>>>>>> >>>>>>> On Tue, Jun 21, 2016 at 4:29 PM Stian Thorgersen < >>>>>>> sthorger(a)redhat.com&gt; wrote: >>>>>>> >>>>>>>> Keycloak by default caches users in-memory, by default it will >>>>>>>> keep up to 10000 entries cached. You can verify that there's no leak by >>>>>>>> disabling the user cache provider. See >>>>>>>> http://keycloak.github.io/docs/userguide/keycloak-server/html/server_cach... >>>>>>>> >>>>>>>> If you're planning on having millions of users I suggest you >>>>>>>> increase the allocated memory for the JVM (512MB which it seems you have is >>>>>>>> not sufficient). >>>>>>>> >>>>>>>> On 22 June 2016 at 00:29, Chris Hairfield <chairfield(a)gmail.com&gt; >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> When testing Keycloak 1.9.8 by ingesting a few million users, we >>>>>>>>> find that Keycloak leaks memory until it is rendered unresponsive (see >>>>>>>>> graph). Increasing JVM memory only increases the time it takes to encounter >>>>>>>>> this issue. >>>>>>>>> >>>>>>>>> We have put together a test project here >>>>>>>>> <https://github.com/anaerobic/keycloak-leakage> and opened an >>>>>>>>> issue here <https://issues.jboss.org/browse/KEYCLOAK-3146> as we >>>>>>>>> continue to investigate. As we are relying on Keycloak as a central >>>>>>>>> infrastructural component, any help would be greatly appreciated. >>>>>>>>> >>>>>>>>> We'll update with more information as we find it. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Chris >>>>>>>>> >>>>>>>>> [image: mem-cpu.png] >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> keycloak-user mailing list >>>>>>>>> keycloak-user(a)lists.jboss.org >>>>>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-user >>>>>>>>> >>>>>>>> >>>>> >

Stian, We don't seem to be out of the woods yet. The leak is resolved with your workaround, but performance suffers mightily when an eviction policy is set. Perhaps you have suggestions on how to get similar performance while employing this workaround? Details of test run: - Eviction policy set at 1 million entries with LRU strategy for realmVersions cache (no eviction policy on other caches) - No statistics enabled (to simplify test) - 1g Xms, 4g Xmx, parallel GC at 4 cores - No authentication, only user creation Observations: - Account creation speeds start out at about half of what I am able to get without the workaround (~170/s rather than ~350/s) - I have only been able to create 132k users over a span of ~6 hours - Memory seems to thrash at a much higher rate than I've seen in previous tests Thoughts? [image: test15-term2.png] On Tue, Jun 28, 2016 at 10:40 AM Chris Hairfield <chairfield(a)gmail.com&gt; wrote: > I can confirm that adding the eviction policy clears the issue. Thank you > so much for your assistance, Stian. This clears the major hurdle in our > path to using Keycloak as our sole identity management store :) > > On Tue, Jun 28, 2016 at 2:13 AM Stian Thorgersen <sthorger(a)redhat.com&gt; > wrote: > >> I've confirmed the issue now and found the cause: >> https://issues.jboss.org/browse/KEYCLOAK-3202 >> >> The leak is in realmVersion cache. A workaround is to set a eviction >> policy on this cache, for example LRU 10000. >> >> On 28 June 216 at 08:47, Stian Thorgersen <sthorger(a)redhat.com&gt; wrote: >> >>> We have had our QE guys confirm this issue and it looks like >>> something ominous is going on. >>> >>> On 27 June 2016 at 17:45, Chris Hairfield <chairfield(a)gmail.com&gt; wrote: >>> >>>> Stian, >>>> >>>> I've noticed some interesting things: >>>> >>>> - When the user cache in enabled, creating a user allocates a >>>> block on memory that cannot be GC'd >>>> - When I restart Keycloak after creating those users, that memory >>>> is not re-allocated >>>> >>>> This begs the question: what is the purpose of the memory allocation >>>> on user creation if it doesn't need to stick around upon restart? >>>> >>> >>> Creating a new user is expected to consume memory that can't be GC'd >>> due to the caching. Upon restart the users are loaded on demand into the >>> cache (user logs in or through admin console/endpoints). >>> >>> >>>> >>>> It seems to me that the user cache is keeping more than 10k users on >>>> user creation. Does this hypothesis agree with your understanding of >>>> Keycloak's design? >>>> >>> >>> By default it should only keep 10K users in memory. Once it hits 10K >>> the least recently used should entries should be evicted from the cache. >>> However, it seems this is not working properly. >>> >>> >>>> >>>> Thanks, >>>> Chris >>>> >>>> P.S. I have yet to test authentication load. >>>> >>>> On Thu, Jun 23, 2016 at 10:52 PM Chris Hairfield <chairfield(a)gmail.com&gt; >>>> wrote: >>>> >>>>> There seems to be a leak when enabling the user cache, even when >>>>> providing Keycloak with all the memory it needs to perform admirably >>>>> (initially). My heap drops to 3.4 GB when I perform a manual GC. >>>>> >>>>> I'm now seeing a number of PessimisticLockExceptions as it fails to >>>>> lock the USER_ENTITY table. I expect Keycloak to have ground to a halt by >>>>> the morning.[image: test8-graph.png] >>>>> >>>>> >>>>> On Thu, Jun 23, 2016 at 4:52 PM Chris Hairfield <chairfield(a)gmail.com&gt; >>>>> wrote: >>>>> >>>>>> I am only testing creating users. With the user cache disabled, I >>>>>> see no evidence of a memory leak at many JVM heap-size settings. >>>>>> >>>>>> Things get interesting when re-enabling the user cache. Performance >>>>>> seems to take a major hit in lower-memory scenarios, with some very >>>>>> worrisome scenarios where the rate at which I ingest continuously decreases. >>>>>> >>>>>> I am able to create users at a high rate of speed with a max heap >>>>>> size of 4g or 8g. Interestingly, Keycloak seems to prefer an Xms setting >>>>>> lower than the Xmx setting, which goes against official JBoss GC >>>>>> performance tuning documentation >>>>>> <http://www.mastertheboss.com/jboss-server/jboss-performance/jboss-perform... >>>>>> . >>>>>> >>>>>> Though speeds are high with a large heap, I *do* see the >>>>>> possibility of a memory leak with the user cache enabled and no >>>>>> authentication happening. I will be running a test overnight to attempt to >>>>>> confirm or deny. >>>>>> >>>>>> Thanks for your help so far, Stian. I will be on vacation tomorrow >>>>>> through Sunday, so I will pick this back up on Monday. >>>>>> >>>>>> P.S. I will be thinking of ways to better the documentation around >>>>>> performance tuning, as my tests indicate that standard JVM options cause >>>>>> Keycloak to run in a very sub-optimal state. >>>>>> >>>>>> On Wed, Jun 22, 2016 at 11:59 PM Stian Thorgersen < >>>>>> sthorger(a)redhat.com&gt; wrote: >>>>>> >>>>>>> Are you only creating users or are you also authenticating users? >>>>>>> User sessions are kept purely in memory so obviously the more you create >>>>>>> the more memory is used. Only creating users should not continue to >>>>>>> increase, but will do so for a while at least due to the way Java garbage >>>>>>> collection works. >>>>>>> >>>>>>> I would only have the user cache disabled for testing memory leak. >>>>>>> Re-enable it and retest with it before you eventually go into production as >>>>>>> it will have a significant impact on performance. >>>>>>> >>>>>>> On 23 June 2016 at 01:10, Chris Hairfield <chairfield(a)gmail.com&gt; >>>>>>> wrote: >>>>>>> >>>>>>>> Scratch the results of the graph I posted. I was running the test >>>>>>>> incorrectly. I'll post back with the results of the test run properly. >>>>>>>> >>>>>>>> On Wed, Jun 22, 2016 at 12:38 PM Chris Hairfield < >>>>>>>> chairfield(a)gmail.com&gt; wrote: >>>>>>>> >>>>>>>>> Thomas, this test is run with whatever local database Keycloak >>>>>>>>> defaults to. We're using Postgres generally, and we will have more >>>>>>>>> information pertaining to tests against Postgres soon. >>>>>>>>> >>>>>>>>> Stian, thanks for the tips. I am currently running a test to >>>>>>>>> ingest about 50m users into the default database with the user cache >>>>>>>>> disabled, 8gb mem (Xmx and Xms), and parallel GC threads == processor count. >>>>>>>>> >>>>>>>>> Though my test is young (430k users ingested), I'm noticing >>>>>>>>> memory allocation increasing in lockstep with the number of ingested users. >>>>>>>>> Is it expected to continue in this fashion, or is Keycloak designed to >>>>>>>>> level off in its memory usage? >>>>>>>>> >>>>>>>>> [image: increasing-heap.png] >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Jun 21, 2016 at 4:29 PM Stian Thorgersen < >>>>>>>>> sthorger(a)redhat.com&gt; wrote: >>>>>>>>> >>>>>>>>>> Keycloak by default caches users in-memory, by default it will >>>>>>>>>> keep up to 10000 entries cached. You can verify that there's no leak by >>>>>>>>>> disabling the user cache provider. See >>>>>>>>>> http://keycloak.github.io/docs/userguide/keycloak-server/html/server_cach... >>>>>>>>>> >>>>>>>>>> If you're planning on having millions of users I suggest you >>>>>>>>>> increase the allocated memory for the JVM (512MB which it seems you have is >>>>>>>>>> not sufficient). >>>>>>>>>> >>>>>>>>>> On 22 June 2016 at 00:29, Chris Hairfield <chairfield(a)gmail.com&gt; >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hello, >>>>>>>>>>> >>>>>>>>>>> When testing Keycloak 1.9.8 by ingesting a few million users, >>>>>>>>>>> we find that Keycloak leaks memory until it is rendered unresponsive (see >>>>>>>>>>> graph). Increasing JVM memory only increases the time it takes to encounter >>>>>>>>>>> this issue. >>>>>>>>>>> >>>>>>>>>>> We have put together a test project here >>>>>>>>>>> <https://github.com/anaerobic/keycloak-leakage> and opened an >>>>>>>>>>> issue here <https://issues.jboss.org/browse/KEYCLOAK-3146> as >>>>>>>>>>> we continue to investigate. As we are relying on Keycloak as a central >>>>>>>>>>> infrastructural component, any help would be greatly appreciated. >>>>>>>>>>> >>>>>>>>>>> We'll update with more information as we find it. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Chris >>>>>>>>>>> >>>>>>>>>>> [image: mem-cpu.png] >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> keycloak-user mailing list >>>>>>>>>>> keycloak-user(a)lists.jboss.org >>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-user >>>>>>>>>>> >>>>>>>>>> >>>>>>> >>> >>