Re: [keycloak-user] Major Memory Leak/Consumption When Creating Many Users

I am only testing creating users. With the user cache disabled, I see no evidence of a memory leak at many JVM heap-size settings. Things get interesting when re-enabling the user cache. Performance seems to take a major hit in lower-memory scenarios, with some very worrisome scenarios where the rate at which I ingest continuously decreases. I am able to create users at a high rate of speed with a max heap size of 4g or 8g. Interestingly, Keycloak seems to prefer an Xms setting lower than the Xmx setting, which goes against official JBoss GC performance tuning documentation <http://www.mastertheboss.com/jboss-server/jboss-performance/jboss-perform... . Though speeds are high with a large heap, I *do* see the possibility of a memory leak with the user cache enabled and no authentication happening. I will be running a test overnight to attempt to confirm or deny. Thanks for your help so far, Stian. I will be on vacation tomorrow through Sunday, so I will pick this back up on Monday. P.S. I will be thinking of ways to better the documentation around performance tuning, as my tests indicate that standard JVM options cause Keycloak to run in a very sub-optimal state. On Wed, Jun 22, 2016 at 11:59 PM Stian Thorgersen <sthorger(a)redhat.com&gt; wrote: > Are you only creating users or are you also authenticating users? User > sessions are kept purely in memory so obviously the more you create the > more memory is used. Only creating users should not continue to increase, > but will do so for a while at least due to the way Java garbage collection > works. > > I would only have the user cache disabled for testing memory leak. > Re-enable it and retest with it before you eventually go into production as > it will have a significant impact on performance. > > On 23 June 2016 at 01:10, Chris Hairfield <chairfield(a)gmail.com&gt; wrote: > >> Scratch the results of the graph I posted. I was running the test >> incorrectly. I'll post back with the results of the test run properly. >> >> On Wed, Jun 22, 2016 at 12:38 PM Chris Hairfield <chairfield(a)gmail.com&gt; >> wrote: >> >>> Thomas, this test is run with whatever local database Keycloak defaults >>> to. We're using Postgres generally, and we will have more information >>> pertaining to tests against Postgres soon. >>> >>> Stian, thanks for the tips. I am currently running a test to ingest >>> about 50m users into the default database with the user cache disabled, 8gb >>> mem (Xmx and Xms), and parallel GC threads == processor count. >>> >>> Though my test is young (430k users ingested), I'm noticing memory >>> allocation increasing in lockstep with the number of ingested users. Is it >>> expected to continue in this fashion, or is Keycloak designed to level off >>> in its memory usage? >>> >>> [image: increasing-heap.png] >>> >>> >>> On Tue, Jun 21, 2016 at 4:29 PM Stian Thorgersen <sthorger(a)redhat.com&gt; >>> wrote: >>> >>>> Keycloak by default caches users in-memory, by default it will keep up >>>> to 10000 entries cached. You can verify that there's no leak by disabling >>>> the user cache provider. See >>>> http://keycloak.github.io/docs/userguide/keycloak-server/html/server_cach... >>>> >>>> If you're planning on having millions of users I suggest you increase >>>> the allocated memory for the JVM (512MB which it seems you have is not >>>> sufficient). >>>> >>>> On 22 June 2016 at 00:29, Chris Hairfield <chairfield(a)gmail.com&gt; >>>> wrote: >>>> >>>>> Hello, >>>>> >>>>> When testing Keycloak 1.9.8 by ingesting a few million users, we find >>>>> that Keycloak leaks memory until it is rendered unresponsive (see graph). >>>>> Increasing JVM memory only increases the time it takes to encounter this >>>>> issue. >>>>> >>>>> We have put together a test project here >>>>> <https://github.com/anaerobic/keycloak-leakage> and opened an issue >>>>> here <https://issues.jboss.org/browse/KEYCLOAK-3146> as we continue >>>>> to investigate. As we are relying on Keycloak as a central infrastructural >>>>> component, any help would be greatly appreciated. >>>>> >>>>> We'll update with more information as we find it. >>>>> >>>>> Thanks, >>>>> Chris >>>>> >>>>> [image: mem-cpu.png] >>>>> >>>>> _______________________________________________ >>>>> keycloak-user mailing list >>>>> keycloak-user(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/keycloak-user >>>>> >>>> >