Thanks Sabastian and Peter. After increasing the JVM memory and setting the cache limits the out-of-memory exception has been arrested. But I started observing lot of get-token requests failing after sometime i.e after 12hrs of run. On debugging, lot of failures are observed once the full GC is executed. Anyone has observed this behavior or any pointers? Arun On Wed, Aug 8, 2018 at 5:26 PM, Nalyvayko, Peter <pnalyvayko(a)agi.com&gt; wrote: > Hi Arun, > Yes, we have seen this issue before. A temporary remedy has been to > increase jVM's maximum heap size, among other things, like changing the > eviction rate and infinispan cache sizes. > --Peter > > -----Original Message----- > From: keycloak-user-bounces(a)lists.jboss.org <keycloak-user-bounces@lists. > jboss.org> On Behalf Of Arun Velayudhan > Sent: Wednesday, August 8, 2018 1:02 AM > To: keycloak-user <keycloak-user(a)lists.jboss.org&gt; > Subject: [keycloak-user] Uncaught server error: > java.lang.OutOfMemoryError: Java heap space > > Hello, > We ran keycloak with some basic load (like auth, gettoken) for few hours > at theand after sometime Keycloak threw an Out-of-memory error. Has anyone > faced similar kind of problem. Would be keen to know what was done to > mitigate. > > Version of Keycloak -> 4.0.0.Final. > > > > ===== > 18:32:47,716 WARN [com.arjuna.ats.arjuna] (Transaction Reaper) > ARJUNA012117: TransactionReaper::check timeout for TX > 0:ffffc0a80c38:-56b32ec9:5b6463c3:54bcab in state RUN > > 18:30:23,749 ERROR [org.keycloak.services.error.KeycloakErrorHandler] > (default task-199) Uncaught server error: java.lang.OutOfMemoryError: Java > heap space > > 18:32:47,717 ERROR [org.keycloak.services.error.KeycloakErrorHandler] > (default task-219) Uncaught server error: java.lang.OutOfMemoryError: Java > heap space > > 18:32:47,717 ERROR [org.keycloak.services.error.KeycloakErrorHandler] > (default task-505) Uncaught server error: java.lang.OutOfMemoryError: Java > heap space =============== > > Pls find with the startup configuration > === > > 19:46:33,121 DEBUG [org.jboss.as.config] (MSC service thread 1-7) VM > Arguments: -D[Standalone] -Xms64m -Xmx512m -XX:MetaspaceSize=96M > -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true > -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true > -Dorg.jboss.boot.log.file=/DG/activeRelease/keycloak/ > standalone/log/server.log > -Dlogging.configuration=file:/DG/activeRelease/keycloak/ > standalone/configuration/logging.properties > > ================== > > > > Arun > _______________________________________________ > keycloak-user mailing list > keycloak-user(a)lists.jboss.org > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists. > jboss.org_mailman_listinfo_keycloak-2Duser&d=DwIFAg&c= > q3cDpHe1hF8lXU5EFjNM_A&r=N8w3J29mjdEYPeHA-d1E_ > 1Kp1KsnJ1yRhHwyWg25CWeqI66NKKjV9HfGziPFIi6m&m=dtKSJp9M6MgNlIhxciRs- > KVyFWOKZNNZNjQjWu5DP34&s=85dtwsUG0nkAMRjT3Jb9LTz0hbeuBXEvN8khcHdLq70&e= > _______________________________________________ keycloak-user mailing list keycloak-user(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/keycloak-user

Thanks for the input. I am ok if there are few failures and few success and the system runs in this manner so that we can tune the cache and make it better. But in our case once the full GC is executed almost all the get-token requests are failing. Keycloak doesn't seem to be recovering from here. Here is my startup JVM params ================= JAVA_OPTS="-Xms2048m -Xmx4096m -XX:-PrintGC -XX:-PrintGCDetails -XX:-PrintGCTimeStamps -Xloggc:/DGlogs/gclogs.txt -XX:-UseGCLogFileRotation -XX:NumberOfGCLogFiles=20 -XX:GCLogFileSize=10M -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true" JAVA_OPTS="$JAVA_OPTS -Djboss.modules.system.pkgs=$JBOSS_MODULES_SYSTEM_PKGS -Djava.awt.headless=true" =================== And below is the cache configuration ...If there is any better recommendation I would be glad to try that out.... =========================== <subsystem xmlns="urn:jboss:domain:infinispan:4.0"> <cache-container name="keycloak" jndi-name="infinispan/Keycloak"> <transport lock-timeout="60000"/> <local-cache name="realms"> <eviction max-entries="10000" strategy="LRU"/> </local-cache> <local-cache name="users"> <eviction max-entries="50000" strategy="LRU"/> </local-cache> <distributed-cache name="sessions" mode="SYNC" owners="2"/> <distributed-cache name="authenticationSessions" mode="SYNC" owners="2"/> <!--<distributed-cache name="offlineSessions" mode="SYNC" owners="2"/>--> <distributed-cache name="offlineSessions" mode="SYNC" owners="2"> <eviction max-entries="10000" strategy="LRU"/> </distributed-cache> <!--<distributed-cache name="clientSessions" mode="SYNC" owners="2"/>--> <distributed-cache name="clientSessions" mode="SYNC" owners="2"> <eviction max-entries="10000" strategy="LRU"/> <expiration max-idle="36000"/> </distributed-cache> <!--<distributed-cache name="offlineClientSessions" mode="SYNC" owners="2"/>--> <distributed-cache name="offlineClientSessions" mode="SYNC" owners="2"> <eviction max-entries="10000" strategy="LRU"/> <expiration max-idle="36000"/> </distributed-cache> <distributed-cache name="loginFailures" mode="SYNC" owners="2"/> <local-cache name="authorization"> <eviction max-entries="1000000" strategy="LRU"/> </local-cache> <replicated-cache name="work" mode="SYNC"/> <local-cache name="keys"> <eviction max-entries="1000" strategy="LRU"/> <expiration max-idle="3600000"/> </local-cache> <distributed-cache name="actionTokens" mode="SYNC" owners="2"> <eviction max-entries="-1" strategy="NONE"/> ================== Thanks Arun On Wed, Sep 19, 2018 at 4:21 PM, Sebastian Laskawiec <slaskawi(a)redhat.com&gt; wrote: > That's the line your balancing on. If the cache is very big, you might > get an OOM Error. If it's too small, you will observe plenty queries to the > database. You need to find what size works best for you. > > On Wed, Sep 19, 2018 at 10:56 AM Arun Velayudhan < > arun.velayudhan(a)motorolasolutions.com&gt; wrote: > >> Thanks Sabastian and Peter. >> After increasing the JVM memory and setting the cache limits the >> out-of-memory exception has been arrested. But I started observing lot of >> get-token requests failing after sometime i.e after 12hrs of run. On >> debugging, lot of failures are observed once the full GC is executed. >> Anyone has observed this behavior or any pointers? >> >> >> Arun >> >> >> On Wed, Aug 8, 2018 at 5:26 PM, Nalyvayko, Peter <pnalyvayko(a)agi.com&gt; >> wrote: >> >> > Hi Arun, >> > Yes, we have seen this issue before. A temporary remedy has been to >> > increase jVM's maximum heap size, among other things, like changing the >> > eviction rate and infinispan cache sizes. >> > --Peter >> > >> > -----Original Message----- >> > From: keycloak-user-bounces(a)lists.jboss.org >> <keycloak-user-bounces@lists. >> > jboss.org >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__jboss.org&d=DwMFa... >> On Behalf Of Arun Velayudhan >> > Sent: Wednesday, August 8, 2018 1:02 AM >> > To: keycloak-user <keycloak-user(a)lists.jboss.org&gt; >> > Subject: [keycloak-user] Uncaught server error: >> > java.lang.OutOfMemoryError: Java heap space >> > >> > Hello, >> > We ran keycloak with some basic load (like auth, gettoken) for few >> hours >> > at theand after sometime Keycloak threw an Out-of-memory error. Has >> anyone >> > faced similar kind of problem. Would be keen to know what was done to >> > mitigate. >> > >> > Version of Keycloak -> 4.0.0.Final. >> > >> > >> > >> > ===== >> > 18:32:47,716 WARN [com.arjuna.ats.arjuna] (Transaction Reaper) >> > ARJUNA012117: TransactionReaper::check timeout for TX >> > 0:ffffc0a80c38:-56b32ec9:5b6463c3:54bcab in state RUN >> > >> > 18:30:23,749 ERROR [org.keycloak.services.error.KeycloakErrorHandler] >> > (default task-199) Uncaught server error: java.lang.OutOfMemoryError: >> Java >> > heap space >> > >> > 18:32:47,717 ERROR [org.keycloak.services.error.KeycloakErrorHandler] >> > (default task-219) Uncaught server error: java.lang.OutOfMemoryError: >> Java >> > heap space >> > >> > 18:32:47,717 ERROR [org.keycloak.services.error.KeycloakErrorHandler] >> > (default task-505) Uncaught server error: java.lang.OutOfMemoryError: >> Java >> > heap space =============== >> > >> > Pls find with the startup configuration >> > === >> > >> > 19:46:33,121 DEBUG [org.jboss.as.config] (MSC service thread 1-7) VM >> > Arguments: -D[Standalone] -Xms64m -Xmx512m -XX:MetaspaceSize=96M >> > -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true >> > -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true >> > -Dorg.jboss.boot.log.file=/DG/activeRelease/keycloak/ >> > standalone/log/server.log >> > -Dlogging.configuration=file:/DG/activeRelease/keycloak/ >> > standalone/configuration/logging.properties >> > >> > ================== >> > >> > >> > >> > Arun >> > _______________________________________________ >> > keycloak-user mailing list >> > keycloak-user(a)lists.jboss.org >> > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists. >> > jboss.org_mailman_listinfo_keycloak-2Duser&d=DwIFAg&c= >> > q3cDpHe1hF8lXU5EFjNM_A&r=N8w3J29mjdEYPeHA-d1E_ >> > 1Kp1KsnJ1yRhHwyWg25CWeqI66NKKjV9HfGziPFIi6m&m=dtKSJp9M6MgNlIhxciRs- >> > KVyFWOKZNNZNjQjWu5DP34&s=85dtwsUG0nkAMRjT3Jb9LTz0hbeuBXEvN8khcHdLq70&e= >> > >> _______________________________________________ >> keycloak-user mailing list >> keycloak-user(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/keycloak-user >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.jboss.org_mail... >> >