[keycloak-user] Uncaught server error: java.lang.OutOfMemoryError: Java heap space
Sebastian Laskawiec
slaskawi at redhat.com
Thu Sep 20 07:21:45 EDT 2018
Ok, so I think you might tune a couple of things there:
- Xms = Xmx, so probably don't need dynamic sizing.
- MetaspaceSize and MaxMetaspaceSize to be removed.
- Use either CMS or G1 (I think the latest JDK builds overrun CMS, which
wasn't true a couple of months back).
- Replace distributed caches with number of owners = 2 with replicated
ones.
I also highly advice reading this documents:
- Infinispan Performance Guide:
http://infinispan.org/docs/dev/performance_guide/performance_guide.html
- My very old conference manual on tuning JGroups. However, you should
probably look up for better materials. Please treat it as a starting point:
https://github.com/slaskawi/presentations/tree/master/2017_infinispan_rolling_upgrade#configuration-tuning-guide
On Wed, Sep 19, 2018 at 3:03 PM Arun Velayudhan <
arun.velayudhan at motorolasolutions.com> wrote:
> Thanks for the input.
> I am ok if there are few failures and few success and the system runs in
> this manner so that we can tune the cache and make it better. But in our
> case once the full GC is executed almost all the get-token requests are
> failing. Keycloak doesn't seem to be recovering from here.
>
> Here is my startup JVM params
>
> =================
> JAVA_OPTS="-Xms2048m -Xmx4096m -XX:-PrintGC -XX:-PrintGCDetails
> -XX:-PrintGCTimeStamps -Xloggc:/DGlogs/gclogs.txt -XX:-UseGCLogFileRotation
> -XX:NumberOfGCLogFiles=20 -XX:GCLogFileSize=10M -XX:MetaspaceSize=96M
> -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true"
> JAVA_OPTS="$JAVA_OPTS
> -Djboss.modules.system.pkgs=$JBOSS_MODULES_SYSTEM_PKGS
> -Djava.awt.headless=true"
> ===================
>
> And below is the cache configuration ...If there is any better
> recommendation I would be glad to try that out....
>
> ===========================
> <subsystem xmlns="urn:jboss:domain:infinispan:4.0">
> <cache-container name="keycloak"
> jndi-name="infinispan/Keycloak">
> <transport lock-timeout="60000"/>
> <local-cache name="realms">
> <eviction max-entries="10000" strategy="LRU"/>
> </local-cache>
> <local-cache name="users">
> <eviction max-entries="50000" strategy="LRU"/>
> </local-cache>
> <distributed-cache name="sessions" mode="SYNC" owners="2"/>
> <distributed-cache name="authenticationSessions"
> mode="SYNC" owners="2"/>
> <!--<distributed-cache name="offlineSessions" mode="SYNC"
> owners="2"/>-->
> <distributed-cache name="offlineSessions" mode="SYNC"
> owners="2">
> <eviction max-entries="10000" strategy="LRU"/>
> </distributed-cache>
> <!--<distributed-cache name="clientSessions" mode="SYNC"
> owners="2"/>-->
> <distributed-cache name="clientSessions" mode="SYNC"
> owners="2">
> <eviction max-entries="10000" strategy="LRU"/>
> <expiration max-idle="36000"/>
> </distributed-cache>
> <!--<distributed-cache name="offlineClientSessions"
> mode="SYNC" owners="2"/>-->
> <distributed-cache name="offlineClientSessions" mode="SYNC"
> owners="2">
> <eviction max-entries="10000" strategy="LRU"/>
> <expiration max-idle="36000"/>
> </distributed-cache>
> <distributed-cache name="loginFailures" mode="SYNC"
> owners="2"/>
> <local-cache name="authorization">
> <eviction max-entries="1000000" strategy="LRU"/>
> </local-cache>
> <replicated-cache name="work" mode="SYNC"/>
> <local-cache name="keys">
> <eviction max-entries="1000" strategy="LRU"/>
> <expiration max-idle="3600000"/>
> </local-cache>
> <distributed-cache name="actionTokens" mode="SYNC"
> owners="2">
> <eviction max-entries="-1" strategy="NONE"/>
> ==================
>
> Thanks
> Arun
>
>
>
> On Wed, Sep 19, 2018 at 4:21 PM, Sebastian Laskawiec <slaskawi at redhat.com>
> wrote:
>
>> That's the line your balancing on. If the cache is very big, you might
>> get an OOM Error. If it's too small, you will observe plenty queries to the
>> database. You need to find what size works best for you.
>>
>> On Wed, Sep 19, 2018 at 10:56 AM Arun Velayudhan <
>> arun.velayudhan at motorolasolutions.com> wrote:
>>
>>> Thanks Sabastian and Peter.
>>> After increasing the JVM memory and setting the cache limits the
>>> out-of-memory exception has been arrested. But I started observing lot of
>>> get-token requests failing after sometime i.e after 12hrs of run. On
>>> debugging, lot of failures are observed once the full GC is executed.
>>> Anyone has observed this behavior or any pointers?
>>>
>>>
>>> Arun
>>>
>>>
>>> On Wed, Aug 8, 2018 at 5:26 PM, Nalyvayko, Peter <pnalyvayko at agi.com>
>>> wrote:
>>>
>>> > Hi Arun,
>>> > Yes, we have seen this issue before. A temporary remedy has been to
>>> > increase jVM's maximum heap size, among other things, like changing the
>>> > eviction rate and infinispan cache sizes.
>>> > --Peter
>>> >
>>> > -----Original Message-----
>>> > From: keycloak-user-bounces at lists.jboss.org
>>> <keycloak-user-bounces at lists.
>>> > jboss.org
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__jboss.org&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=N8w3J29mjdEYPeHA-d1E_1Kp1KsnJ1yRhHwyWg25CWeqI66NKKjV9HfGziPFIi6m&m=8Pvsmoua5c6CWWNh79KqdxXWEU8kyUEVv_6yuushpRY&s=WJqYionlEDTjKU-8EIQZmkAjgEsDbOxey9VNZnxPI8s&e=>>
>>> On Behalf Of Arun Velayudhan
>>> > Sent: Wednesday, August 8, 2018 1:02 AM
>>> > To: keycloak-user <keycloak-user at lists.jboss.org>
>>> > Subject: [keycloak-user] Uncaught server error:
>>> > java.lang.OutOfMemoryError: Java heap space
>>> >
>>> > Hello,
>>> > We ran keycloak with some basic load (like auth, gettoken) for few
>>> hours
>>> > at theand after sometime Keycloak threw an Out-of-memory error. Has
>>> anyone
>>> > faced similar kind of problem. Would be keen to know what was done to
>>> > mitigate.
>>> >
>>> > Version of Keycloak -> 4.0.0.Final.
>>> >
>>> >
>>> >
>>> > =====
>>> > 18:32:47,716 WARN [com.arjuna.ats.arjuna] (Transaction Reaper)
>>> > ARJUNA012117: TransactionReaper::check timeout for TX
>>> > 0:ffffc0a80c38:-56b32ec9:5b6463c3:54bcab in state RUN
>>> >
>>> > 18:30:23,749 ERROR [org.keycloak.services.error.KeycloakErrorHandler]
>>> > (default task-199) Uncaught server error: java.lang.OutOfMemoryError:
>>> Java
>>> > heap space
>>> >
>>> > 18:32:47,717 ERROR [org.keycloak.services.error.KeycloakErrorHandler]
>>> > (default task-219) Uncaught server error: java.lang.OutOfMemoryError:
>>> Java
>>> > heap space
>>> >
>>> > 18:32:47,717 ERROR [org.keycloak.services.error.KeycloakErrorHandler]
>>> > (default task-505) Uncaught server error: java.lang.OutOfMemoryError:
>>> Java
>>> > heap space ===============
>>> >
>>> > Pls find with the startup configuration
>>> > ===
>>> >
>>> > 19:46:33,121 DEBUG [org.jboss.as.config] (MSC service thread 1-7) VM
>>> > Arguments: -D[Standalone] -Xms64m -Xmx512m -XX:MetaspaceSize=96M
>>> > -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true
>>> > -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
>>> > -Dorg.jboss.boot.log.file=/DG/activeRelease/keycloak/
>>> > standalone/log/server.log
>>> > -Dlogging.configuration=file:/DG/activeRelease/keycloak/
>>> > standalone/configuration/logging.properties
>>> >
>>> > ==================
>>> >
>>> >
>>> >
>>> > Arun
>>> > _______________________________________________
>>> > keycloak-user mailing list
>>> > keycloak-user at lists.jboss.org
>>> > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.
>>> > jboss.org_mailman_listinfo_keycloak-2Duser&d=DwIFAg&c=
>>> > q3cDpHe1hF8lXU5EFjNM_A&r=N8w3J29mjdEYPeHA-d1E_
>>> > 1Kp1KsnJ1yRhHwyWg25CWeqI66NKKjV9HfGziPFIi6m&m=dtKSJp9M6MgNlIhxciRs-
>>> > KVyFWOKZNNZNjQjWu5DP34&s=85dtwsUG0nkAMRjT3Jb9LTz0hbeuBXEvN8khcHdLq70&e=
>>> >
>>> _______________________________________________
>>> keycloak-user mailing list
>>> keycloak-user at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/keycloak-user
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.jboss.org_mailman_listinfo_keycloak-2Duser&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=N8w3J29mjdEYPeHA-d1E_1Kp1KsnJ1yRhHwyWg25CWeqI66NKKjV9HfGziPFIi6m&m=8Pvsmoua5c6CWWNh79KqdxXWEU8kyUEVv_6yuushpRY&s=U7nxEsDNiCE0Wpqs1agThkPMt-vBfIazfAmGA2a3IKo&e=>
>>>
>>
>
More information about the keycloak-user
mailing list