[keycloak-user] Uncaught server error: java.lang.OutOfMemoryError: Java heap space
Arun Velayudhan
arun.velayudhan at motorolasolutions.com
Wed Sep 19 09:03:00 EDT 2018
Thanks for the input.
I am ok if there are few failures and few success and the system runs in
this manner so that we can tune the cache and make it better. But in our
case once the full GC is executed almost all the get-token requests are
failing. Keycloak doesn't seem to be recovering from here.
Here is my startup JVM params
=================
JAVA_OPTS="-Xms2048m -Xmx4096m -XX:-PrintGC -XX:-PrintGCDetails
-XX:-PrintGCTimeStamps -Xloggc:/DGlogs/gclogs.txt -XX:-UseGCLogFileRotation
-XX:NumberOfGCLogFiles=20 -XX:GCLogFileSize=10M -XX:MetaspaceSize=96M
-XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true"
JAVA_OPTS="$JAVA_OPTS
-Djboss.modules.system.pkgs=$JBOSS_MODULES_SYSTEM_PKGS
-Djava.awt.headless=true"
===================
And below is the cache configuration ...If there is any better
recommendation I would be glad to try that out....
===========================
<subsystem xmlns="urn:jboss:domain:infinispan:4.0">
<cache-container name="keycloak" jndi-name="infinispan/Keycloak">
<transport lock-timeout="60000"/>
<local-cache name="realms">
<eviction max-entries="10000" strategy="LRU"/>
</local-cache>
<local-cache name="users">
<eviction max-entries="50000" strategy="LRU"/>
</local-cache>
<distributed-cache name="sessions" mode="SYNC" owners="2"/>
<distributed-cache name="authenticationSessions" mode="SYNC"
owners="2"/>
<!--<distributed-cache name="offlineSessions" mode="SYNC"
owners="2"/>-->
<distributed-cache name="offlineSessions" mode="SYNC"
owners="2">
<eviction max-entries="10000" strategy="LRU"/>
</distributed-cache>
<!--<distributed-cache name="clientSessions" mode="SYNC"
owners="2"/>-->
<distributed-cache name="clientSessions" mode="SYNC"
owners="2">
<eviction max-entries="10000" strategy="LRU"/>
<expiration max-idle="36000"/>
</distributed-cache>
<!--<distributed-cache name="offlineClientSessions"
mode="SYNC" owners="2"/>-->
<distributed-cache name="offlineClientSessions" mode="SYNC"
owners="2">
<eviction max-entries="10000" strategy="LRU"/>
<expiration max-idle="36000"/>
</distributed-cache>
<distributed-cache name="loginFailures" mode="SYNC"
owners="2"/>
<local-cache name="authorization">
<eviction max-entries="1000000" strategy="LRU"/>
</local-cache>
<replicated-cache name="work" mode="SYNC"/>
<local-cache name="keys">
<eviction max-entries="1000" strategy="LRU"/>
<expiration max-idle="3600000"/>
</local-cache>
<distributed-cache name="actionTokens" mode="SYNC"
owners="2">
<eviction max-entries="-1" strategy="NONE"/>
==================
Thanks
Arun
On Wed, Sep 19, 2018 at 4:21 PM, Sebastian Laskawiec <slaskawi at redhat.com>
wrote:
> That's the line your balancing on. If the cache is very big, you might get
> an OOM Error. If it's too small, you will observe plenty queries to the
> database. You need to find what size works best for you.
>
> On Wed, Sep 19, 2018 at 10:56 AM Arun Velayudhan <arun.velayudhan@
> motorolasolutions.com> wrote:
>
>> Thanks Sabastian and Peter.
>> After increasing the JVM memory and setting the cache limits the
>> out-of-memory exception has been arrested. But I started observing lot of
>> get-token requests failing after sometime i.e after 12hrs of run. On
>> debugging, lot of failures are observed once the full GC is executed.
>> Anyone has observed this behavior or any pointers?
>>
>>
>> Arun
>>
>>
>> On Wed, Aug 8, 2018 at 5:26 PM, Nalyvayko, Peter <pnalyvayko at agi.com>
>> wrote:
>>
>> > Hi Arun,
>> > Yes, we have seen this issue before. A temporary remedy has been to
>> > increase jVM's maximum heap size, among other things, like changing the
>> > eviction rate and infinispan cache sizes.
>> > --Peter
>> >
>> > -----Original Message-----
>> > From: keycloak-user-bounces at lists.jboss.org
>> <keycloak-user-bounces at lists.
>> > jboss.org
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__jboss.org&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=N8w3J29mjdEYPeHA-d1E_1Kp1KsnJ1yRhHwyWg25CWeqI66NKKjV9HfGziPFIi6m&m=8Pvsmoua5c6CWWNh79KqdxXWEU8kyUEVv_6yuushpRY&s=WJqYionlEDTjKU-8EIQZmkAjgEsDbOxey9VNZnxPI8s&e=>>
>> On Behalf Of Arun Velayudhan
>> > Sent: Wednesday, August 8, 2018 1:02 AM
>> > To: keycloak-user <keycloak-user at lists.jboss.org>
>> > Subject: [keycloak-user] Uncaught server error:
>> > java.lang.OutOfMemoryError: Java heap space
>> >
>> > Hello,
>> > We ran keycloak with some basic load (like auth, gettoken) for few hours
>> > at theand after sometime Keycloak threw an Out-of-memory error. Has
>> anyone
>> > faced similar kind of problem. Would be keen to know what was done to
>> > mitigate.
>> >
>> > Version of Keycloak -> 4.0.0.Final.
>> >
>> >
>> >
>> > =====
>> > 18:32:47,716 WARN [com.arjuna.ats.arjuna] (Transaction Reaper)
>> > ARJUNA012117: TransactionReaper::check timeout for TX
>> > 0:ffffc0a80c38:-56b32ec9:5b6463c3:54bcab in state RUN
>> >
>> > 18:30:23,749 ERROR [org.keycloak.services.error.KeycloakErrorHandler]
>> > (default task-199) Uncaught server error: java.lang.OutOfMemoryError:
>> Java
>> > heap space
>> >
>> > 18:32:47,717 ERROR [org.keycloak.services.error.KeycloakErrorHandler]
>> > (default task-219) Uncaught server error: java.lang.OutOfMemoryError:
>> Java
>> > heap space
>> >
>> > 18:32:47,717 ERROR [org.keycloak.services.error.KeycloakErrorHandler]
>> > (default task-505) Uncaught server error: java.lang.OutOfMemoryError:
>> Java
>> > heap space ===============
>> >
>> > Pls find with the startup configuration
>> > ===
>> >
>> > 19:46:33,121 DEBUG [org.jboss.as.config] (MSC service thread 1-7) VM
>> > Arguments: -D[Standalone] -Xms64m -Xmx512m -XX:MetaspaceSize=96M
>> > -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true
>> > -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
>> > -Dorg.jboss.boot.log.file=/DG/activeRelease/keycloak/
>> > standalone/log/server.log
>> > -Dlogging.configuration=file:/DG/activeRelease/keycloak/
>> > standalone/configuration/logging.properties
>> >
>> > ==================
>> >
>> >
>> >
>> > Arun
>> > _______________________________________________
>> > keycloak-user mailing list
>> > keycloak-user at lists.jboss.org
>> > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.
>> > jboss.org_mailman_listinfo_keycloak-2Duser&d=DwIFAg&c=
>> > q3cDpHe1hF8lXU5EFjNM_A&r=N8w3J29mjdEYPeHA-d1E_
>> > 1Kp1KsnJ1yRhHwyWg25CWeqI66NKKjV9HfGziPFIi6m&m=dtKSJp9M6MgNlIhxciRs-
>> > KVyFWOKZNNZNjQjWu5DP34&s=85dtwsUG0nkAMRjT3Jb9LTz0hbeuBXEvN8khcHdLq70&e=
>> >
>> _______________________________________________
>> keycloak-user mailing list
>> keycloak-user at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/keycloak-user
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.jboss.org_mailman_listinfo_keycloak-2Duser&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=N8w3J29mjdEYPeHA-d1E_1Kp1KsnJ1yRhHwyWg25CWeqI66NKKjV9HfGziPFIi6m&m=8Pvsmoua5c6CWWNh79KqdxXWEU8kyUEVv_6yuushpRY&s=U7nxEsDNiCE0Wpqs1agThkPMt-vBfIazfAmGA2a3IKo&e=>
>>
>
More information about the keycloak-user
mailing list