Thanks for the input.
I am ok if there are few failures and few success and the system runs in
this manner so that we can tune the cache and make it better. But in our
case once the full GC is executed almost all the get-token requests are
failing. Keycloak doesn't seem to be recovering from here.
Here is my startup JVM params
=================
JAVA_OPTS="-Xms2048m -Xmx4096m -XX:-PrintGC -XX:-PrintGCDetails
-XX:-PrintGCTimeStamps -Xloggc:/DGlogs/gclogs.txt -XX:-UseGCLogFileRotation
-XX:NumberOfGCLogFiles=20 -XX:GCLogFileSize=10M -XX:MetaspaceSize=96M
-XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true"
JAVA_OPTS="$JAVA_OPTS
-Djboss.modules.system.pkgs=$JBOSS_MODULES_SYSTEM_PKGS
-Djava.awt.headless=true"
===================
And below is the cache configuration ...If there is any better
recommendation I would be glad to try that out....
===========================
<subsystem xmlns="urn:jboss:domain:infinispan:4.0">
<cache-container name="keycloak"
jndi-name="infinispan/Keycloak">
<transport lock-timeout="60000"/>
<local-cache name="realms">
<eviction max-entries="10000"
strategy="LRU"/>
</local-cache>
<local-cache name="users">
<eviction max-entries="50000"
strategy="LRU"/>
</local-cache>
<distributed-cache name="sessions" mode="SYNC"
owners="2"/>
<distributed-cache name="authenticationSessions"
mode="SYNC"
owners="2"/>
<!--<distributed-cache name="offlineSessions"
mode="SYNC"
owners="2"/>-->
<distributed-cache name="offlineSessions"
mode="SYNC"
owners="2">
<eviction max-entries="10000"
strategy="LRU"/>
</distributed-cache>
<!--<distributed-cache name="clientSessions"
mode="SYNC"
owners="2"/>-->
<distributed-cache name="clientSessions"
mode="SYNC"
owners="2">
<eviction max-entries="10000"
strategy="LRU"/>
<expiration max-idle="36000"/>
</distributed-cache>
<!--<distributed-cache name="offlineClientSessions"
mode="SYNC" owners="2"/>-->
<distributed-cache name="offlineClientSessions"
mode="SYNC"
owners="2">
<eviction max-entries="10000"
strategy="LRU"/>
<expiration max-idle="36000"/>
</distributed-cache>
<distributed-cache name="loginFailures" mode="SYNC"
owners="2"/>
<local-cache name="authorization">
<eviction max-entries="1000000"
strategy="LRU"/>
</local-cache>
<replicated-cache name="work" mode="SYNC"/>
<local-cache name="keys">
<eviction max-entries="1000"
strategy="LRU"/>
<expiration max-idle="3600000"/>
</local-cache>
<distributed-cache name="actionTokens" mode="SYNC"
owners="2">
<eviction max-entries="-1" strategy="NONE"/>
==================
Thanks
Arun
On Wed, Sep 19, 2018 at 4:21 PM, Sebastian Laskawiec <slaskawi(a)redhat.com>
wrote:
That's the line your balancing on. If the cache is very big, you
might get
an OOM Error. If it's too small, you will observe plenty queries to the
database. You need to find what size works best for you.
On Wed, Sep 19, 2018 at 10:56 AM Arun Velayudhan <arun.velayudhan@
motorolasolutions.com> wrote:
> Thanks Sabastian and Peter.
> After increasing the JVM memory and setting the cache limits the
> out-of-memory exception has been arrested. But I started observing lot of
> get-token requests failing after sometime i.e after 12hrs of run. On
> debugging, lot of failures are observed once the full GC is executed.
> Anyone has observed this behavior or any pointers?
>
>
> Arun
>
>
> On Wed, Aug 8, 2018 at 5:26 PM, Nalyvayko, Peter <pnalyvayko(a)agi.com>
> wrote:
>
> > Hi Arun,
> > Yes, we have seen this issue before. A temporary remedy has been to
> > increase jVM's maximum heap size, among other things, like changing the
> > eviction rate and infinispan cache sizes.
> > --Peter
> >
> > -----Original Message-----
> > From: keycloak-user-bounces(a)lists.jboss.org
> <keycloak-user-bounces@lists.
> >
jboss.org
>
<
https://urldefense.proofpoint.com/v2/url?u=http-3A__jboss.org&d=DwMFa...
> On Behalf Of Arun Velayudhan
> > Sent: Wednesday, August 8, 2018 1:02 AM
> > To: keycloak-user <keycloak-user(a)lists.jboss.org>
> > Subject: [keycloak-user] Uncaught server error:
> > java.lang.OutOfMemoryError: Java heap space
> >
> > Hello,
> > We ran keycloak with some basic load (like auth, gettoken) for few hours
> > at theand after sometime Keycloak threw an Out-of-memory error. Has
> anyone
> > faced similar kind of problem. Would be keen to know what was done to
> > mitigate.
> >
> > Version of Keycloak -> 4.0.0.Final.
> >
> >
> >
> > =====
> > 18:32:47,716 WARN [com.arjuna.ats.arjuna] (Transaction Reaper)
> > ARJUNA012117: TransactionReaper::check timeout for TX
> > 0:ffffc0a80c38:-56b32ec9:5b6463c3:54bcab in state RUN
> >
> > 18:30:23,749 ERROR [org.keycloak.services.error.KeycloakErrorHandler]
> > (default task-199) Uncaught server error: java.lang.OutOfMemoryError:
> Java
> > heap space
> >
> > 18:32:47,717 ERROR [org.keycloak.services.error.KeycloakErrorHandler]
> > (default task-219) Uncaught server error: java.lang.OutOfMemoryError:
> Java
> > heap space
> >
> > 18:32:47,717 ERROR [org.keycloak.services.error.KeycloakErrorHandler]
> > (default task-505) Uncaught server error: java.lang.OutOfMemoryError:
> Java
> > heap space ===============
> >
> > Pls find with the startup configuration
> > ===
> >
> > 19:46:33,121 DEBUG [org.jboss.as.config] (MSC service thread 1-7) VM
> > Arguments: -D[Standalone] -Xms64m -Xmx512m -XX:MetaspaceSize=96M
> > -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true
> > -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
> > -Dorg.jboss.boot.log.file=/DG/activeRelease/keycloak/
> > standalone/log/server.log
> > -Dlogging.configuration=file:/DG/activeRelease/keycloak/
> > standalone/configuration/logging.properties
> >
> > ==================
> >
> >
> >
> > Arun
> > _______________________________________________
> > keycloak-user mailing list
> > keycloak-user(a)lists.jboss.org
> >
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.
> > jboss.org_mailman_listinfo_keycloak-2Duser&d=DwIFAg&c=
> > q3cDpHe1hF8lXU5EFjNM_A&r=N8w3J29mjdEYPeHA-d1E_
> > 1Kp1KsnJ1yRhHwyWg25CWeqI66NKKjV9HfGziPFIi6m&m=dtKSJp9M6MgNlIhxciRs-
> > KVyFWOKZNNZNjQjWu5DP34&s=85dtwsUG0nkAMRjT3Jb9LTz0hbeuBXEvN8khcHdLq70&e=
> >
> _______________________________________________
> keycloak-user mailing list
> keycloak-user(a)lists.jboss.org
>
https://lists.jboss.org/mailman/listinfo/keycloak-user
>
<
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.jboss.org_mail...
>