[keycloak-user] Out of memory error on Keycloak cluster

[robinfernandes .]

Hi Guys,

I added the eviction policy to the standalone-ha.xml and it looks like this
:

 *<subsystem xmlns="urn:jboss:domain:infinispan:4.0">*
*            <cache-container name="keycloak"
jndi-name="infinispan/Keycloak">*
*                <transport lock-timeout="60000"/>*
*                <invalidation-cache name="realms" mode="SYNC"/>*
*                <invalidation-cache name="users" mode="SYNC">*
*                    <eviction max-entries="10000" strategy="LRU"/>*
*                </invalidation-cache>*
*                <distributed-cache name="sessions" mode="SYNC"
owners="3"/>*
*                <distributed-cache name="offlineSessions" mode="SYNC"
owners="3"/>*
*                <distributed-cache name="loginFailures" mode="SYNC"
owners="1"/>*
*                <replicated-cache name="work" mode="SYNC"/>*
*                <local-cache name="realmVersions">*
*                    <transaction mode="BATCH" locking="PESSIMISTIC"/>*
*                <eviction max-entries="10000" strategy="LRU"/>*
*            </local-cache>*
*            </cache-container>*

I ran some tests to do concurrent logins by spawning multiple threads and
the Keycloak node still went down after there were around *~170K* active
sessions. Took the thread and heap dump as well if that is helpful. We have
the -Xms set to 512m and -Xmx set to 2048m.

Would you recommend a higher heap size as well?

Thanks,
Robin

On Wed, Oct 12, 2016 at 10:03 AM, Stian Thorgersen <sthorger at redhat.com>
wrote:

> Could be https://issues.jboss.org/browse/KEYCLOAK-3202 if so it's not
> fixed
> in 1.9.8. There's a work around though, you can set "<eviction
> max-entries="10000" strategy="LRU"/>" for the realmVersions cache. Also,
> make sure you have a sane max entries on the users cache.
>
> On 11 October 2016 at 15:33, Bill Burke <bburke at redhat.com> wrote:
>
> > I believe we fixed some cache leakage problems sometime between 1.9.1
> > and 1.9.8.  You'll have to search JIRA.  I strongly suggest you upgrade
> > to 1.9.8.  We did a huge amount of stability, performance, and bug fixes
> > between 1.9.1 and 1.9.8 to get Keycloak ready for product.  RH-SSO is
> > based on Keycloak 1.9.8.
> >
> >
> > On 10/10/16 11:21 AM, robinfernandes . wrote:
> > > Hi,
> > >
> > > We are using Keycloak 1.9.2.Final and have a cluster with an hap and 3
> > > keycloak nodes behind it.
> > > For the first time in about 4-6 months we received errors that java
> heap
> > > space out of memory and the nodes just went down.
> > > We had around 100k users as well as 35k active connections at the time.
> > > We have around 512MB heap space assigned.
> > >
> > > I am not able to reproduce it after restarting the nodes.
> > >
> > > Is there any reason that this could happen?
> > >
> > >
> > > _______________________________________________
> > > keycloak-user mailing list
> > > keycloak-user at lists.jboss.org
> > > https://lists.jboss.org/mailman/listinfo/keycloak-user
> >
> > _______________________________________________
> > keycloak-user mailing list
> > keycloak-user at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/keycloak-user
> >
> _______________________________________________
> keycloak-user mailing list
> keycloak-user at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user
>