[keycloak-user] Keycloak OutOfMemoryError

[Dmitry Telegin]

Hello Jason,

The problem seems to be in the SSL stack, which is not a part of Keycloak itself. Keycloak is built on top of Wildfly application server, and SSL is implemented by one of Wildfly components, namely Undertow.

You seem to be hitting this bug: https://issues.jboss.org/browse/UNDERTOW-472

Though JIRA says that it should have been fixed in Undertow 1.3.10, the version shipped with Keycloak 3.1.0 still seems to be buggy (1.3.15).
Either way, it is highly recommended that you upgrade to the recent Keycloak that uses up-to-date Wildfly (and therefore Undertow).

Cheers,
Dmitry Telegin
CTO, Acutus s.r.o.
Keycloak Consulting and Training

Pod lipami street 339/52, 130 00 Prague 3, Czech Republic
+42 (022) 888-30-71
E-mail: info at acutus.pro

On Fri, 2018-10-26 at 17:45 +0000, Jason Spittel wrote:
>         Hello,
> 
>            We are currently experiencing an OutOfMemoryError / Memory Leak on our Keycloak servers. This occurs intermittently within a span of a few weeks to months between incidents. When it does happen, the entire server is brought down.
> 
>            It's a very small load, less than 3000 users, with default settings across the board. One of the keycloak servers is an identity broker, and the other is an IdP that points to the broker (behind the broker is our actual application).
> 
>            Looking at JVM logs, the memory is GC'ed regularly with no long term increase, then suddenly, over a period of 5 minutes, spikes to beyond what is allocated to the server (2GB).
> 
>            We ran the Eclipse Memory Analyser against the .hprof file and found this as the memory leak suspect:
> 
> 
>         default I/O-4
>             at java.lang.OutOfMemoryError.<init>()V (OutOfMemoryError.java:48)
>             at java.util.ArrayDeque.doubleCapacity()V (ArrayDeque.java:162)
>             at java.util.ArrayDeque.addLast(Ljava/lang/Object;)V (ArrayDeque.java:252)
>             at java.util.ArrayDeque.add(Ljava/lang/Object;)Z (ArrayDeque.java:423)
>             at org.xnio.nio.WorkerThread.execute(Ljava/lang/Runnable;)V (WorkerThread.java:591)
>             at io.undertow.protocols.ssl.SslConduit.runReadListener(Z)V (SslConduit.java:223)
>             at io.undertow.protocols.ssl.SslConduit.access$1300(Lio/undertow/protocols/ssl/SslConduit;Z)V (SslConduit.java:63)
>             at io.undertow.protocols.ssl.SslConduit$SslReadReadyHandler.readReady()V (SslConduit.java:1081)
>             at io.undertow.protocols.ssl.SslConduit$1.run()V (SslConduit.java:229)
>             at org.xnio.nio.WorkerThread.safeRun(Ljava/lang/Runnable;)V (WorkerThread.java:580)
>             at org.xnio.nio.WorkerThread.run()V (WorkerThread.java:464)
> 
> 
>             Which seems related to this bug:
> 
>                 https://stackoverflow.com/questions/43661909/keycloak-1-9-4-using-custom-federation-running-out-off-memory
> 
>             The dev in that situation put Apache in front of keycloak to handle the SSL and seemed to resolve the issue. We'd prefer not to do this. Following this SO post to the mailing list thread:
> 
>                 http://lists.jboss.org/pipermail/keycloak-user/2016-June/006771.html
> 
>             There was some interest in the bug but it was then was abandoned.
> 
>             Now, we are running an older version of Keycloak , 3.1.0.Final. But I looked through all the change logs from 3.1.0.Final to 4.5.0.Final as well as all the Jira Issues between those two versions that have to do with SSL, and found no fixes for this issue.
> 
>             Is this a problem that is on the radar of the Keycloak devs? Is this the sort of bugfix that would only be in RH SSO?
> 
>             Thanks,
> 
>             Jason
> 
> [cid:8dad4d85-d402-4612-81a1-ded4d2092813]
> 
> [cid:ba354506-fb8c-46a0-b587-1430e9afe9a2]
> 
> _______________________________________________
> keycloak-user mailing list
> keycloak-user at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user