Hello Jason,
The problem seems to be in the SSL stack, which is not a part of Keycloak itself. Keycloak
is built on top of Wildfly application server, and SSL is implemented by one of Wildfly
components, namely Undertow.
You seem to be hitting this bug:
https://issues.jboss.org/browse/UNDERTOW-472
Though JIRA says that it should have been fixed in Undertow 1.3.10, the version shipped
with Keycloak 3.1.0 still seems to be buggy (1.3.15).
Either way, it is highly recommended that you upgrade to the recent Keycloak that uses
up-to-date Wildfly (and therefore Undertow).
Cheers,
Dmitry Telegin
CTO, Acutus s.r.o.
Keycloak Consulting and Training
Pod lipami street 339/52, 130 00 Prague 3, Czech Republic
+42 (022) 888-30-71
E-mail: info(a)acutus.pro
On Fri, 2018-10-26 at 17:45 +0000, Jason Spittel wrote:
Hello,
We are currently experiencing an OutOfMemoryError / Memory Leak on our
Keycloak servers. This occurs intermittently within a span of a few weeks to months
between incidents. When it does happen, the entire server is brought down.
It's a very small load, less than 3000 users, with default settings across
the board. One of the keycloak servers is an identity broker, and the other is an IdP that
points to the broker (behind the broker is our actual application).
Looking at JVM logs, the memory is GC'ed regularly with no long term
increase, then suddenly, over a period of 5 minutes, spikes to beyond what is allocated to
the server (2GB).
We ran the Eclipse Memory Analyser against the .hprof file and found this as
the memory leak suspect:
default I/O-4
at java.lang.OutOfMemoryError.<init>()V (OutOfMemoryError.java:48)
at java.util.ArrayDeque.doubleCapacity()V (ArrayDeque.java:162)
at java.util.ArrayDeque.addLast(Ljava/lang/Object;)V (ArrayDeque.java:252)
at java.util.ArrayDeque.add(Ljava/lang/Object;)Z (ArrayDeque.java:423)
at org.xnio.nio.WorkerThread.execute(Ljava/lang/Runnable;)V
(WorkerThread.java:591)
at io.undertow.protocols.ssl.SslConduit.runReadListener(Z)V
(SslConduit.java:223)
at
io.undertow.protocols.ssl.SslConduit.access$1300(Lio/undertow/protocols/ssl/SslConduit;Z)V
(SslConduit.java:63)
at io.undertow.protocols.ssl.SslConduit$SslReadReadyHandler.readReady()V
(SslConduit.java:1081)
at io.undertow.protocols.ssl.SslConduit$1.run()V (SslConduit.java:229)
at org.xnio.nio.WorkerThread.safeRun(Ljava/lang/Runnable;)V
(WorkerThread.java:580)
at org.xnio.nio.WorkerThread.run()V (WorkerThread.java:464)
Which seems related to this bug:
https://stackoverflow.com/questions/43661909/keycloak-1-9-4-using-custom-federation-running-out-off-memory
The dev in that situation put Apache in front of keycloak to handle the SSL
and seemed to resolve the issue. We'd prefer not to do this. Following this SO post to
the mailing list thread:
http://lists.jboss.org/pipermail/keycloak-user/2016-June/006771.html
There was some interest in the bug but it was then was abandoned.
Now, we are running an older version of Keycloak , 3.1.0.Final. But I looked
through all the change logs from 3.1.0.Final to 4.5.0.Final as well as all the Jira Issues
between those two versions that have to do with SSL, and found no fixes for this issue.
Is this a problem that is on the radar of the Keycloak devs? Is this the sort
of bugfix that would only be in RH SSO?
Thanks,
Jason
[cid:8dad4d85-d402-4612-81a1-ded4d2092813]
[cid:ba354506-fb8c-46a0-b587-1430e9afe9a2]
_______________________________________________
keycloak-user mailing list
keycloak-user(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-user