4:06 p.m.
In our deployments, we have need for the individual nodes in our standalone-ha cluster to
be able to be stopped and restarted at any time, without loss of user sessions. We deploy
into AWS Elastic Beanstalk, and join the "new" nodes into the existing JGroups
cluster of "old" nodes, switch CNAMEs, then shut down the "old" nodes.
The default suggested model of using <distributed-cache> for the session and
clientSession caches fails in this scenario, as sessions stored only in memory on the
"old" nodes are lost. As such, we've switched to <replicated-cache>
for most of the caches. Does this make sense? Are most people using Keycloak running
>10 nodes per cluster such that replicated cache doesn't make sense for performance
reasons? If not, can the example standalone-ha.xml be adjusted to use replicated-cache?
When items are placed into the caches, it seems that they are not put() with a lifetime or
expiry time associated with them. This has led us to run into out-of-memory issues as the
sizes of these caches grows without bound. We've added expiration times in the XML
config to adjust for maximum lifetimes based on the longest session timeouts across all
realms. This seems not ideal. It would be better to put() including the current active
session timeouts for the realm for the object from which it originates.
We cannot evict items in the sessions and clientSessions caches as they are not persisted
to disk. The Offline variants are persisted to disk, so evicting these seem OK. Should
this then be an invalidation-cache rather than a replicated-cache?
XML snippet:
<subsystem xmlns="urn:jboss:domain:infinispan:4.0">
<cache-container name="keycloak"
jndi-name="infinispan/Keycloak" statistics-enabled="true">
<replicated-cache name="sessions" mode="SYNC"
statistics-enabled="true">
<!-- NOT persisted to the database, so cannot evict within their
valid lifetime -->
<!-- max-idle = refresh_token:1 day -->
<expiration max-idle="86700000"
interval="300000"/>
</replicated-cache>
<replicated-cache name="clientSessions" mode="SYNC"
statistics-enabled="true">
<!-- NOT persisted to the database, so cannot evict within their
valid lifetime -->
<!-- max-idle = refresh_token:1 day -->
<expiration max-idle="86700000"
interval="300000"/>
</replicated-cache>
<replicated-cache name="authenticationSessions"
mode="SYNC" statistics-enabled="true">
<!-- NOT persisted to the database, so cannot evict within their
valid lifetime -->
<!-- max-idle = login timeout:1 day -->
<expiration max-idle="86700000"
interval="300000"/>
</replicated-cache>
<replicated-cache name="offlineSessions"
mode="SYNC" statistics-enabled="true">
<!-- max-idle = refresh_token:30 days -->
<eviction max-entries="10000"
strategy="LRU"/>
<expiration max-idle="2592000000"
interval="300000"/>
</replicated-cache>
<replicated-cache name="offlineClientSessions"
mode="SYNC" statistics-enabled="true">
<eviction max-entries="10000"
strategy="LRU"/>
<expiration max-idle="2592000000"
interval="300000"/>
</replicated-cache>
Thanks,
Matt
--
Matt Domsch
Executive Director & Senior Distinguished Engineer
Quest | Engineering
Matt.Domsch@quest.com<mailto:Matt.Domsch@quest.com>
Mobile 512.981.6486
1:19 p.m.
While we're at it, shouldn't the realm, user, and key caches all be invalidation
caches too, with a limit on the number of objects in each? The objects are persisted to
disk. In our case, I've got >85k user records, and suspect that the <local>
user cache is growing unbounded, hence I'm seeing steadily increasing memory usage.
Thanks,
Matt
--
Matt Domsch
Executive Director & Senior Distinguished Engineer
Quest | Engineering
Matt.Domsch(a)quest.com
Mobile 512.981.6486
-----Original Message-----
From: keycloak-dev-bounces(a)lists.jboss.org [mailto:keycloak-dev-bounces@lists.jboss.org]
On Behalf Of Matt Domsch (mdomsch)
Sent: Thursday, May 31, 2018 4:06 PM
To: keycloak-dev(a)lists.jboss.org
Subject: [keycloak-dev] session caching questions
CAUTION: This email originated from outside of the organization. Do not click links or
open attachments unless you recognize the sender and know the content is safe.
In our deployments, we have need for the individual nodes in our standalone-ha cluster to
be able to be stopped and restarted at any time, without loss of user sessions. We deploy
into AWS Elastic Beanstalk, and join the "new" nodes into the existing JGroups
cluster of "old" nodes, switch CNAMEs, then shut down the "old" nodes.
The default suggested model of using <distributed-cache> for the session and
clientSession caches fails in this scenario, as sessions stored only in memory on the
"old" nodes are lost. As such, we've switched to <replicated-cache>
for most of the caches. Does this make sense? Are most people using Keycloak running
>10 nodes per cluster such that replicated cache doesn't make sense for performance
reasons? If not, can the example standalone-ha.xml be adjusted to use replicated-cache?
When items are placed into the caches, it seems that they are not put() with a lifetime or
expiry time associated with them. This has led us to run into out-of-memory issues as the
sizes of these caches grows without bound. We've added expiration times in the XML
config to adjust for maximum lifetimes based on the longest session timeouts across all
realms. This seems not ideal. It would be better to put() including the current active
session timeouts for the realm for the object from which it originates.
We cannot evict items in the sessions and clientSessions caches as they are not persisted
to disk. The Offline variants are persisted to disk, so evicting these seem OK. Should
this then be an invalidation-cache rather than a replicated-cache?
XML snippet:
<subsystem xmlns="urn:jboss:domain:infinispan:4.0">
<cache-container name="keycloak"
jndi-name="infinispan/Keycloak" statistics-enabled="true">
<replicated-cache name="sessions" mode="SYNC"
statistics-enabled="true">
<!-- NOT persisted to the database, so cannot evict within their
valid lifetime -->
<!-- max-idle = refresh_token:1 day -->
<expiration max-idle="86700000"
interval="300000"/>
</replicated-cache>
<replicated-cache name="clientSessions" mode="SYNC"
statistics-enabled="true">
<!-- NOT persisted to the database, so cannot evict within their
valid lifetime -->
<!-- max-idle = refresh_token:1 day -->
<expiration max-idle="86700000"
interval="300000"/>
</replicated-cache>
<replicated-cache name="authenticationSessions"
mode="SYNC" statistics-enabled="true">
<!-- NOT persisted to the database, so cannot evict within their
valid lifetime -->
<!-- max-idle = login timeout:1 day -->
<expiration max-idle="86700000"
interval="300000"/>
</replicated-cache>
<replicated-cache name="offlineSessions"
mode="SYNC" statistics-enabled="true">
<!-- max-idle = refresh_token:30 days -->
<eviction max-entries="10000"
strategy="LRU"/>
<expiration max-idle="2592000000"
interval="300000"/>
</replicated-cache>
<replicated-cache name="offlineClientSessions"
mode="SYNC" statistics-enabled="true">
<eviction max-entries="10000"
strategy="LRU"/>
<expiration max-idle="2592000000"
interval="300000"/>
</replicated-cache>
Thanks,
Matt
--
Matt Domsch
Executive Director & Senior Distinguished Engineer Quest | Engineering
Matt.Domsch@quest.com<mailto:Matt.Domsch@quest.com>
Mobile 512.981.6486
_______________________________________________
keycloak-dev mailing list
keycloak-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-dev
3:48 p.m.
Hi Matt,
I’m not an active Keycloak dev but we used to run our Keycloak cluster on AWS Beanstalk.
My advice would be to not use AWS Beanstalk for Keycloak. AWS Beanstalk is OK for simple
applications but we had to write custom Beanstalk hooks to get Infinspan to behave
correctly. I would suggest using AWS Elastic Cluster Service (ECS) and if you don’t want
to manage the instances use Fargate instead of EC2.
Just to be clear: this isn’t a criticism of either Keycloak / WildFily / Infinispan or AWS
Beanstalk. It’s just advice from another engineer who has experience with the combination
of these two products. Since moving to to ECS we haven’t had any issues with the cache
during deployments.
As for your suggested configuration: I don’t think the user, realm and key caches should
be changed. These caches are not an issue during deployments and they reduce DB load
significantly the way they’re configured out of the box.
Best,
Scott
Scott Rossillo
Smartling | Senior Software Engineer
srossillo(a)smartling.com
On Jun 4, 2018, at 2:19 PM, Matt Domsch (mdomsch)
<Matt.Domsch(a)quest.com> wrote:
While we're at it, shouldn't the realm, user, and key caches all be invalidation
caches too, with a limit on the number of objects in each? The objects are persisted to
disk. In our case, I've got >85k user records, and suspect that the <local>
user cache is growing unbounded, hence I'm seeing steadily increasing memory usage.
Thanks,
Matt
--
Matt Domsch
Executive Director & Senior Distinguished Engineer
Quest | Engineering
Matt.Domsch(a)quest.com
Mobile 512.981.6486
-----Original Message-----
From: keycloak-dev-bounces(a)lists.jboss.org [mailto:keycloak-dev-bounces@lists.jboss.org]
On Behalf Of Matt Domsch (mdomsch)
Sent: Thursday, May 31, 2018 4:06 PM
To: keycloak-dev(a)lists.jboss.org
Subject: [keycloak-dev] session caching questions
CAUTION: This email originated from outside of the organization. Do not click links or
open attachments unless you recognize the sender and know the content is safe.
In our deployments, we have need for the individual nodes in our standalone-ha cluster to
be able to be stopped and restarted at any time, without loss of user sessions. We deploy
into AWS Elastic Beanstalk, and join the "new" nodes into the existing JGroups
cluster of "old" nodes, switch CNAMEs, then shut down the "old" nodes.
The default suggested model of using <distributed-cache> for the session and
clientSession caches fails in this scenario, as sessions stored only in memory on the
"old" nodes are lost. As such, we've switched to <replicated-cache>
for most of the caches. Does this make sense? Are most people using Keycloak running
>10 nodes per cluster such that replicated cache doesn't make sense for performance
reasons? If not, can the example standalone-ha.xml be adjusted to use replicated-cache?
When items are placed into the caches, it seems that they are not put() with a lifetime
or expiry time associated with them. This has led us to run into out-of-memory issues as
the sizes of these caches grows without bound. We've added expiration times in the
XML config to adjust for maximum lifetimes based on the longest session timeouts across
all realms. This seems not ideal. It would be better to put() including the current
active session timeouts for the realm for the object from which it originates.
We cannot evict items in the sessions and clientSessions caches as they are not persisted
to disk. The Offline variants are persisted to disk, so evicting these seem OK. Should
this then be an invalidation-cache rather than a replicated-cache?
XML snippet:
<subsystem xmlns="urn:jboss:domain:infinispan:4.0">
<cache-container name="keycloak"
jndi-name="infinispan/Keycloak" statistics-enabled="true">
<replicated-cache name="sessions" mode="SYNC"
statistics-enabled="true">
<!-- NOT persisted to the database, so cannot evict within their
valid lifetime -->
<!-- max-idle = refresh_token:1 day -->
<expiration max-idle="86700000"
interval="300000"/>
</replicated-cache>
<replicated-cache name="clientSessions" mode="SYNC"
statistics-enabled="true">
<!-- NOT persisted to the database, so cannot evict within their
valid lifetime -->
<!-- max-idle = refresh_token:1 day -->
<expiration max-idle="86700000"
interval="300000"/>
</replicated-cache>
<replicated-cache name="authenticationSessions"
mode="SYNC" statistics-enabled="true">
<!-- NOT persisted to the database, so cannot evict within their
valid lifetime -->
<!-- max-idle = login timeout:1 day -->
<expiration max-idle="86700000"
interval="300000"/>
</replicated-cache>
<replicated-cache name="offlineSessions"
mode="SYNC" statistics-enabled="true">
<!-- max-idle = refresh_token:30 days -->
<eviction max-entries="10000"
strategy="LRU"/>
<expiration max-idle="2592000000"
interval="300000"/>
</replicated-cache>
<replicated-cache name="offlineClientSessions"
mode="SYNC" statistics-enabled="true">
<eviction max-entries="10000"
strategy="LRU"/>
<expiration max-idle="2592000000"
interval="300000"/>
</replicated-cache>
Thanks,
Matt
--
Matt Domsch
Executive Director & Senior Distinguished Engineer Quest | Engineering
Matt.Domsch@quest.com<mailto:Matt.Domsch@quest.com>
Mobile 512.981.6486
_______________________________________________
keycloak-dev mailing list
keycloak-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-dev
_______________________________________________
keycloak-dev mailing list
keycloak-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-dev
2:16 a.m.
On 31/05/18 23:06, Matt Domsch (mdomsch) wrote:
In our deployments, we have need for the individual nodes in our
standalone-ha cluster to be able to be stopped and restarted at any time, without loss of
user sessions. We deploy into AWS Elastic Beanstalk, and join the "new" nodes
into the existing JGroups cluster of "old" nodes, switch CNAMEs, then shut down
the "old" nodes. The default suggested model of using <distributed-cache>
for the session and clientSession caches fails in this scenario, as sessions stored only
in memory on the "old" nodes are lost. As such, we've switched to
<replicated-cache> for most of the caches. Does this make sense? Are most people
using Keycloak running >10 nodes per cluster such that replicated cache doesn't
make sense for performance reasons? If not, can the example standalone-ha.xml be adjusted
to use replicated-cache?
You can keep the distributed-cache, but increase the count
of owners to
2 or more instead of default 1. Maybe we can clarify this more clearly
in the docs... Replicated-cache is defacto just distributed cache with
owners == count-of-nodes-in-cluster . It is safe that data won't be ever
lost (unless you stop whole cluster), but memory consumption and
performance is not ideal.
When items are placed into the caches, it seems that they are not put() with a lifetime
or expiry time associated with them. This has led us to run into out-of-memory issues as
the sizes of these caches grows without bound. We've added expiration times in the
XML config to adjust for maximum lifetimes based on the longest session timeouts across
all realms. This seems not ideal. It would be better to put() including the current
active session timeouts for the realm for the object from which it originates.
The
configuration in the standalone.xml looks like this, so there should
be eviction enabled by default on the caches. By default it's like this:
<local-cache name="realms">
<eviction max-entries="10000"
strategy="LRU"/>
</local-cache>
<local-cache name="users">
<eviction max-entries="10000"
strategy="LRU"/>
</local-cache>
Did you change the default configuration and removed the eviction or
expiration by accident?
We cannot evict items in the sessions and clientSessions caches as they are not persisted
to disk. The Offline variants are persisted to disk, so evicting these seem OK. Should
this then be an invalidation-cache rather than a replicated-cache?
We have
background thread in KEycloak, which automatically removes all
the expired userSessions and clientSessions. Expired depends on "Max
Session Lifespan" (10 hours by default) and "Max Idle Lifespan (30
minutes by default).
Marek
XML snippet:
<subsystem xmlns="urn:jboss:domain:infinispan:4.0">
<cache-container name="keycloak"
jndi-name="infinispan/Keycloak" statistics-enabled="true">
<replicated-cache name="sessions" mode="SYNC"
statistics-enabled="true">
<!-- NOT persisted to the database, so cannot evict within their
valid lifetime -->
<!-- max-idle = refresh_token:1 day -->
<expiration max-idle="86700000"
interval="300000"/>
</replicated-cache>
<replicated-cache name="clientSessions"
mode="SYNC" statistics-enabled="true">
<!-- NOT persisted to the database, so cannot evict within their
valid lifetime -->
<!-- max-idle = refresh_token:1 day -->
<expiration max-idle="86700000"
interval="300000"/>
</replicated-cache>
<replicated-cache name="authenticationSessions"
mode="SYNC" statistics-enabled="true">
<!-- NOT persisted to the database, so cannot evict within their
valid lifetime -->
<!-- max-idle = login timeout:1 day -->
<expiration max-idle="86700000"
interval="300000"/>
</replicated-cache>
<replicated-cache name="offlineSessions"
mode="SYNC" statistics-enabled="true">
<!-- max-idle = refresh_token:30 days -->
<eviction max-entries="10000"
strategy="LRU"/>
<expiration max-idle="2592000000"
interval="300000"/>
</replicated-cache>
<replicated-cache name="offlineClientSessions"
mode="SYNC" statistics-enabled="true">
<eviction max-entries="10000"
strategy="LRU"/>
<expiration max-idle="2592000000"
interval="300000"/>
</replicated-cache>
Thanks,
Matt
--
Matt Domsch
Executive Director & Senior Distinguished Engineer
Quest | Engineering
Matt.Domsch@quest.com<mailto:Matt.Domsch@quest.com>
Mobile 512.981.6486
_______________________________________________
keycloak-dev mailing list
keycloak-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-dev
2278
days inactive
2284
days old
3 comments
3 participants
participants (3)
-
Marek Posolda -
Matt Domsch (mdomsch) -
Scott Rossillo