[keycloak-dev] Clustering and user sessions

[Stian Thorgersen]

By far the most common request to Keycloak is going to be refreshing the token and if we're sending a message to the cluster for every token refresh it's just not going to scale very well. 

Will it even scale past 1 node?! Think about it, every refresh causes a request one node to handle one request from the client and send a message to the cluster, while all other nodes have to deal with a message from the cluster. It's going to increase availability, but not scalability.

So we're going to have to do something smarter than simply sending lastSessionRefresh every time.

Following your example I don't think it's a big deal that there's a window where the session is expired on one node, while not on another. We can always come up with a schema to improve that. Two alternatives we could do if a node detects a session as expired:

1. Ask the cluster if the session is expired, if at least one node says it's not expired it's not
2. Tell the cluster that the session is expired

Another solution we could consider is sharding. We have one or more front servers to the cluster which just delegate to a shard to process a request. Each shared handles one set of users and can consist of one or more nodes for increased availability. The front servers only need to know about the shards and members of a shard. This is probably the only solution that's going to scale to really big numbers, so might eventually be required.

----- Original Message -----
> From: "Marek Posolda" <mposolda at redhat.com>
> To: "Stian Thorgersen" <stian at redhat.com>, "Bill Burke" <bburke at redhat.com>
> Cc: keycloak-dev at lists.jboss.org
> Sent: Wednesday, 17 September, 2014 12:46:33 PM
> Subject: Re: [keycloak-dev] Clustering and user sessions
> 
> I am thinking about scenario like this (assuming sessionIdleTimeout is 5
> minutes and cluster update period is 60 seconds) :
> * User login at time 0
> * At time 4:30 he refresh token, which will update lastSessionRefresh on
> his userSession. This will happen on cluster node1
> * At time 5:15 he sends another refresh token request, which would be
> redirected by loadbalancer to node2 this time. Assuming that last
> cluster update from node1 to node2 happened at 4:20, so next update will
> happen at 5:20. So ATM node2 will see that session is idle for 5 minutes
> and 15 seconds (as last refresh at 4:30 is not yet visible to node2). So
> node2 will logout session due to timeout.
> 
> So right now it seems safer to me to update lastSessionRefresh
> immediatelly to cluster. Or am I missing something?
> 
> I wonder if access to each UserSession can always happen just to same
> cluster node, but it seems that we won't be able to guarantee this .
> Even with sticky sessions, the communication from same user (and
> USerSession) can happen either via browser (SSO login to different
> application) or via back channel from adapters (refreshing tokens etc)
> and right now I am not seeing much way to guarantee sticky session
> between those .
> 
> Marek
> 
> On 15.9.2014 14:52, Stian Thorgersen wrote:
> >
> > ----- Original Message -----
> >> From: "Bill Burke" <bburke at redhat.com>
> >> To: keycloak-dev at lists.jboss.org
> >> Sent: Monday, 15 September, 2014 2:41:39 PM
> >> Subject: Re: [keycloak-dev] Clustering and user sessions
> >>
> >> Only works for session refreshes.  Also leaves an open window that the
> >> user is still logged in after they log out.
> > Yes, it's only for session refreshes, but IMO that's going to be the
> > biggest traffic generator. For login and logouts we're going to have to
> > send a message per event.
> >
> >> On 9/15/2014 8:28 AM, Stian Thorgersen wrote:
> >>> Had an idea with regards to clustering and user sessions. Instead of
> >>> sending messages to the cluster when a individual user session is
> >>> refreshed all nodes send a periodic update message. Obviously that's only
> >>> for user sessions and not for admin updates, where we should still send
> >>> invalidation messages for each update.
> >>>
> >>> Each node would keep a note of all user sessions where it has updated
> >>> LastSessionRefresh, and once every period it would send this list to all
> >>> nodes. This should mean that instead of sending a message every time a
> >>> single session is updated, we send a single message per node every 60
> >>> seconds or so (should be configurable). When receiving a message from the
> >>> cluster the node would go through the list and update the user sessions
> >>> where the received LastSessionRefresh is higher than the one it has
> >>> itself. Nodes still use the mem user sessions store, but with the cache
> >>> on
> >>> top.
> >>> _______________________________________________
> >>> keycloak-dev mailing list
> >>> keycloak-dev at lists.jboss.org
> >>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
> >>>
> >> --
> >> Bill Burke
> >> JBoss, a division of Red Hat
> >> http://bill.burkecentral.com
> >> _______________________________________________
> >> keycloak-dev mailing list
> >> keycloak-dev at lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/keycloak-dev
> >>
> > _______________________________________________
> > keycloak-dev mailing list
> > keycloak-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/keycloak-dev
> 
>