[keycloak-dev] Clustering and user sessions

Wed Sep 17 09:15:38 EDT 2014

----- Original Message -----
> From: "Marek Posolda" <mposolda at redhat.com>
> To: "Stian Thorgersen" <stian at redhat.com>
> Cc: "Bill Burke" <bburke at redhat.com>, keycloak-dev at lists.jboss.org
> Sent: Wednesday, 17 September, 2014 3:03:56 PM
> Subject: Re: [keycloak-dev] Clustering and user sessions
> 
> On 17.9.2014 13:03, Stian Thorgersen wrote:
> > By far the most common request to Keycloak is going to be refreshing the
> > token and if we're sending a message to the cluster for every token
> > refresh it's just not going to scale very well.
> Probably yes. Might be interesting to know the ratio between refresh
> requests and other requests in typical deployment though. It also
> depends on the environment and apps which will people use for keycloak.
> For example I am personally hardly spend more than 5 minutes on any web
> page (unless it's real-time work like writing some docs online etc)
> >
> > Will it even scale past 1 node?! Think about it, every refresh causes a
> > request one node to handle one request from the client and send a message
> > to the cluster, while all other nodes have to deal with a message from the
> > cluster. It's going to increase availability, but not scalability.
> >
> > So we're going to have to do something smarter than simply sending
> > lastSessionRefresh every time.
> >
> > Following your example I don't think it's a big deal that there's a window
> > where the session is expired on one node, while not on another. We can
> > always come up with a schema to improve that. Two alternatives we could do
> > if a node detects a session as expired:
> >
> > 1. Ask the cluster if the session is expired, if at least one node says
> > it's not expired it's not
> > 2. Tell the cluster that the session is expired
> yes, asking the cluster should help to avoid the window. Still the
> traffic might be quite big though. Especially with cluster with bigger
> number of nodes it could happen quite often that user is redirected
> every time on different node and hence it's bigger chance that
> UserSession might not be refreshed here. But I agree that it should
> likely reduce the traffic in most cases.

Question though, for initial clustering support would this inconsistency be OK? I think it would. Also, even if update interval was as low as 1 second it would probably generate much less traffic than one message per lastSessionRefresh. Obviously it would only send an update message if there was at least one session with updated lastSessionRefresh.

> >
> > Another solution we could consider is sharding. We have one or more front
> > servers to the cluster which just delegate to a shard to process a
> > request. Each shared handles one set of users and can consist of one or
> > more nodes for increased availability. The front servers only need to know
> > about the shards and members of a shard. This is probably the only
> > solution that's going to scale to really big numbers, so might eventually
> > be required.
> Sharding will be great. Front servers would need to know to which server
> to send the request, which might be quite tricky thing though. Doing
> sharding based on usernames seems to best as UserSession will be created
> in shard group where user login. But in some requests (like refresh
> token request) you don't have access to username even in parsed token.

Sorry, I didn't mean username I was thinking about user id.

> 
> I wonder if we can add some additional info to requests, which will help
> front-servers/loadbalancers to recognize where to resend the request.
> 
> But anyway, I guess it's a priority after we have some basic clustering
> support?

Absolutely, but it will significantly affect how we design clustering.

> 
> Marek
> >
> > ----- Original Message -----
> >> From: "Marek Posolda" <mposolda at redhat.com>
> >> To: "Stian Thorgersen" <stian at redhat.com>, "Bill Burke"
> >> <bburke at redhat.com>
> >> Cc: keycloak-dev at lists.jboss.org
> >> Sent: Wednesday, 17 September, 2014 12:46:33 PM
> >> Subject: Re: [keycloak-dev] Clustering and user sessions
> >>
> >> I am thinking about scenario like this (assuming sessionIdleTimeout is 5
> >> minutes and cluster update period is 60 seconds) :
> >> * User login at time 0
> >> * At time 4:30 he refresh token, which will update lastSessionRefresh on
> >> his userSession. This will happen on cluster node1
> >> * At time 5:15 he sends another refresh token request, which would be
> >> redirected by loadbalancer to node2 this time. Assuming that last
> >> cluster update from node1 to node2 happened at 4:20, so next update will
> >> happen at 5:20. So ATM node2 will see that session is idle for 5 minutes
> >> and 15 seconds (as last refresh at 4:30 is not yet visible to node2). So
> >> node2 will logout session due to timeout.
> >>
> >> So right now it seems safer to me to update lastSessionRefresh
> >> immediatelly to cluster. Or am I missing something?
> >>
> >> I wonder if access to each UserSession can always happen just to same
> >> cluster node, but it seems that we won't be able to guarantee this .
> >> Even with sticky sessions, the communication from same user (and
> >> USerSession) can happen either via browser (SSO login to different
> >> application) or via back channel from adapters (refreshing tokens etc)
> >> and right now I am not seeing much way to guarantee sticky session
> >> between those .
> >>
> >> Marek
> >>
> >> On 15.9.2014 14:52, Stian Thorgersen wrote:
> >>> ----- Original Message -----
> >>>> From: "Bill Burke" <bburke at redhat.com>
> >>>> To: keycloak-dev at lists.jboss.org
> >>>> Sent: Monday, 15 September, 2014 2:41:39 PM
> >>>> Subject: Re: [keycloak-dev] Clustering and user sessions
> >>>>
> >>>> Only works for session refreshes.  Also leaves an open window that the
> >>>> user is still logged in after they log out.
> >>> Yes, it's only for session refreshes, but IMO that's going to be the
> >>> biggest traffic generator. For login and logouts we're going to have to
> >>> send a message per event.
> >>>
> >>>> On 9/15/2014 8:28 AM, Stian Thorgersen wrote:
> >>>>> Had an idea with regards to clustering and user sessions. Instead of
> >>>>> sending messages to the cluster when a individual user session is
> >>>>> refreshed all nodes send a periodic update message. Obviously that's
> >>>>> only
> >>>>> for user sessions and not for admin updates, where we should still send
> >>>>> invalidation messages for each update.
> >>>>>
> >>>>> Each node would keep a note of all user sessions where it has updated
> >>>>> LastSessionRefresh, and once every period it would send this list to
> >>>>> all
> >>>>> nodes. This should mean that instead of sending a message every time a
> >>>>> single session is updated, we send a single message per node every 60
> >>>>> seconds or so (should be configurable). When receiving a message from
> >>>>> the
> >>>>> cluster the node would go through the list and update the user sessions
> >>>>> where the received LastSessionRefresh is higher than the one it has
> >>>>> itself. Nodes still use the mem user sessions store, but with the cache
> >>>>> on
> >>>>> top.
> >>>>> _______________________________________________
> >>>>> keycloak-dev mailing list
> >>>>> keycloak-dev at lists.jboss.org
> >>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
> >>>>>
> >>>> --
> >>>> Bill Burke
> >>>> JBoss, a division of Red Hat
> >>>> http://bill.burkecentral.com
> >>>> _______________________________________________
> >>>> keycloak-dev mailing list
> >>>> keycloak-dev at lists.jboss.org
> >>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
> >>>>
> >>> _______________________________________________
> >>> keycloak-dev mailing list
> >>> keycloak-dev at lists.jboss.org
> >>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
> >>
> 
> 