[keycloak-dev] Clustering and user sessions

Wed Sep 17 09:52:59 EDT 2014


----- Original Message -----
> From: "Marek Posolda" <mposolda at redhat.com>
> To: "Stian Thorgersen" <stian at redhat.com>
> Cc: "Bill Burke" <bburke at redhat.com>, keycloak-dev at lists.jboss.org
> Sent: Wednesday, 17 September, 2014 3:46:21 PM
> Subject: Re: [keycloak-dev] Clustering and user sessions
> 
> On 17.9.2014 15:15, Stian Thorgersen wrote:
> >
> > ----- Original Message -----
> >> From: "Marek Posolda" <mposolda at redhat.com>
> >> To: "Stian Thorgersen" <stian at redhat.com>
> >> Cc: "Bill Burke" <bburke at redhat.com>, keycloak-dev at lists.jboss.org
> >> Sent: Wednesday, 17 September, 2014 3:03:56 PM
> >> Subject: Re: [keycloak-dev] Clustering and user sessions
> >>
> >> On 17.9.2014 13:03, Stian Thorgersen wrote:
> >>> By far the most common request to Keycloak is going to be refreshing the
> >>> token and if we're sending a message to the cluster for every token
> >>> refresh it's just not going to scale very well.
> >> Probably yes. Might be interesting to know the ratio between refresh
> >> requests and other requests in typical deployment though. It also
> >> depends on the environment and apps which will people use for keycloak.
> >> For example I am personally hardly spend more than 5 minutes on any web
> >> page (unless it's real-time work like writing some docs online etc)
> >>> Will it even scale past 1 node?! Think about it, every refresh causes a
> >>> request one node to handle one request from the client and send a message
> >>> to the cluster, while all other nodes have to deal with a message from
> >>> the
> >>> cluster. It's going to increase availability, but not scalability.
> >>>
> >>> So we're going to have to do something smarter than simply sending
> >>> lastSessionRefresh every time.
> >>>
> >>> Following your example I don't think it's a big deal that there's a
> >>> window
> >>> where the session is expired on one node, while not on another. We can
> >>> always come up with a schema to improve that. Two alternatives we could
> >>> do
> >>> if a node detects a session as expired:
> >>>
> >>> 1. Ask the cluster if the session is expired, if at least one node says
> >>> it's not expired it's not
> >>> 2. Tell the cluster that the session is expired
> >> yes, asking the cluster should help to avoid the window. Still the
> >> traffic might be quite big though. Especially with cluster with bigger
> >> number of nodes it could happen quite often that user is redirected
> >> every time on different node and hence it's bigger chance that
> >> UserSession might not be refreshed here. But I agree that it should
> >> likely reduce the traffic in most cases.
> > Question though, for initial clustering support would this inconsistency be
> > OK? I think it would. Also, even if update interval was as low as 1 second
> > it would probably generate much less traffic than one message per
> > lastSessionRefresh. Obviously it would only send an update message if
> > there was at least one session with updated lastSessionRefresh.
> yeah, I think that it's ok as well as long as interval is configurable
> likely with some small value by default. Adding support for "asking
> cluster if session is really expired" shouldn't be so hard to add though..
> >
> >>> Another solution we could consider is sharding. We have one or more front
> >>> servers to the cluster which just delegate to a shard to process a
> >>> request. Each shared handles one set of users and can consist of one or
> >>> more nodes for increased availability. The front servers only need to
> >>> know
> >>> about the shards and members of a shard. This is probably the only
> >>> solution that's going to scale to really big numbers, so might eventually
> >>> be required.
> >> Sharding will be great. Front servers would need to know to which server
> >> to send the request, which might be quite tricky thing though. Doing
> >> sharding based on usernames seems to best as UserSession will be created
> >> in shard group where user login. But in some requests (like refresh
> >> token request) you don't have access to username even in parsed token.
> > Sorry, I didn't mean username I was thinking about user id.
> yeah, userId or sessionId will be great as it's random value and you
> have best shard equality. However I am not sure if you have this info.
> As typical scenario might be:
> * User send request to display KC login page
> (/auth/realms/my-realm/tokens/login) - this is anonymous request, so
> front-server can redirect to any random shard group

No biggy - doesn't need user knowledge so can be sent to any server

> * Once user confirms login form, front-server would need to recognize to
> which shard he send this request (POST to
> /auth/realms/my-realm/tokens/auth/request/login). Once he do it,
> UserSession will be created on this shard so subsequent requests should
> go there. But at the time of sending POST request, userId is not known
> to front-server. Unless front server are so clever, that they can lookup
> KC database (so in the end, they might need to have access to KC
> services too)

Yeah, we would need to be able to somehow lookup userId based on username/email

> >
> >> I wonder if we can add some additional info to requests, which will help
> >> front-servers/loadbalancers to recognize where to resend the request.
> >>
> >> But anyway, I guess it's a priority after we have some basic clustering
> >> support?
> > Absolutely, but it will significantly affect how we design clustering.
> >
> >> Marek
> >>> ----- Original Message -----
> >>>> From: "Marek Posolda" <mposolda at redhat.com>
> >>>> To: "Stian Thorgersen" <stian at redhat.com>, "Bill Burke"
> >>>> <bburke at redhat.com>
> >>>> Cc: keycloak-dev at lists.jboss.org
> >>>> Sent: Wednesday, 17 September, 2014 12:46:33 PM
> >>>> Subject: Re: [keycloak-dev] Clustering and user sessions
> >>>>
> >>>> I am thinking about scenario like this (assuming sessionIdleTimeout is 5
> >>>> minutes and cluster update period is 60 seconds) :
> >>>> * User login at time 0
> >>>> * At time 4:30 he refresh token, which will update lastSessionRefresh on
> >>>> his userSession. This will happen on cluster node1
> >>>> * At time 5:15 he sends another refresh token request, which would be
> >>>> redirected by loadbalancer to node2 this time. Assuming that last
> >>>> cluster update from node1 to node2 happened at 4:20, so next update will
> >>>> happen at 5:20. So ATM node2 will see that session is idle for 5 minutes
> >>>> and 15 seconds (as last refresh at 4:30 is not yet visible to node2). So
> >>>> node2 will logout session due to timeout.
> >>>>
> >>>> So right now it seems safer to me to update lastSessionRefresh
> >>>> immediatelly to cluster. Or am I missing something?
> >>>>
> >>>> I wonder if access to each UserSession can always happen just to same
> >>>> cluster node, but it seems that we won't be able to guarantee this .
> >>>> Even with sticky sessions, the communication from same user (and
> >>>> USerSession) can happen either via browser (SSO login to different
> >>>> application) or via back channel from adapters (refreshing tokens etc)
> >>>> and right now I am not seeing much way to guarantee sticky session
> >>>> between those .
> >>>>
> >>>> Marek
> >>>>
> >>>> On 15.9.2014 14:52, Stian Thorgersen wrote:
> >>>>> ----- Original Message -----
> >>>>>> From: "Bill Burke" <bburke at redhat.com>
> >>>>>> To: keycloak-dev at lists.jboss.org
> >>>>>> Sent: Monday, 15 September, 2014 2:41:39 PM
> >>>>>> Subject: Re: [keycloak-dev] Clustering and user sessions
> >>>>>>
> >>>>>> Only works for session refreshes.  Also leaves an open window that the
> >>>>>> user is still logged in after they log out.
> >>>>> Yes, it's only for session refreshes, but IMO that's going to be the
> >>>>> biggest traffic generator. For login and logouts we're going to have to
> >>>>> send a message per event.
> >>>>>
> >>>>>> On 9/15/2014 8:28 AM, Stian Thorgersen wrote:
> >>>>>>> Had an idea with regards to clustering and user sessions. Instead of
> >>>>>>> sending messages to the cluster when a individual user session is
> >>>>>>> refreshed all nodes send a periodic update message. Obviously that's
> >>>>>>> only
> >>>>>>> for user sessions and not for admin updates, where we should still
> >>>>>>> send
> >>>>>>> invalidation messages for each update.
> >>>>>>>
> >>>>>>> Each node would keep a note of all user sessions where it has updated
> >>>>>>> LastSessionRefresh, and once every period it would send this list to
> >>>>>>> all
> >>>>>>> nodes. This should mean that instead of sending a message every time
> >>>>>>> a
> >>>>>>> single session is updated, we send a single message per node every 60
> >>>>>>> seconds or so (should be configurable). When receiving a message from
> >>>>>>> the
> >>>>>>> cluster the node would go through the list and update the user
> >>>>>>> sessions
> >>>>>>> where the received LastSessionRefresh is higher than the one it has
> >>>>>>> itself. Nodes still use the mem user sessions store, but with the
> >>>>>>> cache
> >>>>>>> on
> >>>>>>> top.
> >>>>>>> _______________________________________________
> >>>>>>> keycloak-dev mailing list
> >>>>>>> keycloak-dev at lists.jboss.org
> >>>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
> >>>>>>>
> >>>>>> --
> >>>>>> Bill Burke
> >>>>>> JBoss, a division of Red Hat
> >>>>>> http://bill.burkecentral.com
> >>>>>> _______________________________________________
> >>>>>> keycloak-dev mailing list
> >>>>>> keycloak-dev at lists.jboss.org
> >>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
> >>>>>>
> >>>>> _______________________________________________
> >>>>> keycloak-dev mailing list
> >>>>> keycloak-dev at lists.jboss.org
> >>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
> >>
> 
>