[keycloak-dev] Clustering and user sessions

[Marek Posolda]

On 17.9.2014 13:03, Stian Thorgersen wrote:
> By far the most common request to Keycloak is going to be refreshing the token and if we're sending a message to the cluster for every token refresh it's just not going to scale very well.
Probably yes. Might be interesting to know the ratio between refresh 
requests and other requests in typical deployment though. It also 
depends on the environment and apps which will people use for keycloak. 
For example I am personally hardly spend more than 5 minutes on any web 
page (unless it's real-time work like writing some docs online etc)
>
> Will it even scale past 1 node?! Think about it, every refresh causes a request one node to handle one request from the client and send a message to the cluster, while all other nodes have to deal with a message from the cluster. It's going to increase availability, but not scalability.
>
> So we're going to have to do something smarter than simply sending lastSessionRefresh every time.
>
> Following your example I don't think it's a big deal that there's a window where the session is expired on one node, while not on another. We can always come up with a schema to improve that. Two alternatives we could do if a node detects a session as expired:
>
> 1. Ask the cluster if the session is expired, if at least one node says it's not expired it's not
> 2. Tell the cluster that the session is expired
yes, asking the cluster should help to avoid the window. Still the 
traffic might be quite big though. Especially with cluster with bigger 
number of nodes it could happen quite often that user is redirected 
every time on different node and hence it's bigger chance that 
UserSession might not be refreshed here. But I agree that it should 
likely reduce the traffic in most cases.
>
> Another solution we could consider is sharding. We have one or more front servers to the cluster which just delegate to a shard to process a request. Each shared handles one set of users and can consist of one or more nodes for increased availability. The front servers only need to know about the shards and members of a shard. This is probably the only solution that's going to scale to really big numbers, so might eventually be required.
Sharding will be great. Front servers would need to know to which server 
to send the request, which might be quite tricky thing though. Doing 
sharding based on usernames seems to best as UserSession will be created 
in shard group where user login. But in some requests (like refresh 
token request) you don't have access to username even in parsed token.

I wonder if we can add some additional info to requests, which will help 
front-servers/loadbalancers to recognize where to resend the request.

But anyway, I guess it's a priority after we have some basic clustering 
support?

Marek
>
> ----- Original Message -----
>> From: "Marek Posolda" <mposolda at redhat.com>
>> To: "Stian Thorgersen" <stian at redhat.com>, "Bill Burke" <bburke at redhat.com>
>> Cc: keycloak-dev at lists.jboss.org
>> Sent: Wednesday, 17 September, 2014 12:46:33 PM
>> Subject: Re: [keycloak-dev] Clustering and user sessions
>>
>> I am thinking about scenario like this (assuming sessionIdleTimeout is 5
>> minutes and cluster update period is 60 seconds) :
>> * User login at time 0
>> * At time 4:30 he refresh token, which will update lastSessionRefresh on
>> his userSession. This will happen on cluster node1
>> * At time 5:15 he sends another refresh token request, which would be
>> redirected by loadbalancer to node2 this time. Assuming that last
>> cluster update from node1 to node2 happened at 4:20, so next update will
>> happen at 5:20. So ATM node2 will see that session is idle for 5 minutes
>> and 15 seconds (as last refresh at 4:30 is not yet visible to node2). So
>> node2 will logout session due to timeout.
>>
>> So right now it seems safer to me to update lastSessionRefresh
>> immediatelly to cluster. Or am I missing something?
>>
>> I wonder if access to each UserSession can always happen just to same
>> cluster node, but it seems that we won't be able to guarantee this .
>> Even with sticky sessions, the communication from same user (and
>> USerSession) can happen either via browser (SSO login to different
>> application) or via back channel from adapters (refreshing tokens etc)
>> and right now I am not seeing much way to guarantee sticky session
>> between those .
>>
>> Marek
>>
>> On 15.9.2014 14:52, Stian Thorgersen wrote:
>>> ----- Original Message -----
>>>> From: "Bill Burke" <bburke at redhat.com>
>>>> To: keycloak-dev at lists.jboss.org
>>>> Sent: Monday, 15 September, 2014 2:41:39 PM
>>>> Subject: Re: [keycloak-dev] Clustering and user sessions
>>>>
>>>> Only works for session refreshes.  Also leaves an open window that the
>>>> user is still logged in after they log out.
>>> Yes, it's only for session refreshes, but IMO that's going to be the
>>> biggest traffic generator. For login and logouts we're going to have to
>>> send a message per event.
>>>
>>>> On 9/15/2014 8:28 AM, Stian Thorgersen wrote:
>>>>> Had an idea with regards to clustering and user sessions. Instead of
>>>>> sending messages to the cluster when a individual user session is
>>>>> refreshed all nodes send a periodic update message. Obviously that's only
>>>>> for user sessions and not for admin updates, where we should still send
>>>>> invalidation messages for each update.
>>>>>
>>>>> Each node would keep a note of all user sessions where it has updated
>>>>> LastSessionRefresh, and once every period it would send this list to all
>>>>> nodes. This should mean that instead of sending a message every time a
>>>>> single session is updated, we send a single message per node every 60
>>>>> seconds or so (should be configurable). When receiving a message from the
>>>>> cluster the node would go through the list and update the user sessions
>>>>> where the received LastSessionRefresh is higher than the one it has
>>>>> itself. Nodes still use the mem user sessions store, but with the cache
>>>>> on
>>>>> top.
>>>>> _______________________________________________
>>>>> keycloak-dev mailing list
>>>>> keycloak-dev at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>>>
>>>> --
>>>> Bill Burke
>>>> JBoss, a division of Red Hat
>>>> http://bill.burkecentral.com
>>>> _______________________________________________
>>>> keycloak-dev mailing list
>>>> keycloak-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>>
>>> _______________________________________________
>>> keycloak-dev mailing list
>>> keycloak-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>