[keycloak-dev] Concurrent sync in cluster

Wed Feb 17 15:26:35 EST 2016

IMO load data from the source repo on each node is better option. That's 
what we are already using on one place (preloading offline userSessions 
from DB to infinispan cache at server startup). And I can see that 
distributed cache helps with startup when 2 nodes are started 
concurrently (startup time is not 2 times better, but it's better then 
single node startup). I tried with MySQL and Mongo and some improvement 
for concurrent startup can be seen for both.

As you mentioned, sending page from coordinator would mean big overhead 
on coordinator and big messages send over the cluster, so even network 
overhead. Doesn't seem to me like very good option, but maybe I am wrong...

One bad thing is, that some federation storages may not support classic 
pagination, which allows easy divide work into pages and share the work 
(ie. page3 starts from offset 100 to offset 150). For example LDAP 
doesn't have classic pagination, but each page return "cookie", which 
you need to use for query next page. This could make it harder to 
properly share the work. But hopefully this is solvable somehow...

Marek

On 17/02/16 16:24, Pedro Igor Silva wrote:
> Btw, I don't know sync works in details, but I assume that you also need to load data from the source repo on each node. Or just send a page to each node from the coordinator.
>
> If you load data from source repo on each node, you have also to consider how much concurrent queries the source repo can handle. If you send a page to each node, the coordinator will spend some time preparing the pages for each node until it is able to send them. Depending on how big is a LDAP server, for instance, that could take some time and you are still "overloading" the coordinator.
>
> ----- Original Message -----
> From: "Pedro Igor Silva" <psilva at redhat.com>
> To: "Marek Posolda" <mposolda at redhat.com>
> Cc: keycloak-dev at lists.jboss.org
> Sent: Wednesday, February 17, 2016 1:13:14 PM
> Subject: Re: [keycloak-dev] Concurrent sync in cluster
>
> ----- Original Message -----
>> From: "Marek Posolda" <mposolda at redhat.com>
>> To: "Pedro Igor Silva" <psilva at redhat.com>
>> Cc: keycloak-dev at lists.jboss.org
>> Sent: Wednesday, February 17, 2016 12:50:42 PM
>> Subject: Re: [keycloak-dev] Concurrent sync in cluster
>>
>> On 17/02/16 15:06, Pedro Igor Silva wrote:
>>> I think it makes more sense to not spread administrative operations on
>>> different nodes, but just on the coordinator. That would make the design
>>> more predictable and make life easier when something wrong happens, given
>>> that you know that only a specific node is able to perform the operation.
>>>
>>> Not sure how manual works, but in theory you can have a specific cache or
>>> just use a known-entry to propagate coordinator related events. So when
>>> you trigger a sync you don't really start the work but indicates to the
>>> coordinator that a sync was triggered. You still need the lock though, but
>>> that will be only a coordinator specific thing.
>> Yeah, some kind of "locking" is always needed as issue can be seen even
>> on single node. When the locking mechanism is cluster-aware then even
>> better, regarding implementation it's not a big difference.
>>
>> Regarding every background task running on coordinator, there is the
>> disadvantage of this IMO, that coordinator is under bigger workload.
> True, but I would avoid distribute processing as much as possible. However, I'm sure you want that to address some existing performance issues though.
>
>> For the future versions, we discussed some improvements planned like:
>> - Possibility to see the progress in admin console (how many users were
>> synced already, possibility to cancel task etc)
>> - Possibility to run the sync in "distributable" manner. So that the
>> sync can be always started on coordinator as you suggested, but
>> coordinator will share the workload with other cluster nodes and
>> "manage" the work (ie. node1 is supposed to do page1+page2 and node2
>> page3+page4 etc). We already use something like this based on infinispan
>> Distributable executor service [1], which is very cool stuff IMO. Don't
>> you think this is better regarding workload (and time of the task) then
>> always execute everything on coordinator?
> Yeah, it is a very cool feature in ISPN, but how better is, depends. At the end you are using multiple producers (each node with their respective pages) writing to the same repository. Before answering your question, I would evaluate how much the end repository can handle concurrent transactions. Depending on how a database is configured, a single writer can perform better.
>
>> Unfortunately we are in the "bugfixing" phase and just addressing bugs,
>> seems that all the other features will need to wait...
>>
>> [1]
>> http://infinispan.org/docs/8.2.x/user_guide/user_guide.html#DistributedExecutor
>>
>> Marek
>>> Regards.
>>> Pedro Igor
>>>
>>> ----- Original Message -----
>>> From: "Marek Posolda" <mposolda at redhat.com>
>>> To: "Pedro Igor Silva" <psilva at redhat.com>
>>> Cc: keycloak-dev at lists.jboss.org
>>> Sent: Wednesday, February 17, 2016 11:48:43 AM
>>> Subject: Re: [keycloak-dev] Concurrent sync in cluster
>>>
>>> Was thinking about it. The thing is that we support both periodic and
>>> manual sync. And the manual sync can be triggered on any cluster node.
>>> You can even reproduce issue in non-cluster environment with single host
>>> if you trigger concurrently 2 sync at the same time (or if periodic is
>>> in progress and you trigger manual etc).
>>>
>>> The possibility to trigger on coordinator should work for scheduled
>>> periodic cleanup tasks though. We don't support manual triggering for
>>> them. Wonder if I should change this to trigger it always just on
>>> coordinator.
>>>
>>> Btv. I am not using any real long-live lock, just the kind of
>>> "pseudo-lock" (based on the presence of some particular item in the
>>> cache, which is removed once the task is finished).
>>>
>>>
>>> Marek
>>>
>>>
>>> On 17/02/16 14:14, Pedro Igor Silva wrote:
>>>> Instead of locking could you identify the coordinator and only sync from
>>>> federation from the corresponding node ?
>>>>
>>>> Regards.
>>>> Pedro Igor
>>>>
>>>> ----- Original Message -----
>>>> From: "Marek Posolda" <mposolda at redhat.com>
>>>> To: keycloak-dev at lists.jboss.org
>>>> Sent: Wednesday, February 17, 2016 10:50:08 AM
>>>> Subject: [keycloak-dev] Concurrent sync in cluster
>>>>
>>>> We had a bug https://issues.jboss.org/browse/KEYCLOAK-2412 that there
>>>> are errors when sync of users from federationProvider is triggered
>>>> concurrently in more cluster nodes. This affects periodic sync as well.
>>>>
>>>> To avoid concurrent executions of same task, I've added ClusterProvider.
>>>> This is based on infinispan and it provides some locking functionality
>>>> to ensures that sync from federation can be executed just by one cluster
>>>> node at a time. Even on single node (non-cluster setup), now you can't
>>>> trigger sync multiple times concurrently. So for example if there is
>>>> periodic sync in progress and you click in admin console on "Sync
>>>> users", the sync won't happen.
>>>>
>>>> The same mechanism is now also used for scheduled tasks (Removing
>>>> expired user sessions and expired events). Nobody reported any bug yet,
>>>> however when removing of expired events/sessions is triggered
>>>> concurrently by more cluster nodes, it can be issue too. So this is now
>>>> avoided. Maybe we can improve even more and ensure that just cluster
>>>> coordinator will run scheduled tasks and other nodes will just ignore
>>>> them?
>>>>
>>>> ClusterProvider also adds possibility to register ClusterListener with
>>>> any task, that should be executed once notification from any cluster
>>>> node comes. This allows that when some federation provider is
>>>> created/updated/removed, then all nodes are aware of the change and will
>>>> immediately change (or remove) scheduled timer.
>>>>
>>>> PR is here https://github.com/keycloak/keycloak/pull/2234
>>>>
>>>> Marek
>>>> _______________________________________________
>>>> keycloak-dev mailing list
>>>> keycloak-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>
> _______________________________________________
> keycloak-dev mailing list
> keycloak-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-dev