[keycloak-dev] Concurrent sync in cluster

[Pedro Igor Silva]

I think it makes more sense to not spread administrative operations on different nodes, but just on the coordinator. That would make the design more predictable and make life easier when something wrong happens, given that you know that only a specific node is able to perform the operation.

Not sure how manual works, but in theory you can have a specific cache or just use a known-entry to propagate coordinator related events. So when you trigger a sync you don't really start the work but indicates to the coordinator that a sync was triggered. You still need the lock though, but that will be only a coordinator specific thing.

Regards.
Pedro Igor

----- Original Message -----
From: "Marek Posolda" <mposolda at redhat.com>
To: "Pedro Igor Silva" <psilva at redhat.com>
Cc: keycloak-dev at lists.jboss.org
Sent: Wednesday, February 17, 2016 11:48:43 AM
Subject: Re: [keycloak-dev] Concurrent sync in cluster

Was thinking about it. The thing is that we support both periodic and 
manual sync. And the manual sync can be triggered on any cluster node. 
You can even reproduce issue in non-cluster environment with single host 
if you trigger concurrently 2 sync at the same time (or if periodic is 
in progress and you trigger manual etc).

The possibility to trigger on coordinator should work for scheduled 
periodic cleanup tasks though. We don't support manual triggering for 
them. Wonder if I should change this to trigger it always just on 
coordinator.

Btv. I am not using any real long-live lock, just the kind of 
"pseudo-lock" (based on the presence of some particular item in the 
cache, which is removed once the task is finished).


Marek


On 17/02/16 14:14, Pedro Igor Silva wrote:
> Instead of locking could you identify the coordinator and only sync from federation from the corresponding node ?
>
> Regards.
> Pedro Igor
>
> ----- Original Message -----
> From: "Marek Posolda" <mposolda at redhat.com>
> To: keycloak-dev at lists.jboss.org
> Sent: Wednesday, February 17, 2016 10:50:08 AM
> Subject: [keycloak-dev] Concurrent sync in cluster
>
> We had a bug https://issues.jboss.org/browse/KEYCLOAK-2412 that there
> are errors when sync of users from federationProvider is triggered
> concurrently in more cluster nodes. This affects periodic sync as well.
>
> To avoid concurrent executions of same task, I've added ClusterProvider.
> This is based on infinispan and it provides some locking functionality
> to ensures that sync from federation can be executed just by one cluster
> node at a time. Even on single node (non-cluster setup), now you can't
> trigger sync multiple times concurrently. So for example if there is
> periodic sync in progress and you click in admin console on "Sync
> users", the sync won't happen.
>
> The same mechanism is now also used for scheduled tasks (Removing
> expired user sessions and expired events). Nobody reported any bug yet,
> however when removing of expired events/sessions is triggered
> concurrently by more cluster nodes, it can be issue too. So this is now
> avoided. Maybe we can improve even more and ensure that just cluster
> coordinator will run scheduled tasks and other nodes will just ignore them?
>
> ClusterProvider also adds possibility to register ClusterListener with
> any task, that should be executed once notification from any cluster
> node comes. This allows that when some federation provider is
> created/updated/removed, then all nodes are aware of the change and will
> immediately change (or remove) scheduled timer.
>
> PR is here https://github.com/keycloak/keycloak/pull/2234
>
> Marek
> _______________________________________________
> keycloak-dev mailing list
> keycloak-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-dev