Instead of locking could you identify the coordinator and only sync from federation from the corresponding node ? Regards. Pedro Igor ----- Original Message ----- From: "Marek Posolda" <mposolda(a)redhat.com&gt; To: keycloak-dev(a)lists.jboss.org Sent: Wednesday, February 17, 2016 10:50:08 AM Subject: [keycloak-dev] Concurrent sync in cluster We had a bug https://issues.jboss.org/browse/KEYCLOAK-2412 that there are errors when sync of users from federationProvider is triggered concurrently in more cluster nodes. This affects periodic sync as well. To avoid concurrent executions of same task, I've added ClusterProvider. This is based on infinispan and it provides some locking functionality to ensures that sync from federation can be executed just by one cluster node at a time. Even on single node (non-cluster setup), now you can't trigger sync multiple times concurrently. So for example if there is periodic sync in progress and you click in admin console on "Sync users", the sync won't happen. The same mechanism is now also used for scheduled tasks (Removing expired user sessions and expired events). Nobody reported any bug yet, however when removing of expired events/sessions is triggered concurrently by more cluster nodes, it can be issue too. So this is now avoided. Maybe we can improve even more and ensure that just cluster coordinator will run scheduled tasks and other nodes will just ignore them? ClusterProvider also adds possibility to register ClusterListener with any task, that should be executed once notification from any cluster node comes. This allows that when some federation provider is created/updated/removed, then all nodes are aware of the change and will immediately change (or remove) scheduled timer. PR is here https://github.com/keycloak/keycloak/pull/2234 Marek _______________________________________________ keycloak-dev mailing list keycloak-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/keycloak-dev

From: "Marek Posolda" <mposolda(a)redhat.com&gt; To: "Pedro Igor Silva" <psilva(a)redhat.com&gt; Cc: keycloak-dev(a)lists.jboss.org Sent: Wednesday, February 17, 2016 12:50:42 PM Subject: Re: [keycloak-dev] Concurrent sync in cluster On 17/02/16 15:06, Pedro Igor Silva wrote: > I think it makes more sense to not spread administrative operations on > different nodes, but just on the coordinator. That would make the design > more predictable and make life easier when something wrong happens, given > that you know that only a specific node is able to perform the operation. > > Not sure how manual works, but in theory you can have a specific cache or > just use a known-entry to propagate coordinator related events. So when > you trigger a sync you don't really start the work but indicates to the > coordinator that a sync was triggered. You still need the lock though, but > that will be only a coordinator specific thing. Yeah, some kind of "locking" is always needed as issue can be seen even on single node. When the locking mechanism is cluster-aware then even better, regarding implementation it's not a big difference. Regarding every background task running on coordinator, there is the disadvantage of this IMO, that coordinator is under bigger workload.

For the future versions, we discussed some improvements planned like: - Possibility to see the progress in admin console (how many users were synced already, possibility to cancel task etc) - Possibility to run the sync in "distributable" manner. So that the sync can be always started on coordinator as you suggested, but coordinator will share the workload with other cluster nodes and "manage" the work (ie. node1 is supposed to do page1+page2 and node2 page3+page4 etc). We already use something like this based on infinispan Distributable executor service [1], which is very cool stuff IMO. Don't you think this is better regarding workload (and time of the task) then always execute everything on coordinator?

Unfortunately we are in the "bugfixing" phase and just addressing bugs, seems that all the other features will need to wait... [1] http://infinispan.org/docs/8.2.x/user_guide/user_guide.html#DistributedEx... Marek > > Regards. > Pedro Igor > > ----- Original Message ----- > From: "Marek Posolda" <mposolda(a)redhat.com&gt; > To: "Pedro Igor Silva" <psilva(a)redhat.com&gt; > Cc: keycloak-dev(a)lists.jboss.org > Sent: Wednesday, February 17, 2016 11:48:43 AM > Subject: Re: [keycloak-dev] Concurrent sync in cluster > > Was thinking about it. The thing is that we support both periodic and > manual sync. And the manual sync can be triggered on any cluster node. > You can even reproduce issue in non-cluster environment with single host > if you trigger concurrently 2 sync at the same time (or if periodic is > in progress and you trigger manual etc). > > The possibility to trigger on coordinator should work for scheduled > periodic cleanup tasks though. We don't support manual triggering for > them. Wonder if I should change this to trigger it always just on > coordinator. > > Btv. I am not using any real long-live lock, just the kind of > "pseudo-lock" (based on the presence of some particular item in the > cache, which is removed once the task is finished). > > > Marek > > > On 17/02/16 14:14, Pedro Igor Silva wrote: >> Instead of locking could you identify the coordinator and only sync from >> federation from the corresponding node ? >> >> Regards. >> Pedro Igor >> >> ----- Original Message ----- >> From: "Marek Posolda" <mposolda(a)redhat.com&gt; >> To: keycloak-dev(a)lists.jboss.org >> Sent: Wednesday, February 17, 2016 10:50:08 AM >> Subject: [keycloak-dev] Concurrent sync in cluster >> >> We had a bug https://issues.jboss.org/browse/KEYCLOAK-2412 that there >> are errors when sync of users from federationProvider is triggered >> concurrently in more cluster nodes. This affects periodic sync as well. >> >> To avoid concurrent executions of same task, I've added ClusterProvider. >> This is based on infinispan and it provides some locking functionality >> to ensures that sync from federation can be executed just by one cluster >> node at a time. Even on single node (non-cluster setup), now you can't >> trigger sync multiple times concurrently. So for example if there is >> periodic sync in progress and you click in admin console on "Sync >> users", the sync won't happen. >> >> The same mechanism is now also used for scheduled tasks (Removing >> expired user sessions and expired events). Nobody reported any bug yet, >> however when removing of expired events/sessions is triggered >> concurrently by more cluster nodes, it can be issue too. So this is now >> avoided. Maybe we can improve even more and ensure that just cluster >> coordinator will run scheduled tasks and other nodes will just ignore >> them? >> >> ClusterProvider also adds possibility to register ClusterListener with >> any task, that should be executed once notification from any cluster >> node comes. This allows that when some federation provider is >> created/updated/removed, then all nodes are aware of the change and will >> immediately change (or remove) scheduled timer. >> >> PR is here https://github.com/keycloak/keycloak/pull/2234 >> >> Marek >> _______________________________________________ >> keycloak-dev mailing list >> keycloak-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/keycloak-dev

Btw, I don't know sync works in details, but I assume that you also need to load data from the source repo on each node. Or just send a page to each node from the coordinator. If you load data from source repo on each node, you have also to consider how much concurrent queries the source repo can handle. If you send a page to each node, the coordinator will spend some time preparing the pages for each node until it is able to send them. Depending on how big is a LDAP server, for instance, that could take some time and you are still "overloading" the coordinator. ----- Original Message ----- From: "Pedro Igor Silva" <psilva(a)redhat.com&gt; To: "Marek Posolda" <mposolda(a)redhat.com&gt; Cc: keycloak-dev(a)lists.jboss.org Sent: Wednesday, February 17, 2016 1:13:14 PM Subject: Re: [keycloak-dev] Concurrent sync in cluster ----- Original Message ----- > From: "Marek Posolda" <mposolda(a)redhat.com&gt; > To: "Pedro Igor Silva" <psilva(a)redhat.com&gt; > Cc: keycloak-dev(a)lists.jboss.org > Sent: Wednesday, February 17, 2016 12:50:42 PM > Subject: Re: [keycloak-dev] Concurrent sync in cluster > > On 17/02/16 15:06, Pedro Igor Silva wrote: >> I think it makes more sense to not spread administrative operations on >> different nodes, but just on the coordinator. That would make the design >> more predictable and make life easier when something wrong happens, given >> that you know that only a specific node is able to perform the operation. >> >> Not sure how manual works, but in theory you can have a specific cache or >> just use a known-entry to propagate coordinator related events. So when >> you trigger a sync you don't really start the work but indicates to the >> coordinator that a sync was triggered. You still need the lock though, but >> that will be only a coordinator specific thing. > Yeah, some kind of "locking" is always needed as issue can be seen even > on single node. When the locking mechanism is cluster-aware then even > better, regarding implementation it's not a big difference. > > Regarding every background task running on coordinator, there is the > disadvantage of this IMO, that coordinator is under bigger workload. True, but I would avoid distribute processing as much as possible. However, I'm sure you want that to address some existing performance issues though. > For the future versions, we discussed some improvements planned like: > - Possibility to see the progress in admin console (how many users were > synced already, possibility to cancel task etc) > - Possibility to run the sync in "distributable" manner. So that the > sync can be always started on coordinator as you suggested, but > coordinator will share the workload with other cluster nodes and > "manage" the work (ie. node1 is supposed to do page1+page2 and node2 > page3+page4 etc). We already use something like this based on infinispan > Distributable executor service [1], which is very cool stuff IMO. Don't > you think this is better regarding workload (and time of the task) then > always execute everything on coordinator? Yeah, it is a very cool feature in ISPN, but how better is, depends. At the end you are using multiple producers (each node with their respective pages) writing to the same repository. Before answering your question, I would evaluate how much the end repository can handle concurrent transactions. Depending on how a database is configured, a single writer can perform better. > Unfortunately we are in the "bugfixing" phase and just addressing bugs, > seems that all the other features will need to wait... > > [1] > http://infinispan.org/docs/8.2.x/user_guide/user_guide.html#DistributedEx... > > Marek >> Regards. >> Pedro Igor >> >> ----- Original Message ----- >> From: "Marek Posolda" <mposolda(a)redhat.com&gt; >> To: "Pedro Igor Silva" <psilva(a)redhat.com&gt; >> Cc: keycloak-dev(a)lists.jboss.org >> Sent: Wednesday, February 17, 2016 11:48:43 AM >> Subject: Re: [keycloak-dev] Concurrent sync in cluster >> >> Was thinking about it. The thing is that we support both periodic and >> manual sync. And the manual sync can be triggered on any cluster node. >> You can even reproduce issue in non-cluster environment with single host >> if you trigger concurrently 2 sync at the same time (or if periodic is >> in progress and you trigger manual etc). >> >> The possibility to trigger on coordinator should work for scheduled >> periodic cleanup tasks though. We don't support manual triggering for >> them. Wonder if I should change this to trigger it always just on >> coordinator. >> >> Btv. I am not using any real long-live lock, just the kind of >> "pseudo-lock" (based on the presence of some particular item in the >> cache, which is removed once the task is finished). >> >> >> Marek >> >> >> On 17/02/16 14:14, Pedro Igor Silva wrote: >>> Instead of locking could you identify the coordinator and only sync from >>> federation from the corresponding node ? >>> >>> Regards. >>> Pedro Igor >>> >>> ----- Original Message ----- >>> From: "Marek Posolda" <mposolda(a)redhat.com&gt; >>> To: keycloak-dev(a)lists.jboss.org >>> Sent: Wednesday, February 17, 2016 10:50:08 AM >>> Subject: [keycloak-dev] Concurrent sync in cluster >>> >>> We had a bug https://issues.jboss.org/browse/KEYCLOAK-2412 that there >>> are errors when sync of users from federationProvider is triggered >>> concurrently in more cluster nodes. This affects periodic sync as well. >>> >>> To avoid concurrent executions of same task, I've added ClusterProvider. >>> This is based on infinispan and it provides some locking functionality >>> to ensures that sync from federation can be executed just by one cluster >>> node at a time. Even on single node (non-cluster setup), now you can't >>> trigger sync multiple times concurrently. So for example if there is >>> periodic sync in progress and you click in admin console on "Sync >>> users", the sync won't happen. >>> >>> The same mechanism is now also used for scheduled tasks (Removing >>> expired user sessions and expired events). Nobody reported any bug yet, >>> however when removing of expired events/sessions is triggered >>> concurrently by more cluster nodes, it can be issue too. So this is now >>> avoided. Maybe we can improve even more and ensure that just cluster >>> coordinator will run scheduled tasks and other nodes will just ignore >>> them? >>> >>> ClusterProvider also adds possibility to register ClusterListener with >>> any task, that should be executed once notification from any cluster >>> node comes. This allows that when some federation provider is >>> created/updated/removed, then all nodes are aware of the change and will >>> immediately change (or remove) scheduled timer. >>> >>> PR is here https://github.com/keycloak/keycloak/pull/2234 >>> >>> Marek >>> _______________________________________________ >>> keycloak-dev mailing list >>> keycloak-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/keycloak-dev > _______________________________________________ keycloak-dev mailing list keycloak-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/keycloak-dev