[keycloak-dev] sync/federation requirements/ideas

[Marek Posolda]

On 11.7.2014 16:26, Bill Burke wrote:
>
>
> On 7/10/2014 3:01 PM, Marek Posolda wrote:
>>> The problem with this approach is that new users don't get imported
>>> into Keycloak until the sync gets invoked.  Syncing more than once per
>>> day, or even once per week may not be feasible.  If the external
>>> storage does not have a changelog, syncing would involve iterating
>>> through each and every user in external storage and syncing it with
>>> the keycloak database.  A full sync could take hours.
>> Yes, but how to do it differently?
>> (a) For methods like: UserProvider.getUsers(), UserProvider.searchFor***
>> we can either:
>> 1) retrieve users just from "local" store
>> 2) Federate users and merge them from both "local" and "external" store.
>>
>> Federation approach (2) has quite bad performance issues (especially
>> with pagination+sorting), so I would prefer (1). But this really
>> requires full sync from external store into KC IMO.
>>
>>
>> (b) Then we have methods for retrieve single user:
>> getUserBy*** --- here we can try to retrieve user from 'local' store and
>> fallback to 'external'. In that case, user will be synced on demand to
>> KC database (similar approach like AuthenticationProvider), but then we
>> have the issue that just those users, who were retrieved from
>> externalStore will be available in KC.
>>
>> I think that we can either:
>> (1) Federate users from both stores
>> (2) Temporarily allow that just some users from LDAP are available in KC
>> database (those who were authenticated or for which getUserBy*** was
>> called)
>> (3) Do full sync
>>
>> I don't know if you see some other solution, but I would say that if we
>> want to avoid (1) and (2) we really need to do full sync. So I would
>> imagine that if someone configure externalStore, he will also need to
>> trigger full sync to KC to have all users added to Keycloak.
>
>
> Really depends on the UserProvider implementation IMO.  The 
> UserProvider doesn't have to import the users on search/getAll 
> operations. The UserModels returned could be proxies.  Probably harder 
> than I'm stating, but its quite possible its not.
>
>
>> Personally I would prefer to always do sync from KC immediatelly as
>> event queue doesn't always work.
>
> +1.  We'll probably eventually have to have an event queue, but I'd 
> like to avoid it.
>
>>
>> But fact is that if externalProvider doesn't support changelog, then how
>> to do sync without something like "getAllUsers" (or "syncAllUsers")? I
>> know that this is quite overhead, but I really don't see how to do
>> "partial" sync from 3rd party provider if it doesn't support 
>> changelogs...
>>
>
>
> How do we do external store->Keycloak sync?  *WE* don't.  The user 
> does! :)  Again  Have a very simple ChronJob that can be scheduled.  I 
> don't think there's any detailed SPI we can write that will solve all 
> user needs.
>
>> Marek
>>>
>>>
>>> IMO it would be something like this:
>>>
>>> public interface ChronJob {
>>>
>>>    void invoke(KeycloakSessionFactory factory);
>>> }
>>>
>
> The ChronJob has full access to local keycloak storage and can begin 
> and end transactions as needed.  We can probably provide one out of 
> the box for LDAP based on simple mapping metadata that can be 
> configured, but anything more complex has to be written by the user.
>
>
ok, once I finish export/import I can take a look how it looks with LDAP 
and changelogs, but unfortunately I don't think that it's standardized 
way to support this for all LDAP servers... Will also take a look at 
pagination, which is surely supported by most of LDAP servers AFAIK. The 
point for pagination is, that for example sync large LDAP with 100K 
users is probably not going to happen in single Keycloak transaction, 
but ChronJob would likely need to do it paginated in more transactions.

Marek