[keycloak-dev] federated pagination Re: federation commited need feedback

[Bill Burke]

On 7/28/2014 5:17 AM, Marek Posolda wrote:
> On 25.7.2014 21:28, Bill Burke wrote:
>>
>>
>> On 7/25/2014 2:19 PM, Marek Posolda wrote:
>>> I am already on picketlink master and trying to address the issues we
>>> have. Right now mapping of dynamic attributes aka
>>> https://issues.jboss.org/browse/PLINK-533 . I don't know when exactly is
>>> picketlink release planned, but maybe we can ask Pedro to do at least
>>> timestamped release somewhen next week?
>>>
>>> I wonder that random searches with FederationProvider might work for
>>> implementations, which supports offset/limit (like external database
>>> provided by customer) with your idea of omit local users with federation
>>> link? There are more tricky things though, like if user "john" available
>>> in in both local and LDAP store, and then directly deleted in LDAP,
>>> won't be returned if queries even if he is still in Keycloak database.
>>> Not sure if it's good or bad as he may not be able to login anyway...
>>>
>>
>> Was going to change it in that if a returned UserModel is linked, to
>> check that it is still valid for the federation and delete it if it is
>> not.
> +1
>>
>>> However for LDAP it's tricky... I've tested with ActiveDirectory that I
>>> can query with same cookie to random LDAP pages (next page or any page
>>> which I already previously visited), but looks that just next page is
>>> always guaranteed to work but previous pages are not guaranteed
>>> according to http://www.ietf.org/rfc/rfc2696.txt sentence:
>>> "Returning cookies from previous searchResultDone responses besides the
>>> last one is undefined, as the server implementation may restrict cookies
>>> from being reused."
>>>
>>
>> for now the federation interface for queries will be:
>>
>> UserModel getUserByUsername(RealmModel realm, String username)
>> UserModel getUserByEmail(RealmModel realm, String email)
>> List<UserModel> getUserByLastName((RealmModel realm, String last)
>> List<UserModel> getUserByFullName((RealmModel realm, String first,
>> String last);
>>
>> No pagination.  No paginated getAllUsers() and thus, no sync API. If
>> you can figure out something reliable in the next week *AND* get us a
>> new Picketlink release, we can use it in 1.0.Final. Otherwise, it is
>> just going to have to wait I guess.
> +1 to skip pagination from FederationProvider.
>
> But I think that at least one way sync from LDAP to Keycloak database
> should be possible. We already have support for ScheduledTask, which
> allow to do periodic syncs. IMO the tricky parts are:
>
> * pagination --- It might be needed as for example with 1 million users
> in LDAP, you will need multiple transaction to sync them all into
> Keycloak database. Some LDAP servers supports it and some not (generally
> said some Sync providers may support but some may not. It must be
> possible to configure if particular Sync Provider supports it).
>
> Regarding pagination, note that it's quite an issue for
> federationProvider, but not for sync as sync process have full control
> when to begin/commit transactions and also have access to all needed
> info (like cookies from LDAP)
>
> * Changelogs --- More tricky, but at least Active Directory (which seems
> to be major LDAP vendor) has support for it. The tricky part (also in
> AD) seems to be tracking of removed users (those which were directly
> removed in LDAP). So from time to time, it may be always good to do full
> sync
>
> * Configuration in admin console - It should be possible to choose Sync
> provider and then to configure:
> - if pagination will be used or not
> - if changelogs (partial sync) will be used or not
> - if to use periodic sync
> - how often to do full sync, how often to do partial sync (if supported)
> etc...
> - Possibility to trigger full or partial sync directly from admin
> console etc.
>

Sync's should be configured directly with the federation provider.  Or, 
when defining a sync, you can link it to a federation provider.  I don't 
want to have to specify configuration twice.

All of those configuration options are all LDAP specific and should be 
configured with the LDAP provider.

-- 
Bill Burke
JBoss, a division of Red Hat
http://bill.burkecentral.com