[keycloak-dev] federated pagination Re: federation commited need feedback

[Marek Posolda]

On 25.7.2014 21:28, Bill Burke wrote:
>
>
> On 7/25/2014 2:19 PM, Marek Posolda wrote:
>> I am already on picketlink master and trying to address the issues we
>> have. Right now mapping of dynamic attributes aka
>> https://issues.jboss.org/browse/PLINK-533 . I don't know when exactly is
>> picketlink release planned, but maybe we can ask Pedro to do at least
>> timestamped release somewhen next week?
>>
>> I wonder that random searches with FederationProvider might work for
>> implementations, which supports offset/limit (like external database
>> provided by customer) with your idea of omit local users with federation
>> link? There are more tricky things though, like if user "john" available
>> in in both local and LDAP store, and then directly deleted in LDAP,
>> won't be returned if queries even if he is still in Keycloak database.
>> Not sure if it's good or bad as he may not be able to login anyway...
>>
>
> Was going to change it in that if a returned UserModel is linked, to 
> check that it is still valid for the federation and delete it if it is 
> not.
+1
>
>> However for LDAP it's tricky... I've tested with ActiveDirectory that I
>> can query with same cookie to random LDAP pages (next page or any page
>> which I already previously visited), but looks that just next page is
>> always guaranteed to work but previous pages are not guaranteed
>> according to http://www.ietf.org/rfc/rfc2696.txt sentence:
>> "Returning cookies from previous searchResultDone responses besides the
>> last one is undefined, as the server implementation may restrict cookies
>> from being reused."
>>
>
> for now the federation interface for queries will be:
>
> UserModel getUserByUsername(RealmModel realm, String username)
> UserModel getUserByEmail(RealmModel realm, String email)
> List<UserModel> getUserByLastName((RealmModel realm, String last)
> List<UserModel> getUserByFullName((RealmModel realm, String first, 
> String last);
>
> No pagination.  No paginated getAllUsers() and thus, no sync API. If 
> you can figure out something reliable in the next week *AND* get us a 
> new Picketlink release, we can use it in 1.0.Final. Otherwise, it is 
> just going to have to wait I guess.
+1 to skip pagination from FederationProvider.

But I think that at least one way sync from LDAP to Keycloak database 
should be possible. We already have support for ScheduledTask, which 
allow to do periodic syncs. IMO the tricky parts are:

* pagination --- It might be needed as for example with 1 million users 
in LDAP, you will need multiple transaction to sync them all into 
Keycloak database. Some LDAP servers supports it and some not (generally 
said some Sync providers may support but some may not. It must be 
possible to configure if particular Sync Provider supports it).

Regarding pagination, note that it's quite an issue for 
federationProvider, but not for sync as sync process have full control 
when to begin/commit transactions and also have access to all needed 
info (like cookies from LDAP)

* Changelogs --- More tricky, but at least Active Directory (which seems 
to be major LDAP vendor) has support for it. The tricky part (also in 
AD) seems to be tracking of removed users (those which were directly 
removed in LDAP). So from time to time, it may be always good to do full 
sync

* Configuration in admin console - It should be possible to choose Sync 
provider and then to configure:
- if pagination will be used or not
- if changelogs (partial sync) will be used or not
- if to use periodic sync
- how often to do full sync, how often to do partial sync (if supported) 
etc...
- Possibility to trigger full or partial sync directly from admin 
console etc.

I hope to figure it out more during this week as you suggested..

Marek