Re: [keycloak-dev] federated pagination Re: federation commited need feedback

On 25.7.2014 21:28, Bill Burke wrote: > > > On 7/25/2014 2:19 PM, Marek Posolda wrote: >> I am already on picketlink master and trying to address the issues we >> have. Right now mapping of dynamic attributes aka >> https://issues.jboss.org/browse/PLINK-533 . I don't know when exactly is >> picketlink release planned, but maybe we can ask Pedro to do at least >> timestamped release somewhen next week? >> >> I wonder that random searches with FederationProvider might work for >> implementations, which supports offset/limit (like external database >> provided by customer) with your idea of omit local users with federation >> link? There are more tricky things though, like if user "john" available >> in in both local and LDAP store, and then directly deleted in LDAP, >> won't be returned if queries even if he is still in Keycloak database. >> Not sure if it's good or bad as he may not be able to login anyway... >> > > Was going to change it in that if a returned UserModel is linked, to > check that it is still valid for the federation and delete it if it is > not. +1 > >> However for LDAP it's tricky... I've tested with ActiveDirectory that I >> can query with same cookie to random LDAP pages (next page or any page >> which I already previously visited), but looks that just next page is >> always guaranteed to work but previous pages are not guaranteed >> according to http://www.ietf.org/rfc/rfc2696.txt sentence: >> "Returning cookies from previous searchResultDone responses besides the >> last one is undefined, as the server implementation may restrict cookies >> from being reused." >> > > for now the federation interface for queries will be: > > UserModel getUserByUsername(RealmModel realm, String username) > UserModel getUserByEmail(RealmModel realm, String email) > List<UserModel> getUserByLastName((RealmModel realm, String last) > List<UserModel> getUserByFullName((RealmModel realm, String first, > String last); > > No pagination. No paginated getAllUsers() and thus, no sync API. If > you can figure out something reliable in the next week *AND* get us a > new Picketlink release, we can use it in 1.0.Final. Otherwise, it is > just going to have to wait I guess. +1 to skip pagination from FederationProvider. But I think that at least one way sync from LDAP to Keycloak database should be possible. We already have support for ScheduledTask, which allow to do periodic syncs. IMO the tricky parts are: * pagination --- It might be needed as for example with 1 million users in LDAP, you will need multiple transaction to sync them all into Keycloak database. Some LDAP servers supports it and some not (generally said some Sync providers may support but some may not. It must be possible to configure if particular Sync Provider supports it). Regarding pagination, note that it's quite an issue for federationProvider, but not for sync as sync process have full control when to begin/commit transactions and also have access to all needed info (like cookies from LDAP) * Changelogs --- More tricky, but at least Active Directory (which seems to be major LDAP vendor) has support for it. The tricky part (also in AD) seems to be tracking of removed users (those which were directly removed in LDAP). So from time to time, it may be always good to do full sync * Configuration in admin console - It should be possible to choose Sync provider and then to configure: - if pagination will be used or not - if changelogs (partial sync) will be used or not - if to use periodic sync - how often to do full sync, how often to do partial sync (if supported) etc... - Possibility to trigger full or partial sync directly from admin console etc.