[keycloak-dev] sync/federation requirements/ideas

Fri Jul 11 09:05:55 EDT 2014

This sounds like exactly what we need. There are quite a lot of kinks to work out though.

With that in mind I think we should limit the scope of this for 1.0.final to something along the lines of what AuthProvider does now. Question is for 1.0.final do we:

* Extract user related methods into UserProvider - my vote is yes
* Deprecate AuthProvider and provide something similar through UserProvider - I'm unsure about this
* Try to cover all basis in UserProvider such as sync, etc - my vote is no

With regards to SPIs in general there are some that are aimed at internal use only, others that are aimed at public use, and I guess we could flag certain SPIs as likely to change in the future (such as AuthProvider).

----- Original Message -----
> From: "Marek Posolda" <mposolda at redhat.com>
> To: "Bill Burke" <bburke at redhat.com>, keycloak-dev at lists.jboss.org
> Sent: Thursday, 10 July, 2014 8:01:33 PM
> Subject: Re: [keycloak-dev] sync/federation requirements/ideas
> 
> On 10.7.2014 15:47, Bill Burke wrote:
> >
> >
> > On 7/10/2014 7:41 AM, Marek Posolda wrote:
> >>
> >> That's why I would prefer "sync" among "federation" approach.
> >
> > Which is exactly what I'm proposing.  Except it allows for
> > syncs/imports into Keycloak on demand.  You already use
> > AuthenticationProvider in this manner!
> oops, I though that with AuthProvider you don't like the fact that just
> LDAP users authenticated with Keycloak are available in Keycloak DB;-)
> 
> 
> >
> > My impression of a sync model is that Keycloak works with user data in
> > local Keycloak storage.  User metadata is imported from external
> > storage (either on demand or at startp and periodically synced).
> > Keycloak can manage/modify/augment this imported user data locally.
> > The changelog is obtained from the Sync API and updates sent back to
> > the external storage.
> >
> >
> >> In
> >> shortcut with "sync" approach will be users (and eventually their role
> >> mappings) synced from external store into Keycloak and most of the
> >> UserProvider methods would deal just with "local" provider. Only
> >> exception are authentication related methods, and also "update" methods,
> >> which will need to update local user and also "external" users. But
> >> SyncProvider won't need to implement methods like getUsers(),
> >> searchFor**, getUserBy** etc as all users will be available locally.
> >>
> >
> > The problem with this approach is that new users don't get imported
> > into Keycloak until the sync gets invoked.  Syncing more than once per
> > day, or even once per week may not be feasible.  If the external
> > storage does not have a changelog, syncing would involve iterating
> > through each and every user in external storage and syncing it with
> > the keycloak database.  A full sync could take hours.
> Yes, but how to do it differently?
> (a) For methods like: UserProvider.getUsers(), UserProvider.searchFor***
> we can either:
> 1) retrieve users just from "local" store
> 2) Federate users and merge them from both "local" and "external" store.
> 
> Federation approach (2) has quite bad performance issues (especially
> with pagination+sorting), so I would prefer (1). But this really
> requires full sync from external store into KC IMO.
> 
> 
> (b) Then we have methods for retrieve single user:
> getUserBy*** --- here we can try to retrieve user from 'local' store and
> fallback to 'external'. In that case, user will be synced on demand to
> KC database (similar approach like AuthenticationProvider), but then we
> have the issue that just those users, who were retrieved from
> externalStore will be available in KC.
> 
> I think that we can either:
> (1) Federate users from both stores
> (2) Temporarily allow that just some users from LDAP are available in KC
> database (those who were authenticated or for which getUserBy*** was called)
> (3) Do full sync
> 
> I don't know if you see some other solution, but I would say that if we
> want to avoid (1) and (2) we really need to do full sync. So I would
> imagine that if someone configure externalStore, he will also need to
> trigger full sync to KC to have all users added to Keycloak.
> >
> >
> >>
> >>
> >>
> >> In details, I can imagine to have things working this way:
> >>
> >> * There will be Sync SPI configured per realm, which will allow to sync
> >> users from "external store" to Keycloak.
> >>
> >
> > +1
> >
> >> * It will be possible to configure when to sync users. For example it
> >> can be full sync from LDAP at server startup or triggered from admin
> >> console. Then some periodic sync (triggered for example once per day).
> >>
> >
> > +1
> >
> >> * I am not sure if we need full sync from Keycloak to "external
> >> provider" but probably not. Once user, role or user credential will be
> >> updated in Keycloak, it will be also immediatelly synced to external
> >> provider (if provider is not read-only).
> >>
> >
> > If you don't implement through the UserProvider/UserModel interface
> > then you would need an Event model for which the SyncProvider listens
> > to so that it can sync the user on demand.
> yep, I think that it works to implement this in UserProvider (and
> RealmProvider for roles). Basically you have something like
> "WrapperUserProvider" and once add/update/delete user is invoked, it
> will perform the operation on "local" UserProvider and then invoke
> particular method on SyncProvider to sync it back into external storage.
> >
> >> * It depends on SyncProvider implementation, what user data are synced.
> >> For example in case of LDAP it will be just email+first,lastName and
> >> eventually role mappings.
> >>
> >> * SyncProvider may also support roles + role mappings from external
> >> store. SyncProvider will be able to import roles into RealmProvider and
> >> then particular role mappings into UserProvider.
> >>
> >> * Users synced from "external provider" will have link to this provider
> >> similarly like it's now. In case that provider is read-write, then all
> >> newly created users in Keycloak will be synced into "external provider"
> >> immediatelly also with the credential (if syncProvider supports
> >> particular credential)
> >>
> >> I would propose the interface like this to handle both
> >> sync/authentication (not sure if "SyncProvider" is good name, maybe
> >> rather "ExternalUserProvider"). I can imagine something like this:
> >>
> >>
> >> public interface SyncProvider {
> >>
> >
> > I think your SyncProvider interface is very limiting and even
> > unusable.  For example, getAllUsers() is completely unfeasible if
> > there is a large user database.
> oops, sorry. It's maybe not clear from method name and javadoc that
> "getAllUsers" is intended to be used for sync *from* External storage
> *to* Keycloak. The idea is that "getAllUsers" (or
> "syncAllUsersFromExternalStore" or whatever it's named) will be called
> by admin when he wants full sync from external storage to KC.
> 
> IMO sync from Keycloak to external storage is quite easy. We can either
> do it immediatelly when add/update/delete user/role is invoked or we can
> use some event approach you proposed with periodic (or on demand) sync
> of just "dirty" entities. We already have AuditProvider, which could be
> eventually used.
> 
> Personally I would prefer to always do sync from KC immediatelly as
> event queue doesn't always work. For example when you have RW LDAP and
> you register new user in Keycloak, you would need to save this user and
> his password immediatelly into external store. Save it somewhere in
> Keycloak database and then clear once Chron Job is triggered doesn't
> work IMO as it would mean that we will need to save passwords in
> plain-text until they are synced into LDAP. Which doesn't make much
> sense IMO.
> 
> So for sync from Keycloak to externalStore, I would personally always do
> it immediatelly after operation is performed on
> UserProvider/RealmProvider. It's not so big overhead IMO.
> 
> But the main issue is sync from externalStore to Keycloak. We can easily
> add changelog to Keycloak, but we don't have control under external
> storage...
> 
> I don't know if LDAP servers support changelog, probably yes but I don't
> think that it's standardized, maybe some LDAP servers support
> "proprietary" way of doing changelog. I can check if some LDAP supports
> this and how it works with them...
> 
> But fact is that if externalProvider doesn't support changelog, then how
> to do sync without something like "getAllUsers" (or "syncAllUsers")? I
> know that this is quite overhead, but I really don't see how to do
> "partial" sync from 3rd party provider if it doesn't support changelogs...
> 
> Marek
> >
> >
> > IMO it would be something like this:
> >
> > public interface ChronJob {
> >
> >    void invoke(KeycloakSessionFactory factory);
> > }
> >
> > ChronJob would be scheduled to run at boot time and/or periodically by
> > the admin console.  Periodic and boot time syncs would be implemented
> > here.  The sync operation needs to have full control of when
> > transactions begin and end so updates/creates/deletes can be executed
> > in batches.
> >
> > On demand syncing would be done through an implementation of
> > UserProvider as I proposed earlier.  On-demand syncing is:
> >
> > * Importing a user on demand
> > * Updating external storage on demand.
> >
> > We would also have an event changelog that would be implemented as, or
> > act like a persistent JMS Topic.  This event topic shouldn't be used
> > to implement on-demand sync, but rather for when the external storage
> > is read only.:
> >
> > interface ChangeEvent {
> >    enum EventType {
> >        CREATE, UPDATE, DELETE
> >    }
> >
> >
> >
> >    long getTimestamp();
> >    EventType getEventType();
> >    String getItemId();
> >    String getItemType(); // UserModel, RoleModel
> > }
> >
> > EventListeners could perform the syncs on demand.  Or, a ChronJob
> > could be something like a persistence JMS Topic subscriber and replay
> > change events.
> >
> > We could decide that we don't support user updates for read-only
> > stores.  Then, in that case, IMO, we don't need a changelog event queue.
> >
> >
> >
> >
> 
> _______________________________________________
> keycloak-dev mailing list
> keycloak-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-dev
> 