From: "Bill Burke" <bburke(a)redhat.com&gt; To: keycloak-dev(a)lists.jboss.org Sent: Thursday, 31 July, 2014 2:32:07 PM Subject: Re: [keycloak-dev] Sync + federation On 7/31/2014 7:33 AM, Marek Posolda wrote: > Hi, > > at this moment I have Sync with LDAP working (one way sync for importing > LDAP users into Keycloak). I tested with ActiveDirectory and OpenDS. On > both I have paginated and also full/partial periodic sync working. Also > UserFederationProvider seems to be working with Active Directory now! > What is the different between a partial and full sync? Wouldn't there just be one background periodic sync job? > Pedro mentioned that he will do timestamped picketlink release once we > need it. So far all the LDAP related issues should be fixed, so I've > asked him for doing one. Let me know if you are aware of more issues. > > The question is how to properly incorporate sync with current federation > stuff? It looks that both should work together (for example if customer > is going to write his FederationProvider to retrieve data from his DB, > he will also likely want to sync users from same DB). Also sync process > would likely need access to the federation configuration in > UserFederationProviderModel, but some configuration of Sync will be > common for all FederationProviders (For example pagination, periodic for > full/partial sync etc). So I wonder if we can add properties like this > directly into UserFederationProviderModel: > > boolean enablePagination; > int pageSize; Why would these ever need to be configured. Either the provider supports pagination or it doesn't. > long fullSyncPeriod; // -1 if periodic fullSync should be disabled > long partialSyncPeriod; // -1 if perodic partialSync should be disabled > Another option is to let the UserFederationProviderFactory handle synchronization and be configured through keycloak-server.json. Then there is no UI to do and no changes to the SPIs. Keycloak would have a generic Job scheduler (does it already?) and in the UserFederationProviderFactory.init() method it would just schedule the appropriate jobs.

> And for Admin console UI, we can have some common template, which can be > added into page of particular Federation Provider. For example on > federated-ldap.html or federated-generic.html there can be checkbox on > the bottom of the page like "enable synchronization of users" and when > people check it, it will display other settings (pagination, period for > full/partial sync, button for trigger sync directly from admin console > etc). > > Also not sure how to properly incorporate it into UserFederationProvider > API... Actually UserFederationProvider is supposed to be per-session > component whenever Sync process may actually use more > session/transaction lifecycles. So adding methods for sync directly into > UserFederationProvider may not work though... I wonder if we can have > method on UserFederationProviderFactory: > > UserSyncProvider getInstance(KeycloakSessionFactory sessionFactory, > UserFederationProviderModel model); > > And UserSyncProvider being something like this: > > public interface UserSyncProvider { > void syncAllUsers(KeycloakSessionFactory sessionFactory, > UserFederationProviderFactory fedFactory, String realmId, > UserFederationProviderModel fedModel) > void syncChangedUsers(KeycloakSessionFactory sessionFactory, > UserFederationProviderFactory fedFactory, String realmId, > UserFederationProviderModel fedModel, Date lastSync); > } What is the difference between syncAllUsers() and syncChangedUsers()? Is syncAllUsers() an import/sync from LDAP to Keycloak of all users in LDAP store? Is synchChangedUsers() only a synchronization from LDAP to Keycloak of only users that are currently imported into Keycloak? Depending on the answers to above questions, maybe UserFederationProviderFactory would have the appropriate sync methods instead? Then there would be one less interface that needs to be implemented. UserFederationProviderFactory { void sync(KeycloakSessionFactory sessionFactory, String realmId, UserFederationProviderModel model); } Depending on the above answer to questions, the UI should have a manual sync button(s) so that the admin can trigger the sync if desired. Period synchronization should be OFF be default. -- Bill Burke JBoss, a division of Red Hat http://bill.burkecentral.com _______________________________________________ keycloak-dev mailing list keycloak-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/keycloak-dev

>> boolean enablePagination; >> int pageSize; > Why would these ever need to be configured. Either the provider > supports pagination or it doesn't. It's tricky to rely just on implementation though. For example LDAP implementation generally supports pagination, but some LDAP servers support it, some are not. But you will use same implementation (aka LDAPFederationProvider) for both. So I believe that it should be a way to disable pagination if you are for example on ApacheDS. Also configuring pageSize might be useful too according to environment? And for unit test, you usually want to test with small pageSize just to verify that it works;-) Right now, I already have configurable pageSize and each "page" from LDAP is synced into Keycloak DB in separate transaction.

> > > >> long fullSyncPeriod; // -1 if periodic fullSync should be disabled >> long partialSyncPeriod; // -1 if perodic partialSync should be >> disabled >> > Another option is to let the UserFederationProviderFactory handle > synchronization and be configured through keycloak-server.json. Then > there is no UI to do and no changes to the SPIs. > > Keycloak would have a generic Job scheduler (does it already?) and in > the UserFederationProviderFactory.init() method it would just schedule > the appropriate jobs. Do we want to support the usecase, when you have 2 realms and you want sync from LDAP on "realm1" each 1 hour and on "realm2" each 5 hours? Also you may want to use different pageSize for each LDAP server (or you may have ActiveDirectory on "realm1" which supports pagination and "ApacheDS" on "realm2" which doesn't etc). In this case generic configuration in keycloak-server.json won't work though?

> > >> And for Admin console UI, we can have some common template, which can be >> added into page of particular Federation Provider. For example on >> federated-ldap.html or federated-generic.html there can be checkbox on >> the bottom of the page like "enable synchronization of users" and when >> people check it, it will display other settings (pagination, period for >> full/partial sync, button for trigger sync directly from admin >> console etc). >> >> Also not sure how to properly incorporate it into UserFederationProvider >> API... Actually UserFederationProvider is supposed to be per-session >> component whenever Sync process may actually use more >> session/transaction lifecycles. So adding methods for sync directly into >> UserFederationProvider may not work though... I wonder if we can have >> method on UserFederationProviderFactory: >> >> UserSyncProvider getInstance(KeycloakSessionFactory sessionFactory, >> UserFederationProviderModel model); >> >> And UserSyncProvider being something like this: >> >> public interface UserSyncProvider { >> void syncAllUsers(KeycloakSessionFactory sessionFactory, >> UserFederationProviderFactory fedFactory, String realmId, >> UserFederationProviderModel fedModel) >> void syncChangedUsers(KeycloakSessionFactory sessionFactory, >> UserFederationProviderFactory fedFactory, String realmId, >> UserFederationProviderModel fedModel, Date lastSync); >> } > > What is the difference between syncAllUsers() and syncChangedUsers()? > Is syncAllUsers() an import/sync from LDAP to Keycloak of all users in > LDAP store? yes. And likely also checking all "local" users with federation link, that link is still valid (ie. user wasn't deleted in LDAP, basically what UserFederationProvider.isValid is doing) and delete if not. > Is synchChangedUsers() only a synchronization from LDAP to > Keycloak of only users that are currently imported into Keycloak? It will leverage changelogs and sync just those "remote" users, which were added/updated (or removed if supported, but it's tricky for LDAP as I mentioned) remotely from the last sync.

> > > > Depending on the answers to above questions, maybe > UserFederationProviderFactory would have the appropriate sync methods > instead? Then there would be one less interface that needs to be > implemented. > > UserFederationProviderFactory { > > void sync(KeycloakSessionFactory sessionFactory, String realmId, > UserFederationProviderModel model); > > } I am fine with adding the methods directly on the factory. Was also thinking about it, but wasn't sure if having methods on "factory" class is not a workaround though;-)

From: "Marek Posolda" <mposolda(a)redhat.com&gt; To: "Bill Burke" <bburke(a)redhat.com&gt;, keycloak-dev(a)lists.jboss.org Sent: Thursday, 31 July, 2014 9:38:46 PM Subject: Re: [keycloak-dev] Sync + federation On 31.7.2014 22:01, Bill Burke wrote: > > > On 7/31/2014 3:00 PM, Marek Posolda wrote: >>>> boolean enablePagination; >>>> int pageSize; >>> Why would these ever need to be configured. Either the provider >>> supports pagination or it doesn't. >> It's tricky to rely just on implementation though. For example LDAP >> implementation generally supports pagination, but some LDAP servers >> support it, some are not. But you will use same implementation (aka >> LDAPFederationProvider) for both. So I believe that it should be a way >> to disable pagination if you are for example on ApacheDS. Also >> configuring pageSize might be useful too according to environment? And >> for unit test, you usually want to test with small pageSize just to >> verify that it works;-)

>> >> Right now, I already have configurable pageSize and each "page" from >> LDAP is synced into Keycloak DB in separate transaction. > > I just don't want the general SPI polluted with implementation details > of one adapter. Ok, it can be part of the LDAP FederationProvider configuration then. > >>> >>> >>> >>>> long fullSyncPeriod; // -1 if periodic fullSync should be >>>> disabled >>>> long partialSyncPeriod; // -1 if perodic partialSync should be >>>> disabled >>>> >>> Another option is to let the UserFederationProviderFactory handle >>> synchronization and be configured through keycloak-server.json. Then >>> there is no UI to do and no changes to the SPIs. >>> >>> Keycloak would have a generic Job scheduler (does it already?) and in >>> the UserFederationProviderFactory.init() method it would just schedule >>> the appropriate jobs. >> Do we want to support the usecase, when you have 2 realms and you want >> sync from LDAP on "realm1" each 1 hour and on "realm2" each 5 hours? >> Also you may want to use different pageSize for each LDAP server (or you >> may have ActiveDirectory on "realm1" which supports pagination and >> "ApacheDS" on "realm2" which doesn't etc). In this case generic >> configuration in keycloak-server.json won't work though? >> > > It was just an idea... > >>> >>> >>>> And for Admin console UI, we can have some common template, which >>>> can be >>>> added into page of particular Federation Provider. For example on >>>> federated-ldap.html or federated-generic.html there can be checkbox on >>>> the bottom of the page like "enable synchronization of users" and when >>>> people check it, it will display other settings (pagination, period >>>> for >>>> full/partial sync, button for trigger sync directly from admin >>>> console etc). >>>> >>>> Also not sure how to properly incorporate it into >>>> UserFederationProvider >>>> API... Actually UserFederationProvider is supposed to be per-session >>>> component whenever Sync process may actually use more >>>> session/transaction lifecycles. So adding methods for sync directly >>>> into >>>> UserFederationProvider may not work though... I wonder if we can have >>>> method on UserFederationProviderFactory: >>>> >>>> UserSyncProvider getInstance(KeycloakSessionFactory sessionFactory, >>>> UserFederationProviderModel model); >>>> >>>> And UserSyncProvider being something like this: >>>> >>>> public interface UserSyncProvider { >>>> void syncAllUsers(KeycloakSessionFactory sessionFactory, >>>> UserFederationProviderFactory fedFactory, String realmId, >>>> UserFederationProviderModel fedModel) >>>> void syncChangedUsers(KeycloakSessionFactory sessionFactory, >>>> UserFederationProviderFactory fedFactory, String realmId, >>>> UserFederationProviderModel fedModel, Date lastSync); >>>> } >>> >>> What is the difference between syncAllUsers() and syncChangedUsers()? >>> Is syncAllUsers() an import/sync from LDAP to Keycloak of all users in >>> LDAP store? >> yes. And likely also checking all "local" users with federation link, >> that link is still valid (ie. user wasn't deleted in LDAP, basically >> what UserFederationProvider.isValid is doing) and delete if not. >> >>> Is synchChangedUsers() only a synchronization from LDAP to >>> Keycloak of only users that are currently imported into Keycloak? >> It will leverage changelogs and sync just those "remote" users, which >> were added/updated (or removed if supported, but it's tricky for LDAP as >> I mentioned) remotely from the last sync. > > Ya, those separate methods soound good. > >> >>> >>> >>> >>> Depending on the answers to above questions, maybe >>> UserFederationProviderFactory would have the appropriate sync methods >>> instead? Then there would be one less interface that needs to be >>> implemented. >>> >>> UserFederationProviderFactory { >>> >>> void sync(KeycloakSessionFactory sessionFactory, String realmId, >>> UserFederationProviderModel model); >>> >>> } >> I am fine with adding the methods directly on the factory. Was also >> thinking about it, but wasn't sure if having methods on "factory" class >> is not a workaround though;-) >> > > I say put them on factory. > > Great, will do. Marek _______________________________________________ keycloak-dev mailing list keycloak-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/keycloak-dev