[keycloak-dev] sync/federation requirements/ideas

[Marek Posolda]

On 9.7.2014 23:37, Bill Burke wrote:
> To expand on our discussion around AuthenticationProvider.   With the
> currrent implementation of AuthenticationProvider, Keycloak admin
> console cannot be used to manage external users until the user has
> logged in and a UserModel creation has happened.  We would need to
> duplicate the "import" logic that is within AuthenticationManager
> wherever UserProvider.getUserByXXXX() is invoked.  Not only that, but
> role mappings and other claim data may be contained in external storage.
>    The authentictionProvider interface is inadequate for these scenarios.
>
>
> My thinking is that for federation and sync would come in three flavors:
>
> * Keycloak storage.  What we currently implement.
> * Full external storage.  The app developer will use their own
> persistence model to store and manage users.  In this case, they need to
> fully implement the UserProvider interface.  Handle social links,
> required actions, role mappings etc.
> * A sync/import model.  In this case, keycloak is augmenting an existing
> user storage.  This would require partially or fully importing the user
> from the external store into Keycloak UserModel/Provider storage.   The
> import is required so that Keycloak can managed REQUIRED ACTIONS, social
> links, and add additional or manage existing credential types.  We also
> need the import to establish relationships between the user and
> UserSession metadata.  We may also have to handle role mappings within
> Keycloak.  A sync/import model example could be an LDAP store that just
> contains username, email, first/last, and password.  And the admin wants
> to add role mappings and totp.
>
> How would it work?
>
> I think all this should be done through a common UserProvider interface.
>    Initially be able to federate only 1 additional external store as I'm
> worried that there may be the possibility of duplicate user names if you
> allow more than one and you'd need a way at login to pick which one to
> log in from.
>
> UserProviderFactory would specify its feature set:
>
> Query Capabilities:
> REGISTRATION
> QUERY_BY_USER_ID
> QUERY_BY_USERNAME
> QUERY_BY_EMAIL
> QUERY_BY_ATTRIBUTES
>
> And its supported credential validation:
> PASSWORD_VALIDATION
> TOTP_VALIDATION
> CERT_VALIDATION
>
> And its supported credential storage:
> PASSWORD_STORAGE
> TOTP_STORAGE
> CERT_STORAGE
>
> There would a FederatedUserProvider to manage local and federated storage.
>
> *Locating a user:*
>
> 1.  FederatedUserProvider queries local storage first.
> 2. Check realm to see if federated storage configured
> 3. Load that provider
> 4. See if that provider supports query method, if not return null
> 5. Call query method on provider
>
> Provider performs the following on user location:
> 6. Do custom query to find user
> 7. Get a local storage session and create a user within local storage
> importing whatever can be imported.
> 8. Mark in local user storage which provider loaded it.
> 9. return the local user
>
> *Required Actions and Credential management*
> 1. Find provider used to load user
> 2. Check credential storage options
> 3. If provider doesn't support credential update/storage, then store in
> local storage
>
>
> *Authentication*
> 1. Find user in local storage
> 2. See if credential is stored in local storage, validate from there if so.
> 3. If not stored, check provider validation options.  Abort if not supported
> 4. Invoke on external provider to validate credentials supported by that
> provider
>
>
> Did I cover everything?
>
How about creating of users? Will be new users added just to local 
storage and not to the "external provider" storage?

Also not sure what's your plan for role mappings? Will be role mappings 
loaded from "external provider" just in case that provider supports role 
mappings? But what if roles from provider doesn't exist in Keycloak?

And finally getUsers(), searchForUser, searchForUserByAttributes in both 
non-paginated and paginated versions?

IMO Federation approach is quite tricky. We already did this in GateIn 
(picketlink 1.X) and had quite bad issues. Especially performance 
issues. For example getUsers() with pagination+sorting is almost 
impossible if you have some users just in "local" store, some users in 
"external" store, and some synced in both. For example if there are 
users "a", "b" in external storage, then users "b", "c" in local storage 
and you call getUsers() you would need to merge them from both storages 
to have correct list with "a", "b" and "c". If you want this with 
pagination+sorting then it's much more tricky... If you want proper 
results, you defacto need to load all users from both stores and sort 
them programmatically, which completely bypasses pagination.

That's why I would prefer "sync" among "federation" approach. In 
shortcut with "sync" approach will be users (and eventually their role 
mappings) synced from external store into Keycloak and most of the 
UserProvider methods would deal just with "local" provider. Only 
exception are authentication related methods, and also "update" methods, 
which will need to update local user and also "external" users. But 
SyncProvider won't need to implement methods like getUsers(), 
searchFor**, getUserBy** etc as all users will be available locally.




In details, I can imagine to have things working this way:

* There will be Sync SPI configured per realm, which will allow to sync 
users from "external store" to Keycloak.

* It will be possible to configure when to sync users. For example it 
can be full sync from LDAP at server startup or triggered from admin 
console. Then some periodic sync (triggered for example once per day).

* I am not sure if we need full sync from Keycloak to "external 
provider" but probably not. Once user, role or user credential will be 
updated in Keycloak, it will be also immediatelly synced to external 
provider (if provider is not read-only).

* It depends on SyncProvider implementation, what user data are synced. 
For example in case of LDAP it will be just email+first,lastName and 
eventually role mappings.

* SyncProvider may also support roles + role mappings from external 
store. SyncProvider will be able to import roles into RealmProvider and 
then particular role mappings into UserProvider.

* Users synced from "external provider" will have link to this provider 
similarly like it's now. In case that provider is read-write, then all 
newly created users in Keycloak will be synced into "external provider" 
immediatelly also with the credential (if syncProvider supports 
particular credential)

I would propose the interface like this to handle both 
sync/authentication (not sure if "SyncProvider" is good name, maybe 
rather "ExternalUserProvider"). I can imagine something like this:


public interface SyncProvider {

   /**
    * Invoked when new user is registered (or added through admin 
console) and just if SyncProvider supports RW_USER
    *
    * @return user with added "link" . Link is filled by external 
provider (for example ID from LDAP)
    */
   UserModel addUser(UserModel user);

   /**
     * Invoked when credential of user is changed and just if: 
SyncProvider is RW_PASSWORD (or RW_TOTP) and user has "link" to provider
     */
   void updateCredential(UserModel user, Credential credential);

  /**
    * Invoked when user has link to externalProvider and particular 
credential is supported.  (Otherwise credential is validated against 
"local" UserProvider)
    */
   boolean validateCredential(UserModel user, Credential credential);

   /**
     * Invoked just when full sync from "external store" to Keycloak is 
requested. Method could be empty if SyncProvider doesn't suppport Roles 
+ Role mappings
     * Not sure if role also needs to have "link" . Probably not.
     */
   List<RoleModel> getRoles();

   /**
    * Invoked just when full sync from "external store" to Keycloak is 
requested. Method could be empty if SyncProvider doesn't support Roles + 
Role mappings
    */
   void updateRole(RoleModel role);

   /**
    * Invoked just when full sync from "external store" to Keycloak is 
requested. It's invoked *after* getAllRoles(), so Keycloak knows roles 
mapped to users
    */
   List<UserModel> getAllUsers();

   /**
    * List features supported by particular provider
    */
   List<Feature> getMetadata();
}

For metadata I can see the list with RW_ROLE, RO_ROLE (If provider 
supports storing of roles and role mappings), RW_USER, RO_USER (If 
provider supports storing of users), RW_PASSWORD, RO_PASSWORD (If 
provider supports passwords), RO_TOTP, RW_TOTP (If provider supports 
storing of totp)

In admin console, it will be possible to configure just subset of 
available features. For example LDAPSyncProvider supports both RO and RW 
for users,roles and passwords, but for RO deployments, people can check 
just RO_ROLE, RO_USER, RO_PASSWORD, which means that LDAP is read-only 
and users will be synced just from LDAP to Keycloak and passwords will 
be verified against LDAP, but nothing will be written back to LDAP.

* Finally we will have WrapperUserProvider, which will delegate most 
method calls to "local" UserProvider. Just methods like verifyCredential 
will be eventually delegated to SyncProvider (if credential is supported 
and user has link to this provider) . Also method "addUser" will add 
user to both "local" and SyncProvider (if RW_USER is configured for 
SyncProvider) etc.

* Currently we support to configure more AuthenticationProviders per 
realm. I agree that this is probably overhead and most deployments will 
be fine with just provider. In case of read-only LDAP, just users synced 
from LDAP will be authenticated against LDAP (they will have "link" to 
LDAP/Picketlink provider). Newly created users won't have link to LDAP 
and hence their password will be validated against "local" UserProvider.

wdyt?

Marek