[keycloak-dev] User Federation Provider Cache

[Bill Burke]

This redesign is quite complex as you have to take a lot of variables 
into account.  I also am trying to make things backward compatible and 
have the new SPI co-exist with the old.  Just remember though, the 
biggest reason for this rewrite is to AVOID IMPORTING!

Here's what I'm currently doing:

Federated Data
--------------
There will be a built-in federated user data service.  It exists to 
store additional data about a user, for example:

public interface UserAttributeFederatedStorage {
     void setSingleAttribute(RealmModel realm, UserModel user, String 
name, String value);
     void setAttribute(RealmModel realm, UserModel user, String name, 
List<String> values);
     void removeAttribute(RealmModel realm, UserModel user, String name);
     MultivaluedHashMap<String, String> getAttributes(RealmModel realm, 
UserModel user);
}

There will be federated storage options for all data.

User Storage Provider SPI
-----------------
Current UserProvider is going to be split up into smaller interface. 
SPI will remain the same though.  There is going to be a 
UserStorageProvider SPI.  Instances can implement one or more of the 
smaller UserProvider interfaces.

User Storage Provider instances will be returning UserModel instances. 
This means providers will be responsible for implementing some sort of 
UserAdapter implementation.  These custom UserAdapters will be 
responsible for interacting with the Federated Data Store and their 
custom storage:  Merging data between i.e. LDAP and the Federated 
Storage, managing which store stores what data, managing what data lives 
where.  Going to provide a helper abstract UserAdapter class that helps 
managing this data.

There will be a UserStorageManager class that implements all 
UserProvider instances.  Our caching layer will delegate to that instead 
of the actual storage provider.  The UserStorageManager will work 
similarly to UserFederationManager.  So, it would work like this:

1. session.users().getUserByUsername(realm, username)
2. User cache looks in cache, doesn't find it,
3. UserStorageManager.getUserByUsername()
4. UserStorageManager iterates all UserStorageProviders looking to see 
which one offers session.users().getUserByUsername() and invoke it.
5. LDAP.getUserByUsername()
6. LDAP finds the user, creates a UserAdapter, returns it.
7. User cache gets the UserModel returned by LDAP, it caches exactly 
like it is implemented now.  The LDAP UserAdapter, is responsible for 
pulling data and merging it from LDAP store and Federation Store.

User ids are an URN

"f:" + providerId + ":" + provider-user-metadata

UserStorageManager.getUserById() will parse the ID and call directly on 
the UserStorageProvider that initially loaded the user.

Updates are invoked directly on UserModel class.  LDAP UserAdapter will 
be responsible for figuring out what data is stored/updated in LDAP and 
what should be stored in Federated Storage.

More comments follow:


On 6/15/16 4:47 AM, Marek Posolda wrote:
> Hi Bill,
>
> few notes from me. I am mostly adding the scenarios, which will be good
> to support/improve or which we should still keep supporting after
> refactoring IMO. Sorry for bit longer email :)
>
> Note: When I talk "LDAP", I usually mean "LDAP or any other 3rd party
> federation storage"
>
>
> Persistent vs. Transient modes
> ----------------------------------------
>
> I think we should still keep some support for "persistent" mode (the
> case when user is imported to Keycloak local DB).
>

I want to get rid of import and only have it as an option for Brokering 
or for transient users (users that can't be looked up from a store out 
of band).  The LDAP adapter will be responsible for knowing what data is 
stored in LDAP and what data is stored in the Federated Store, so there 
is no need to "import"


> Assuming the scenario:
> - john is imported from LDAP with "unsynced" mode
> - john changes his password in account management. LDAP is unsynced
> (read-only), so the password needs to be changed in Keycloak storage
> - After server restart, john should have set the new password, not the
> old one from LDAP. Hence the only possibility seems to keep support for
> "import" this user into Keycloak DB and have user persistent.
>
> The similar situation applies for any update of user, which can't be
> saved back to LDAP (either because LDAP is read-only or because it
> doesn't support store the keycloak metadata like social links, consents,
> required actions etc)
>
> Regarding this, I am thinking about possible import modes like:
> 1) Persistent : User attributes from LDAP are imported into Keycloak DB
> (same behaviour like now)
>

No more importing.

> 2) Transient : User attributes from LDAP are not imported into Keycloak
> DB, but just cached locally. Any updates of the user are either dropped
> after server restart or disallowed. For example: Maybe consents can be
> just dropped, but some other things like requiredRoles should be rather
> disallowed to change? For example if admin adds some requiredAction to
> user "john", the requiredAction shouldn't disappear after server
> restart. This would be a security hole IMO. So in that case, if admin
> tries to manually add requiredAction to such user, the exception will be
> thrown so admin is aware that this is not supported. Anyway, for support
> any writing to cache, the cache will need to be replicated though (for
> example: user consent saved into cache on node1 should be visible on
> node2 as well).
>


> 3) Hybrid : LDAP user is not imported into Keycloak DB immediatelly.
> They are imported "on-demand" after user is updated and the update can't
> be saved to LDAP
>

No more on-demand loading of data.  IIRC, we discussed this in Brno. 
LDAP users will be cached in the same way JPA Users currently are.



>
> Caching
> -----------
>
> I hope new SPI will allow us more flexibility to support various
> scenarios. It will be good to support at least those IMO:
>

See above.  LDAP users will be cached in same way JPA Users currently are.

> 3) I agree that will be cool to support different expiration time based
> if it's LDAP user or just Keycloak-local user. Infinispan allows it
> though with something like : cache.put("ldapuser", ldapUser, 60,
> TimeUnit.SECONDS);
>

I think this will be an option that we will want to support.

>
>
> Transactions and 3rd party updates
> -----------------------------------------------
>
> - Will be good to improve registration of user to LDAP. Ideally during
> registration new user to LDAP, we should allow to send all data at once.
> (currently UserFederationProvider.register supports sending just
> username). Also we should allow to specify if register to 3rd party
> provider should be done *before* or *after* the registration to Keycloak
> local DB. For details, see https://issues.jboss.org/browse/KEYCLOAK-1075
> and all the comments from users...
>
> - Also updating user should be ideally at once. For example if you call:
> user.setFirstName("john");
> user.setLastName("Doe");
> user.setEmail("john at email.cz");
>
> we shouldn't have 3 update calls to LDAP, but just one. Maybe we can
> address this with transaction abstraction? I've already did something
> for LDAP provider (see TxAwareLDAPUserModelDelegate ), however will be
> good to provide something more generic for user storage SPI. Then when
> KeycloakTransactionManager.commit is called, the data are send to
> federation storage
>

This would be an implementation detail.  Similar to JPA Users, the LDAP 
UserAdapter will be remembered per session.  The LDAP adapter could 
queue up updates then at commit time flush them all.

Maybe you should consider writing an LDAP version of JPA?  ;-)  Probably 
a lot of code could be borrowed from PL IDM API.



>
> Sync users
> --------------
> We should still keep the option to sync users into Keycloak DB as we
> have now. Note some persistent storages like LDAP are limited with
> pagination. So the easiest possibility for some admins is just to sync
> users, so they can easily search them in admin console.
>
Doing a full import just to support pagination is overkill.  I'm 
guessing that a lot of deployments will not manage users through the 
Keycload admin console.  We can offer a manual a Sync SPI that providers 
can implement.




>
> Proxy yes/no?
> ------------------
>

Proxy yes, but it won't work the same.  See above.

Bill