[keycloak-dev] Import users from new User Federation

[Stian Thorgersen]

With regards to decommissioning I don't think it's up to the user store to
do it. We should have a separate migration manager or something that takes
care of it. The migration of users from one store is a separate piece of
logic to the storage of users in the first place. It should be possible to
create a re-usable migration manager that can do the job both for our built
in LDAP store and a custom relation db store. Especially when you consider
things like:

* For a migration that happens when users authenticate (LDAP for example)
you want to be able to display progress in the admin console
* For remaining users at some point you want to decide if they should be
dropped, imported without password and if a password recovery email should
be sent
* Probably more stuff to make it real nice

I added https://issues.jboss.org/browse/KEYCLOAK-3478 to cover user
migration (or decommissioning of a user provider, not sure what's the best
name for it). Ideal would be to have it included in 2.3, but I don't think
we have the resources to do that.

On 24 August 2016 at 11:58, Marek Posolda <mposolda at redhat.com> wrote:

> On 23/08/16 17:58, Bill Burke wrote:
>
>
>
> On 8/23/16 10:32 AM, Marek Posolda wrote:
>
> On 23/08/16 15:04, Bill Burke wrote:
>
>
>
> On 8/23/16 3:39 AM, Marek Posolda wrote:
>
> On 19/08/16 15:52, Bill Burke wrote:
>
>
>
> On 8/19/16 2:37 AM, Stian Thorgersen wrote:
>
>
>
> On 18 August 2016 at 20:30, Bill Burke < <bburke at redhat.com>
> bburke at redhat.com> wrote:
>
>>
>> On 8/18/16 4:59 AM, Stian Thorgersen wrote:
>> > Bill,
>> >
>> > Are you planing to have an option to allow import of users with the
>> > new user federation SPI? I'm not convinced we should completely remove
>> > this option.
>> >
>>
>> The only callback that does not exist in the new SPI is
>> validateAndProxy().  With the current federation SPI, the developer
>> implements everything themselves for import.  There are no
>> synchronization APIs/SPIs either.
>>
>
> Sounds like we're removing built-in features around synchronization just
> to make the user have to do everything themselves.
>
> I think you misinterpreted me,  The old User Federation SPI forces the
> developer to write all the import code themselves.  The old User Federation
> SPI does not have any synchronization callbacks, methods or interfaces
> other than validateAndProxy(), the logic of which the user has to write
> themselves too.
>
>
>
>
>> > Some use-cases I could imagine:
>> >
>> > * Allow users to authenticate even if LDAP server is down
>> Our current LDAP provider will not work if LDAP is down, even with the
>> import :)
>>
>
> Yes, I know. However, the fact that we don't currently support it doesn't
> mean we shouldn't in the future.
>
> If the user can only be authenticated via LDAP, an offline mode is not
> possible.  In other words, if LDAP does not expose the password of a user
> (so it can be imported), then offline mode is not possible.  It would only
> be possible if the user has logged in at least once, then the validated
> password could be imported.
>
>
> So, do you still think we should support import/offline mode given all
> this?
>
> From some recent discussions I saw, it seems that quite many people are
> interested in the "import-and-forget" mode. So they need to import user
> from their old legacy store (3rd party storage or LDAP) but once user is
> fully in Keycloak DB, they want to completely forget about the 3rd party
> storage and do all operations around this user against Keycloak DB.
>
> The credentials/password validation seems to be the most complicated part
> around this as you pointed, as the password needs to be first successfully
> validated against 3rdparty storage or LDAP . Then once password is
> successfully validated and updated to Keycloak DB, user can be "forgotten"
> and unlinked from the federationProvider. I hope the new SPI has a way to
> deal with this usecase? Or at least have a hook, so the people can easily
> unlink the user by themselves whenever they want.
>
> As I said  before, the current SPI does not have any support for import.
> It also does not have any SPIs around synchronization or any
> synchronization buttons in the admin console.  It is up to the developer to
> write the code to import the user.  And our current LDAP implmementation is
> not "import and forget", you already mentioned password validation, but
> there is also validateAndProxy which is called every time the user is
> accessed and which hits LDAP every time.  There's also no way to unlink the
> user.
>
> Not right now, but it seems that many people consider the
> "import-and-forget" as important usecase? You just want to import the users
> from 3rd party storage or LDAP, but you need to do in multiple steps and
> "wait" until password is successfully validated for the first time.
>
> As an example this blogpost from Scott Rossillo
> <https://tech.smartling.com/migrate-to-keycloak-with-zero-downtime-8dcab9e7cb2c#.1e8sy1o8n>
> https://tech.smartling.com/migrate-to-keycloak-with-zero-
> downtime-8dcab9e7cb2c#.1e8sy1o8n, which AFAIK seemed to have some
> positive feedback from more community users.
>
> I don't know how deeply to go with directly supporting it at SPI level.
> However IMO it will be good to have at least same level like the current
> UserFederation SPI. So at least at some point (ie. after successful
> password validation), the people can manually unlink the 3rd party provider
> from the user and migrate all the data to Keycloak DB and then use it just
> from Keycloak DB.
>
> Ok, good feedback.  You are convincing me.  Are we absolutely sure this is
> actually a best practice and not an anti-pattern?  Seems scary to be half
> in and half out.  I guess it does make sense if you need to keep something
> like LDAP up for legacy apps.
>
>
> Just thinking around this we should have an additional interface for
> imports:
>
> interface UserStorageSynchornization {
>
> void validate(UserModel).
> void synchronize()
> void unlink()
>
>
> }
>
>
>
> validate is called whenever a user is looked up.  Possibly used to find
> deleted users and to synchronize updates on both sides on demand.  I want
> to add cache policies per provider, so maybe validate is called only when
> pulled from persistence storage and not cache.
>
> I don't think we need different synchronize methods.  We should instead
> store last sync timestamp and last updated timestamp for each user and add
> queries to local storage to find users for a specific provider that were
> synced and/or updated after a certain time.   Then the synchronize
> implementation can make the assessment on what to synchronize or not.  I'd
> also like to be able to fire off synchronization in the background and to
> obtain a status on it from the admin console.  If it fails, how many users
> synchronized, and error message, etc.
>
> The support for "background" will be nice. That's what we missed until now.
>
> If I understand correctly, this will sync between any UserStorage to any
> other UserStorage, so it will defacto provider 2-ways sync ?
>
>
> unlink() would just be a callback whenever the admin console fires of an
> unlink all users event.
>
> Sounds good to have callback for unlink. Still it will be good to have
> possibility to unlink individual users at specific moment (again, the
> example when you want to unlink user "john" from LDAP after he successfuly
> authenticated to LDAP with password "bar" as you can then immediatelly
> update the password to Keycloak DB and hence you don't need LDAP anymore).
>
> So for example the usecase like:
> 1) Keycloak configured with LDAP with 1000 users
>
> 2) 600 users authenticated with their passwords during week1, so they were
> already unlinked from LDAP as their passwords (And whole profile) imported
> to Keycloak DB
>
> 3) After week1, admin triggers the "unlink" event from admin console. At
> this point he wants to forcefully unlink remaining 400 users from LDAP and
> import them to Keycloak DB. He will also need to reset their password and
> send them email etc. This all can be implemented in the "unlink" callback
> method right?
>
> Not sure whether to support alternative of step3, like:
> 3.a) After week1, admin sends email to remaining 400 users like "Hey,
> please login in next 7 days. Otherwise your password will be restarted."
> 3.b) After week2, the real unlink is done with the password reset of
> users, which didn't login in both week1 and week2.
>
> Not sure if just "unlink" method is sufficient then...
>
> Overally it seems that the userStorage is super-complicated as various
> people have various use-cases and almost everyone has a bit different
> requirements and it's almost impossible to properly support everything. So
> IMO it's good if SPI has enough callbacks/extension points, so people can
> hook their actions and eventually implement themselves exactly what they
> want.
>
> Marek
>
>
> Bill
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/keycloak-dev/attachments/20160829/0529ed06/attachment-0001.html