5:37 p.m.
To expand on our discussion around AuthenticationProvider. With the
currrent implementation of AuthenticationProvider, Keycloak admin
console cannot be used to manage external users until the user has
logged in and a UserModel creation has happened. We would need to
duplicate the "import" logic that is within AuthenticationManager
wherever UserProvider.getUserByXXXX() is invoked. Not only that, but
role mappings and other claim data may be contained in external storage.
The authentictionProvider interface is inadequate for these scenarios.
My thinking is that for federation and sync would come in three flavors:
* Keycloak storage. What we currently implement.
* Full external storage. The app developer will use their own
persistence model to store and manage users. In this case, they need to
fully implement the UserProvider interface. Handle social links,
required actions, role mappings etc.
* A sync/import model. In this case, keycloak is augmenting an existing
user storage. This would require partially or fully importing the user
from the external store into Keycloak UserModel/Provider storage. The
import is required so that Keycloak can managed REQUIRED ACTIONS, social
links, and add additional or manage existing credential types. We also
need the import to establish relationships between the user and
UserSession metadata. We may also have to handle role mappings within
Keycloak. A sync/import model example could be an LDAP store that just
contains username, email, first/last, and password. And the admin wants
to add role mappings and totp.
How would it work?
I think all this should be done through a common UserProvider interface.
Initially be able to federate only 1 additional external store as I'm
worried that there may be the possibility of duplicate user names if you
allow more than one and you'd need a way at login to pick which one to
log in from.
UserProviderFactory would specify its feature set:
Query Capabilities:
REGISTRATION
QUERY_BY_USER_ID
QUERY_BY_USERNAME
QUERY_BY_EMAIL
QUERY_BY_ATTRIBUTES
And its supported credential validation:
PASSWORD_VALIDATION
TOTP_VALIDATION
CERT_VALIDATION
And its supported credential storage:
PASSWORD_STORAGE
TOTP_STORAGE
CERT_STORAGE
There would a FederatedUserProvider to manage local and federated storage.
*Locating a user:*
1. FederatedUserProvider queries local storage first.
2. Check realm to see if federated storage configured
3. Load that provider
4. See if that provider supports query method, if not return null
5. Call query method on provider
Provider performs the following on user location:
6. Do custom query to find user
7. Get a local storage session and create a user within local storage
importing whatever can be imported.
8. Mark in local user storage which provider loaded it.
9. return the local user
*Required Actions and Credential management*
1. Find provider used to load user
2. Check credential storage options
3. If provider doesn't support credential update/storage, then store in
local storage
*Authentication*
1. Find user in local storage
2. See if credential is stored in local storage, validate from there if so.
3. If not stored, check provider validation options. Abort if not supported
4. Invoke on external provider to validate credentials supported by that
provider
Did I cover everything?
--
Bill Burke
JBoss, a division of Red Hat
http://bill.burkecentral.com
7:41 a.m.
On 9.7.2014 23:37, Bill Burke wrote:
To expand on our discussion around AuthenticationProvider. With
the
currrent implementation of AuthenticationProvider, Keycloak admin
console cannot be used to manage external users until the user has
logged in and a UserModel creation has happened. We would need to
duplicate the "import" logic that is within AuthenticationManager
wherever UserProvider.getUserByXXXX() is invoked. Not only that, but
role mappings and other claim data may be contained in external storage.
The authentictionProvider interface is inadequate for these scenarios.
My thinking is that for federation and sync would come in three flavors:
* Keycloak storage. What we currently implement.
* Full external storage. The app developer will use their own
persistence model to store and manage users. In this case, they need to
fully implement the UserProvider interface. Handle social links,
required actions, role mappings etc.
* A sync/import model. In this case, keycloak is augmenting an existing
user storage. This would require partially or fully importing the user
from the external store into Keycloak UserModel/Provider storage. The
import is required so that Keycloak can managed REQUIRED ACTIONS, social
links, and add additional or manage existing credential types. We also
need the import to establish relationships between the user and
UserSession metadata. We may also have to handle role mappings within
Keycloak. A sync/import model example could be an LDAP store that just
contains username, email, first/last, and password. And the admin wants
to add role mappings and totp.
How would it work?
I think all this should be done through a common UserProvider interface.
Initially be able to federate only 1 additional external store as I'm
worried that there may be the possibility of duplicate user names if you
allow more than one and you'd need a way at login to pick which one to
log in from.
UserProviderFactory would specify its feature set:
Query Capabilities:
REGISTRATION
QUERY_BY_USER_ID
QUERY_BY_USERNAME
QUERY_BY_EMAIL
QUERY_BY_ATTRIBUTES
And its supported credential validation:
PASSWORD_VALIDATION
TOTP_VALIDATION
CERT_VALIDATION
And its supported credential storage:
PASSWORD_STORAGE
TOTP_STORAGE
CERT_STORAGE
There would a FederatedUserProvider to manage local and federated storage.
*Locating a user:*
1. FederatedUserProvider queries local storage first.
2. Check realm to see if federated storage configured
3. Load that provider
4. See if that provider supports query method, if not return null
5. Call query method on provider
Provider performs the following on user location:
6. Do custom query to find user
7. Get a local storage session and create a user within local storage
importing whatever can be imported.
8. Mark in local user storage which provider loaded it.
9. return the local user
*Required Actions and Credential management*
1. Find provider used to load user
2. Check credential storage options
3. If provider doesn't support credential update/storage, then store in
local storage
*Authentication*
1. Find user in local storage
2. See if credential is stored in local storage, validate from there if so.
3. If not stored, check provider validation options. Abort if not supported
4. Invoke on external provider to validate credentials supported by that
provider
Did I cover everything?
How about creating of users? Will be new users added just to local
storage and not to the "external provider" storage?
Also not sure what's your plan for role mappings? Will be role mappings
loaded from "external provider" just in case that provider supports role
mappings? But what if roles from provider doesn't exist in Keycloak?
And finally getUsers(), searchForUser, searchForUserByAttributes in both
non-paginated and paginated versions?
IMO Federation approach is quite tricky. We already did this in GateIn
(picketlink 1.X) and had quite bad issues. Especially performance
issues. For example getUsers() with pagination+sorting is almost
impossible if you have some users just in "local" store, some users in
"external" store, and some synced in both. For example if there are
users "a", "b" in external storage, then users "b",
"c" in local storage
and you call getUsers() you would need to merge them from both storages
to have correct list with "a", "b" and "c". If you want this
with
pagination+sorting then it's much more tricky... If you want proper
results, you defacto need to load all users from both stores and sort
them programmatically, which completely bypasses pagination.
That's why I would prefer "sync" among "federation" approach. In
shortcut with "sync" approach will be users (and eventually their role
mappings) synced from external store into Keycloak and most of the
UserProvider methods would deal just with "local" provider. Only
exception are authentication related methods, and also "update" methods,
which will need to update local user and also "external" users. But
SyncProvider won't need to implement methods like getUsers(),
searchFor**, getUserBy** etc as all users will be available locally.
In details, I can imagine to have things working this way:
* There will be Sync SPI configured per realm, which will allow to sync
users from "external store" to Keycloak.
* It will be possible to configure when to sync users. For example it
can be full sync from LDAP at server startup or triggered from admin
console. Then some periodic sync (triggered for example once per day).
* I am not sure if we need full sync from Keycloak to "external
provider" but probably not. Once user, role or user credential will be
updated in Keycloak, it will be also immediatelly synced to external
provider (if provider is not read-only).
* It depends on SyncProvider implementation, what user data are synced.
For example in case of LDAP it will be just email+first,lastName and
eventually role mappings.
* SyncProvider may also support roles + role mappings from external
store. SyncProvider will be able to import roles into RealmProvider and
then particular role mappings into UserProvider.
* Users synced from "external provider" will have link to this provider
similarly like it's now. In case that provider is read-write, then all
newly created users in Keycloak will be synced into "external provider"
immediatelly also with the credential (if syncProvider supports
particular credential)
I would propose the interface like this to handle both
sync/authentication (not sure if "SyncProvider" is good name, maybe
rather "ExternalUserProvider"). I can imagine something like this:
public interface SyncProvider {
/**
* Invoked when new user is registered (or added through admin
console) and just if SyncProvider supports RW_USER
*
* @return user with added "link" . Link is filled by external
provider (for example ID from LDAP)
*/
UserModel addUser(UserModel user);
/**
* Invoked when credential of user is changed and just if:
SyncProvider is RW_PASSWORD (or RW_TOTP) and user has "link" to provider
*/
void updateCredential(UserModel user, Credential credential);
/**
* Invoked when user has link to externalProvider and particular
credential is supported. (Otherwise credential is validated against
"local" UserProvider)
*/
boolean validateCredential(UserModel user, Credential credential);
/**
* Invoked just when full sync from "external store" to Keycloak is
requested. Method could be empty if SyncProvider doesn't suppport Roles
+ Role mappings
* Not sure if role also needs to have "link" . Probably not.
*/
List<RoleModel> getRoles();
/**
* Invoked just when full sync from "external store" to Keycloak is
requested. Method could be empty if SyncProvider doesn't support Roles +
Role mappings
*/
void updateRole(RoleModel role);
/**
* Invoked just when full sync from "external store" to Keycloak is
requested. It's invoked *after* getAllRoles(), so Keycloak knows roles
mapped to users
*/
List<UserModel> getAllUsers();
/**
* List features supported by particular provider
*/
List<Feature> getMetadata();
}
For metadata I can see the list with RW_ROLE, RO_ROLE (If provider
supports storing of roles and role mappings), RW_USER, RO_USER (If
provider supports storing of users), RW_PASSWORD, RO_PASSWORD (If
provider supports passwords), RO_TOTP, RW_TOTP (If provider supports
storing of totp)
In admin console, it will be possible to configure just subset of
available features. For example LDAPSyncProvider supports both RO and RW
for users,roles and passwords, but for RO deployments, people can check
just RO_ROLE, RO_USER, RO_PASSWORD, which means that LDAP is read-only
and users will be synced just from LDAP to Keycloak and passwords will
be verified against LDAP, but nothing will be written back to LDAP.
* Finally we will have WrapperUserProvider, which will delegate most
method calls to "local" UserProvider. Just methods like verifyCredential
will be eventually delegated to SyncProvider (if credential is supported
and user has link to this provider) . Also method "addUser" will add
user to both "local" and SyncProvider (if RW_USER is configured for
SyncProvider) etc.
* Currently we support to configure more AuthenticationProviders per
realm. I agree that this is probably overhead and most deployments will
be fine with just provider. In case of read-only LDAP, just users synced
from LDAP will be authenticated against LDAP (they will have "link" to
LDAP/Picketlink provider). Newly created users won't have link to LDAP
and hence their password will be validated against "local" UserProvider.
wdyt?
Marek
9:47 a.m.
On 7/10/2014 7:41 AM, Marek Posolda wrote:
That's why I would prefer "sync" among "federation" approach.
Which is exactly what I'm proposing. Except it allows for syncs/imports
into Keycloak on demand. You already use AuthenticationProvider in this
manner!
My impression of a sync model is that Keycloak works with user data in
local Keycloak storage. User metadata is imported from external storage
(either on demand or at startp and periodically synced). Keycloak can
manage/modify/augment this imported user data locally. The changelog is
obtained from the Sync API and updates sent back to the external storage.
In
shortcut with "sync" approach will be users (and eventually their role
mappings) synced from external store into Keycloak and most of the
UserProvider methods would deal just with "local" provider. Only
exception are authentication related methods, and also "update" methods,
which will need to update local user and also "external" users. But
SyncProvider won't need to implement methods like getUsers(),
searchFor**, getUserBy** etc as all users will be available locally.
The problem with this approach is that new users don't get imported into
Keycloak until the sync gets invoked. Syncing more than once per day,
or even once per week may not be feasible. If the external storage does
not have a changelog, syncing would involve iterating through each and
every user in external storage and syncing it with the keycloak
database. A full sync could take hours.
In details, I can imagine to have things working this way:
* There will be Sync SPI configured per realm, which will allow to sync
users from "external store" to Keycloak.
+1
* It will be possible to configure when to sync users. For example
it
can be full sync from LDAP at server startup or triggered from admin
console. Then some periodic sync (triggered for example once per day).
+1
* I am not sure if we need full sync from Keycloak to "external
provider" but probably not. Once user, role or user credential will be
updated in Keycloak, it will be also immediatelly synced to external
provider (if provider is not read-only).
If you don't implement through the UserProvider/UserModel interface then
you would need an Event model for which the SyncProvider listens to so
that it can sync the user on demand.
* It depends on SyncProvider implementation, what user data are
synced.
For example in case of LDAP it will be just email+first,lastName and
eventually role mappings.
* SyncProvider may also support roles + role mappings from external
store. SyncProvider will be able to import roles into RealmProvider and
then particular role mappings into UserProvider.
* Users synced from "external provider" will have link to this provider
similarly like it's now. In case that provider is read-write, then all
newly created users in Keycloak will be synced into "external provider"
immediatelly also with the credential (if syncProvider supports
particular credential)
I would propose the interface like this to handle both
sync/authentication (not sure if "SyncProvider" is good name, maybe
rather "ExternalUserProvider"). I can imagine something like this:
public interface SyncProvider {
I think your SyncProvider interface is very limiting and even unusable.
For example, getAllUsers() is completely unfeasible if there is a
large user database.
IMO it would be something like this:
public interface ChronJob {
void invoke(KeycloakSessionFactory factory);
}
ChronJob would be scheduled to run at boot time and/or periodically by
the admin console. Periodic and boot time syncs would be implemented
here. The sync operation needs to have full control of when
transactions begin and end so updates/creates/deletes can be executed in
batches.
On demand syncing would be done through an implementation of
UserProvider as I proposed earlier. On-demand syncing is:
* Importing a user on demand
* Updating external storage on demand.
We would also have an event changelog that would be implemented as, or
act like a persistent JMS Topic. This event topic shouldn't be used to
implement on-demand sync, but rather for when the external storage is
read only.:
interface ChangeEvent {
enum EventType {
CREATE, UPDATE, DELETE
}
long getTimestamp();
EventType getEventType();
String getItemId();
String getItemType(); // UserModel, RoleModel
}
EventListeners could perform the syncs on demand. Or, a ChronJob could
be something like a persistence JMS Topic subscriber and replay change
events.
We could decide that we don't support user updates for read-only stores.
Then, in that case, IMO, we don't need a changelog event queue.
--
Bill Burke
JBoss, a division of Red Hat
http://bill.burkecentral.com
3:01 p.m.
On 10.7.2014 15:47, Bill Burke wrote:
On 7/10/2014 7:41 AM, Marek Posolda wrote:
>
> That's why I would prefer "sync" among "federation"
approach.
Which is exactly what I'm proposing. Except it allows for
syncs/imports into Keycloak on demand. You already use
AuthenticationProvider in this manner!
oops, I though that with AuthProvider you
don't like the fact that just
LDAP users authenticated with Keycloak are available in Keycloak DB;-)
My impression of a sync model is that Keycloak works with user data in
local Keycloak storage. User metadata is imported from external
storage (either on demand or at startp and periodically synced).
Keycloak can manage/modify/augment this imported user data locally.
The changelog is obtained from the Sync API and updates sent back to
the external storage.
> In
> shortcut with "sync" approach will be users (and eventually their role
> mappings) synced from external store into Keycloak and most of the
> UserProvider methods would deal just with "local" provider. Only
> exception are authentication related methods, and also "update" methods,
> which will need to update local user and also "external" users. But
> SyncProvider won't need to implement methods like getUsers(),
> searchFor**, getUserBy** etc as all users will be available locally.
>
The problem with this approach is that new users don't get imported
into Keycloak until the sync gets invoked. Syncing more than once per
day, or even once per week may not be feasible. If the external
storage does not have a changelog, syncing would involve iterating
through each and every user in external storage and syncing it with
the keycloak database. A full sync could take hours.
Yes, but how to do it
differently?
(a) For methods like: UserProvider.getUsers(), UserProvider.searchFor***
we can either:
1) retrieve users just from "local" store
2) Federate users and merge them from both "local" and "external"
store.
Federation approach (2) has quite bad performance issues (especially
with pagination+sorting), so I would prefer (1). But this really
requires full sync from external store into KC IMO.
(b) Then we have methods for retrieve single user:
getUserBy*** --- here we can try to retrieve user from 'local' store and
fallback to 'external'. In that case, user will be synced on demand to
KC database (similar approach like AuthenticationProvider), but then we
have the issue that just those users, who were retrieved from
externalStore will be available in KC.
I think that we can either:
(1) Federate users from both stores
(2) Temporarily allow that just some users from LDAP are available in KC
database (those who were authenticated or for which getUserBy*** was called)
(3) Do full sync
I don't know if you see some other solution, but I would say that if we
want to avoid (1) and (2) we really need to do full sync. So I would
imagine that if someone configure externalStore, he will also need to
trigger full sync to KC to have all users added to Keycloak.
>
>
>
> In details, I can imagine to have things working this way:
>
> * There will be Sync SPI configured per realm, which will allow to sync
> users from "external store" to Keycloak.
>
+1
> * It will be possible to configure when to sync users. For example it
> can be full sync from LDAP at server startup or triggered from admin
> console. Then some periodic sync (triggered for example once per day).
>
+1
> * I am not sure if we need full sync from Keycloak to "external
> provider" but probably not. Once user, role or user credential will be
> updated in Keycloak, it will be also immediatelly synced to external
> provider (if provider is not read-only).
>
If you don't implement through the UserProvider/UserModel interface
then you would need an Event model for which the SyncProvider listens
to so that it can sync the user on demand.
yep, I think that it works to implement
this in UserProvider (and
RealmProvider for roles). Basically you have something like
"WrapperUserProvider" and once add/update/delete user is invoked, it
will perform the operation on "local" UserProvider and then invoke
particular method on SyncProvider to sync it back into external storage.
> * It depends on SyncProvider implementation, what user data are synced.
> For example in case of LDAP it will be just email+first,lastName and
> eventually role mappings.
>
> * SyncProvider may also support roles + role mappings from external
> store. SyncProvider will be able to import roles into RealmProvider and
> then particular role mappings into UserProvider.
>
> * Users synced from "external provider" will have link to this provider
> similarly like it's now. In case that provider is read-write, then all
> newly created users in Keycloak will be synced into "external provider"
> immediatelly also with the credential (if syncProvider supports
> particular credential)
>
> I would propose the interface like this to handle both
> sync/authentication (not sure if "SyncProvider" is good name, maybe
> rather "ExternalUserProvider"). I can imagine something like this:
>
>
> public interface SyncProvider {
>
I think your SyncProvider interface is very limiting and even
unusable. For example, getAllUsers() is completely unfeasible if
there is a large user database.
oops, sorry. It's maybe not clear from method
name and javadoc that
"getAllUsers" is intended to be used for sync *from* External storage
*to* Keycloak. The idea is that "getAllUsers" (or
"syncAllUsersFromExternalStore" or whatever it's named) will be called
by admin when he wants full sync from external storage to KC.
IMO sync from Keycloak to external storage is quite easy. We can either
do it immediatelly when add/update/delete user/role is invoked or we can
use some event approach you proposed with periodic (or on demand) sync
of just "dirty" entities. We already have AuditProvider, which could be
eventually used.
Personally I would prefer to always do sync from KC immediatelly as
event queue doesn't always work. For example when you have RW LDAP and
you register new user in Keycloak, you would need to save this user and
his password immediatelly into external store. Save it somewhere in
Keycloak database and then clear once Chron Job is triggered doesn't
work IMO as it would mean that we will need to save passwords in
plain-text until they are synced into LDAP. Which doesn't make much
sense IMO.
So for sync from Keycloak to externalStore, I would personally always do
it immediatelly after operation is performed on
UserProvider/RealmProvider. It's not so big overhead IMO.
But the main issue is sync from externalStore to Keycloak. We can easily
add changelog to Keycloak, but we don't have control under external
storage...
I don't know if LDAP servers support changelog, probably yes but I don't
think that it's standardized, maybe some LDAP servers support
"proprietary" way of doing changelog. I can check if some LDAP supports
this and how it works with them...
But fact is that if externalProvider doesn't support changelog, then how
to do sync without something like "getAllUsers" (or "syncAllUsers")? I
know that this is quite overhead, but I really don't see how to do
"partial" sync from 3rd party provider if it doesn't support changelogs...
Marek
IMO it would be something like this:
public interface ChronJob {
void invoke(KeycloakSessionFactory factory);
}
ChronJob would be scheduled to run at boot time and/or periodically by
the admin console. Periodic and boot time syncs would be implemented
here. The sync operation needs to have full control of when
transactions begin and end so updates/creates/deletes can be executed
in batches.
On demand syncing would be done through an implementation of
UserProvider as I proposed earlier. On-demand syncing is:
* Importing a user on demand
* Updating external storage on demand.
We would also have an event changelog that would be implemented as, or
act like a persistent JMS Topic. This event topic shouldn't be used
to implement on-demand sync, but rather for when the external storage
is read only.:
interface ChangeEvent {
enum EventType {
CREATE, UPDATE, DELETE
}
long getTimestamp();
EventType getEventType();
String getItemId();
String getItemType(); // UserModel, RoleModel
}
EventListeners could perform the syncs on demand. Or, a ChronJob
could be something like a persistence JMS Topic subscriber and replay
change events.
We could decide that we don't support user updates for read-only
stores. Then, in that case, IMO, we don't need a changelog event queue.
9:05 a.m.
This sounds like exactly what we need. There are quite a lot of kinks to work out though.
With that in mind I think we should limit the scope of this for 1.0.final to something
along the lines of what AuthProvider does now. Question is for 1.0.final do we:
* Extract user related methods into UserProvider - my vote is yes
* Deprecate AuthProvider and provide something similar through UserProvider - I'm
unsure about this
* Try to cover all basis in UserProvider such as sync, etc - my vote is no
With regards to SPIs in general there are some that are aimed at internal use only, others
that are aimed at public use, and I guess we could flag certain SPIs as likely to change
in the future (such as AuthProvider).
----- Original Message -----
From: "Marek Posolda" <mposolda(a)redhat.com>
To: "Bill Burke" <bburke(a)redhat.com>, keycloak-dev(a)lists.jboss.org
Sent: Thursday, 10 July, 2014 8:01:33 PM
Subject: Re: [keycloak-dev] sync/federation requirements/ideas
On 10.7.2014 15:47, Bill Burke wrote:
>
>
> On 7/10/2014 7:41 AM, Marek Posolda wrote:
>>
>> That's why I would prefer "sync" among "federation"
approach.
>
> Which is exactly what I'm proposing. Except it allows for
> syncs/imports into Keycloak on demand. You already use
> AuthenticationProvider in this manner!
oops, I though that with AuthProvider you don't like the fact that just
LDAP users authenticated with Keycloak are available in Keycloak DB;-)
>
> My impression of a sync model is that Keycloak works with user data in
> local Keycloak storage. User metadata is imported from external
> storage (either on demand or at startp and periodically synced).
> Keycloak can manage/modify/augment this imported user data locally.
> The changelog is obtained from the Sync API and updates sent back to
> the external storage.
>
>
>> In
>> shortcut with "sync" approach will be users (and eventually their
role
>> mappings) synced from external store into Keycloak and most of the
>> UserProvider methods would deal just with "local" provider. Only
>> exception are authentication related methods, and also "update"
methods,
>> which will need to update local user and also "external" users. But
>> SyncProvider won't need to implement methods like getUsers(),
>> searchFor**, getUserBy** etc as all users will be available locally.
>>
>
> The problem with this approach is that new users don't get imported
> into Keycloak until the sync gets invoked. Syncing more than once per
> day, or even once per week may not be feasible. If the external
> storage does not have a changelog, syncing would involve iterating
> through each and every user in external storage and syncing it with
> the keycloak database. A full sync could take hours.
Yes, but how to do it differently?
(a) For methods like: UserProvider.getUsers(), UserProvider.searchFor***
we can either:
1) retrieve users just from "local" store
2) Federate users and merge them from both "local" and "external"
store.
Federation approach (2) has quite bad performance issues (especially
with pagination+sorting), so I would prefer (1). But this really
requires full sync from external store into KC IMO.
(b) Then we have methods for retrieve single user:
getUserBy*** --- here we can try to retrieve user from 'local' store and
fallback to 'external'. In that case, user will be synced on demand to
KC database (similar approach like AuthenticationProvider), but then we
have the issue that just those users, who were retrieved from
externalStore will be available in KC.
I think that we can either:
(1) Federate users from both stores
(2) Temporarily allow that just some users from LDAP are available in KC
database (those who were authenticated or for which getUserBy*** was called)
(3) Do full sync
I don't know if you see some other solution, but I would say that if we
want to avoid (1) and (2) we really need to do full sync. So I would
imagine that if someone configure externalStore, he will also need to
trigger full sync to KC to have all users added to Keycloak.
>
>
>>
>>
>>
>> In details, I can imagine to have things working this way:
>>
>> * There will be Sync SPI configured per realm, which will allow to sync
>> users from "external store" to Keycloak.
>>
>
> +1
>
>> * It will be possible to configure when to sync users. For example it
>> can be full sync from LDAP at server startup or triggered from admin
>> console. Then some periodic sync (triggered for example once per day).
>>
>
> +1
>
>> * I am not sure if we need full sync from Keycloak to "external
>> provider" but probably not. Once user, role or user credential will be
>> updated in Keycloak, it will be also immediatelly synced to external
>> provider (if provider is not read-only).
>>
>
> If you don't implement through the UserProvider/UserModel interface
> then you would need an Event model for which the SyncProvider listens
> to so that it can sync the user on demand.
yep, I think that it works to implement this in UserProvider (and
RealmProvider for roles). Basically you have something like
"WrapperUserProvider" and once add/update/delete user is invoked, it
will perform the operation on "local" UserProvider and then invoke
particular method on SyncProvider to sync it back into external storage.
>
>> * It depends on SyncProvider implementation, what user data are synced.
>> For example in case of LDAP it will be just email+first,lastName and
>> eventually role mappings.
>>
>> * SyncProvider may also support roles + role mappings from external
>> store. SyncProvider will be able to import roles into RealmProvider and
>> then particular role mappings into UserProvider.
>>
>> * Users synced from "external provider" will have link to this
provider
>> similarly like it's now. In case that provider is read-write, then all
>> newly created users in Keycloak will be synced into "external
provider"
>> immediatelly also with the credential (if syncProvider supports
>> particular credential)
>>
>> I would propose the interface like this to handle both
>> sync/authentication (not sure if "SyncProvider" is good name, maybe
>> rather "ExternalUserProvider"). I can imagine something like this:
>>
>>
>> public interface SyncProvider {
>>
>
> I think your SyncProvider interface is very limiting and even
> unusable. For example, getAllUsers() is completely unfeasible if
> there is a large user database.
oops, sorry. It's maybe not clear from method name and javadoc that
"getAllUsers" is intended to be used for sync *from* External storage
*to* Keycloak. The idea is that "getAllUsers" (or
"syncAllUsersFromExternalStore" or whatever it's named) will be called
by admin when he wants full sync from external storage to KC.
IMO sync from Keycloak to external storage is quite easy. We can either
do it immediatelly when add/update/delete user/role is invoked or we can
use some event approach you proposed with periodic (or on demand) sync
of just "dirty" entities. We already have AuditProvider, which could be
eventually used.
Personally I would prefer to always do sync from KC immediatelly as
event queue doesn't always work. For example when you have RW LDAP and
you register new user in Keycloak, you would need to save this user and
his password immediatelly into external store. Save it somewhere in
Keycloak database and then clear once Chron Job is triggered doesn't
work IMO as it would mean that we will need to save passwords in
plain-text until they are synced into LDAP. Which doesn't make much
sense IMO.
So for sync from Keycloak to externalStore, I would personally always do
it immediatelly after operation is performed on
UserProvider/RealmProvider. It's not so big overhead IMO.
But the main issue is sync from externalStore to Keycloak. We can easily
add changelog to Keycloak, but we don't have control under external
storage...
I don't know if LDAP servers support changelog, probably yes but I don't
think that it's standardized, maybe some LDAP servers support
"proprietary" way of doing changelog. I can check if some LDAP supports
this and how it works with them...
But fact is that if externalProvider doesn't support changelog, then how
to do sync without something like "getAllUsers" (or "syncAllUsers")?
I
know that this is quite overhead, but I really don't see how to do
"partial" sync from 3rd party provider if it doesn't support changelogs...
Marek
>
>
> IMO it would be something like this:
>
> public interface ChronJob {
>
> void invoke(KeycloakSessionFactory factory);
> }
>
> ChronJob would be scheduled to run at boot time and/or periodically by
> the admin console. Periodic and boot time syncs would be implemented
> here. The sync operation needs to have full control of when
> transactions begin and end so updates/creates/deletes can be executed
> in batches.
>
> On demand syncing would be done through an implementation of
> UserProvider as I proposed earlier. On-demand syncing is:
>
> * Importing a user on demand
> * Updating external storage on demand.
>
> We would also have an event changelog that would be implemented as, or
> act like a persistent JMS Topic. This event topic shouldn't be used
> to implement on-demand sync, but rather for when the external storage
> is read only.:
>
> interface ChangeEvent {
> enum EventType {
> CREATE, UPDATE, DELETE
> }
>
>
>
> long getTimestamp();
> EventType getEventType();
> String getItemId();
> String getItemType(); // UserModel, RoleModel
> }
>
> EventListeners could perform the syncs on demand. Or, a ChronJob
> could be something like a persistence JMS Topic subscriber and replay
> change events.
>
> We could decide that we don't support user updates for read-only
> stores. Then, in that case, IMO, we don't need a changelog event queue.
>
>
>
>
_______________________________________________
keycloak-dev mailing list
keycloak-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-dev
10:12 a.m.
On 7/11/2014 9:05 AM, Stian Thorgersen wrote:
This sounds like exactly what we need. There are quite a lot of kinks
to work out though.
My vote is we fix AuthProvider and sync now. Do another iteration.
Users *really* want this. You're right, there's a lot of kinks we need
to work out, but we don't want a large set of users we have to move off
of an existing SPI.
--
Bill Burke
JBoss, a division of Red Hat
http://bill.burkecentral.com
10:26 a.m.
On 7/10/2014 3:01 PM, Marek Posolda wrote:
> The problem with this approach is that new users don't get
imported
> into Keycloak until the sync gets invoked. Syncing more than once per
> day, or even once per week may not be feasible. If the external
> storage does not have a changelog, syncing would involve iterating
> through each and every user in external storage and syncing it with
> the keycloak database. A full sync could take hours.
Yes, but how to do it differently?
(a) For methods like: UserProvider.getUsers(), UserProvider.searchFor***
we can either:
1) retrieve users just from "local" store
2) Federate users and merge them from both "local" and "external"
store.
Federation approach (2) has quite bad performance issues (especially
with pagination+sorting), so I would prefer (1). But this really
requires full sync from external store into KC IMO.
(b) Then we have methods for retrieve single user:
getUserBy*** --- here we can try to retrieve user from 'local' store and
fallback to 'external'. In that case, user will be synced on demand to
KC database (similar approach like AuthenticationProvider), but then we
have the issue that just those users, who were retrieved from
externalStore will be available in KC.
I think that we can either:
(1) Federate users from both stores
(2) Temporarily allow that just some users from LDAP are available in KC
database (those who were authenticated or for which getUserBy*** was
called)
(3) Do full sync
I don't know if you see some other solution, but I would say that if we
want to avoid (1) and (2) we really need to do full sync. So I would
imagine that if someone configure externalStore, he will also need to
trigger full sync to KC to have all users added to Keycloak.
Really depends on the UserProvider implementation IMO. The UserProvider
doesn't have to import the users on search/getAll operations. The
UserModels returned could be proxies. Probably harder than I'm stating,
but its quite possible its not.
Personally I would prefer to always do sync from KC immediatelly as
event queue doesn't always work.
+1. We'll probably eventually have to have an event queue, but I'd like
to avoid it.
But fact is that if externalProvider doesn't support changelog, then how
to do sync without something like "getAllUsers" (or "syncAllUsers")?
I
know that this is quite overhead, but I really don't see how to do
"partial" sync from 3rd party provider if it doesn't support changelogs...
How do we do external store->Keycloak sync? *WE* don't. The user does!
:) Again Have a very simple ChronJob that can be scheduled. I don't
think there's any detailed SPI we can write that will solve all user needs.
Marek
>
>
> IMO it would be something like this:
>
> public interface ChronJob {
>
> void invoke(KeycloakSessionFactory factory);
> }
>
The ChronJob has full access to local keycloak storage and can begin and
end transactions as needed. We can probably provide one out of the box
for LDAP based on simple mapping metadata that can be configured, but
anything more complex has to be written by the user.
--
Bill Burke
JBoss, a division of Red Hat
http://bill.burkecentral.com
11:21 a.m.
I'm happy to push for this if you think we can still deliver 1.0.final in a reasonable
time.
Can we do this in stages? First split ModelProvider into RealmProvider and UserProvider,
update cache etc. Then start work on this in a branch? So we can roll 1.0.final without it
if it turns out to take to long to delivery?
Would be good to have a Hangout again next week to discuss the next steps!
----- Original Message -----
From: "Bill Burke" <bburke(a)redhat.com>
To: "Stian Thorgersen" <stian(a)redhat.com>, "Marek Posolda"
<mposolda(a)redhat.com>
Cc: keycloak-dev(a)lists.jboss.org
Sent: Friday, 11 July, 2014 3:12:41 PM
Subject: Re: [keycloak-dev] sync/federation requirements/ideas
On 7/11/2014 9:05 AM, Stian Thorgersen wrote:
> This sounds like exactly what we need. There are quite a lot of kinks to
> work out though.
>
My vote is we fix AuthProvider and sync now. Do another iteration.
Users *really* want this. You're right, there's a lot of kinks we need
to work out, but we don't want a large set of users we have to move off
of an existing SPI.
--
Bill Burke
JBoss, a division of Red Hat
http://bill.burkecentral.com
11:57 a.m.
What else is there after this? THere's nothing major left.
On 7/11/2014 11:21 AM, Stian Thorgersen wrote:
I'm happy to push for this if you think we can still deliver
1.0.final in a reasonable time.
Can we do this in stages? First split ModelProvider into RealmProvider and UserProvider,
update cache etc. Then start work on this in a branch? So we can roll 1.0.final without it
if it turns out to take to long to delivery?
Would be good to have a Hangout again next week to discuss the next steps!
----- Original Message -----
> From: "Bill Burke" <bburke(a)redhat.com>
> To: "Stian Thorgersen" <stian(a)redhat.com>, "Marek Posolda"
<mposolda(a)redhat.com>
> Cc: keycloak-dev(a)lists.jboss.org
> Sent: Friday, 11 July, 2014 3:12:41 PM
> Subject: Re: [keycloak-dev] sync/federation requirements/ideas
>
>
> On 7/11/2014 9:05 AM, Stian Thorgersen wrote:
>> This sounds like exactly what we need. There are quite a lot of kinks to
>> work out though.
>>
>
> My vote is we fix AuthProvider and sync now. Do another iteration.
> Users *really* want this. You're right, there's a lot of kinks we need
> to work out, but we don't want a large set of users we have to move off
> of an existing SPI.
>
> --
> Bill Burke
> JBoss, a division of Red Hat
> http://bill.burkecentral.com
>
--
Bill Burke
JBoss, a division of Red Hat
http://bill.burkecentral.com
12:02 p.m.
----- Original Message -----
From: "Bill Burke" <bburke(a)redhat.com>
To: "Stian Thorgersen" <stian(a)redhat.com>
Cc: "Marek Posolda" <mposolda(a)redhat.com>, keycloak-dev(a)lists.jboss.org
Sent: Friday, 11 July, 2014 4:57:37 PM
Subject: Re: [keycloak-dev] sync/federation requirements/ideas
What else is there after this? THere's nothing major left.
Yep, only minor things. We can't all work on this though? It'll probably also take
a while?
On 7/11/2014 11:21 AM, Stian Thorgersen wrote:
> I'm happy to push for this if you think we can still deliver 1.0.final in a
> reasonable time.
>
> Can we do this in stages? First split ModelProvider into RealmProvider and
> UserProvider, update cache etc. Then start work on this in a branch? So we
> can roll 1.0.final without it if it turns out to take to long to delivery?
>
> Would be good to have a Hangout again next week to discuss the next steps!
>
> ----- Original Message -----
>> From: "Bill Burke" <bburke(a)redhat.com>
>> To: "Stian Thorgersen" <stian(a)redhat.com>, "Marek
Posolda"
>> <mposolda(a)redhat.com>
>> Cc: keycloak-dev(a)lists.jboss.org
>> Sent: Friday, 11 July, 2014 3:12:41 PM
>> Subject: Re: [keycloak-dev] sync/federation requirements/ideas
>>
>>
>> On 7/11/2014 9:05 AM, Stian Thorgersen wrote:
>>> This sounds like exactly what we need. There are quite a lot of kinks to
>>> work out though.
>>>
>>
>> My vote is we fix AuthProvider and sync now. Do another iteration.
>> Users *really* want this. You're right, there's a lot of kinks we need
>> to work out, but we don't want a large set of users we have to move off
>> of an existing SPI.
>>
>> --
>> Bill Burke
>> JBoss, a division of Red Hat
>> http://bill.burkecentral.com
>>
--
Bill Burke
JBoss, a division of Red Hat
http://bill.burkecentral.com
12:19 p.m.
On 7/11/2014 12:02 PM, Stian Thorgersen wrote:
----- Original Message -----
> From: "Bill Burke" <bburke(a)redhat.com>
> To: "Stian Thorgersen" <stian(a)redhat.com>
> Cc: "Marek Posolda" <mposolda(a)redhat.com>,
keycloak-dev(a)lists.jboss.org
> Sent: Friday, 11 July, 2014 4:57:37 PM
> Subject: Re: [keycloak-dev] sync/federation requirements/ideas
>
> What else is there after this? THere's nothing major left.
Yep, only minor things. We can't all work on this though? It'll probably also
take a while?
There's still stuff to do. pagination, and about 20 other issues.
--
Bill Burke
JBoss, a division of Red Hat
http://bill.burkecentral.com
3:20 a.m.
On 11.7.2014 16:26, Bill Burke wrote:
On 7/10/2014 3:01 PM, Marek Posolda wrote:
>> The problem with this approach is that new users don't get imported
>> into Keycloak until the sync gets invoked. Syncing more than once per
>> day, or even once per week may not be feasible. If the external
>> storage does not have a changelog, syncing would involve iterating
>> through each and every user in external storage and syncing it with
>> the keycloak database. A full sync could take hours.
> Yes, but how to do it differently?
> (a) For methods like: UserProvider.getUsers(), UserProvider.searchFor***
> we can either:
> 1) retrieve users just from "local" store
> 2) Federate users and merge them from both "local" and "external"
store.
>
> Federation approach (2) has quite bad performance issues (especially
> with pagination+sorting), so I would prefer (1). But this really
> requires full sync from external store into KC IMO.
>
>
> (b) Then we have methods for retrieve single user:
> getUserBy*** --- here we can try to retrieve user from 'local' store and
> fallback to 'external'. In that case, user will be synced on demand to
> KC database (similar approach like AuthenticationProvider), but then we
> have the issue that just those users, who were retrieved from
> externalStore will be available in KC.
>
> I think that we can either:
> (1) Federate users from both stores
> (2) Temporarily allow that just some users from LDAP are available in KC
> database (those who were authenticated or for which getUserBy*** was
> called)
> (3) Do full sync
>
> I don't know if you see some other solution, but I would say that if we
> want to avoid (1) and (2) we really need to do full sync. So I would
> imagine that if someone configure externalStore, he will also need to
> trigger full sync to KC to have all users added to Keycloak.
Really depends on the UserProvider implementation IMO. The
UserProvider doesn't have to import the users on search/getAll
operations. The UserModels returned could be proxies. Probably harder
than I'm stating, but its quite possible its not.
> Personally I would prefer to always do sync from KC immediatelly as
> event queue doesn't always work.
+1. We'll probably eventually have to have an event queue, but I'd
like to avoid it.
>
> But fact is that if externalProvider doesn't support changelog, then how
> to do sync without something like "getAllUsers" (or
"syncAllUsers")? I
> know that this is quite overhead, but I really don't see how to do
> "partial" sync from 3rd party provider if it doesn't support
> changelogs...
>
How do we do external store->Keycloak sync? *WE* don't. The user
does! :) Again Have a very simple ChronJob that can be scheduled. I
don't think there's any detailed SPI we can write that will solve all
user needs.
> Marek
>>
>>
>> IMO it would be something like this:
>>
>> public interface ChronJob {
>>
>> void invoke(KeycloakSessionFactory factory);
>> }
>>
The ChronJob has full access to local keycloak storage and can begin
and end transactions as needed. We can probably provide one out of
the box for LDAP based on simple mapping metadata that can be
configured, but anything more complex has to be written by the user.
ok, once I finish export/import I can take a look how it looks with LDAP
and changelogs, but unfortunately I don't think that it's standardized
way to support this for all LDAP servers... Will also take a look at
pagination, which is surely supported by most of LDAP servers AFAIK. The
point for pagination is, that for example sync large LDAP with 100K
users is probably not going to happen in single Keycloak transaction,
but ChronJob would likely need to do it paginated in more transactions.
Marek
8:02 a.m.
-------- Original Message --------
Subject: Re: [keycloak-dev] sync/federation requirements/ideas
Date: Mon, 14 Jul 2014 09:20:00 +0200
From: Marek Posolda <mposolda(a)redhat.com>
To: Bill Burke <bburke(a)redhat.com>, "keycloak-dev(a)lists.jboss.org"
<keycloak-dev(a)lists.jboss.org>
On 11.7.2014 16:26, Bill Burke wrote:
On 7/10/2014 3:01 PM, Marek Posolda wrote:
>> The problem with this approach is that new users don't get imported
>> into Keycloak until the sync gets invoked. Syncing more than once per
>> day, or even once per week may not be feasible. If the external
>> storage does not have a changelog, syncing would involve iterating
>> through each and every user in external storage and syncing it with
>> the keycloak database. A full sync could take hours.
> Yes, but how to do it differently?
> (a) For methods like: UserProvider.getUsers(), UserProvider.searchFor***
> we can either:
> 1) retrieve users just from "local" store
> 2) Federate users and merge them from both "local" and "external"
store.
>
> Federation approach (2) has quite bad performance issues (especially
> with pagination+sorting), so I would prefer (1). But this really
> requires full sync from external store into KC IMO.
>
>
> (b) Then we have methods for retrieve single user:
> getUserBy*** --- here we can try to retrieve user from 'local' store and
> fallback to 'external'. In that case, user will be synced on demand to
> KC database (similar approach like AuthenticationProvider), but then we
> have the issue that just those users, who were retrieved from
> externalStore will be available in KC.
>
> I think that we can either:
> (1) Federate users from both stores
> (2) Temporarily allow that just some users from LDAP are available in KC
> database (those who were authenticated or for which getUserBy*** was
> called)
> (3) Do full sync
>
> I don't know if you see some other solution, but I would say that if we
> want to avoid (1) and (2) we really need to do full sync. So I would
> imagine that if someone configure externalStore, he will also need to
> trigger full sync to KC to have all users added to Keycloak.
Really depends on the UserProvider implementation IMO. The
UserProvider doesn't have to import the users on search/getAll
operations. The UserModels returned could be proxies. Probably harder
than I'm stating, but its quite possible its not.
> Personally I would prefer to always do sync from KC immediatelly as
> event queue doesn't always work.
+1. We'll probably eventually have to have an event queue, but I'd
like to avoid it.
>
> But fact is that if externalProvider doesn't support changelog, then how
> to do sync without something like "getAllUsers" (or
"syncAllUsers")? I
> know that this is quite overhead, but I really don't see how to do
> "partial" sync from 3rd party provider if it doesn't support
> changelogs...
>
How do we do external store->Keycloak sync? *WE* don't. The user
does! :) Again Have a very simple ChronJob that can be scheduled. I
don't think there's any detailed SPI we can write that will solve all
user needs.
> Marek
>>
>>
>> IMO it would be something like this:
>>
>> public interface ChronJob {
>>
>> void invoke(KeycloakSessionFactory factory);
>> }
>>
The ChronJob has full access to local keycloak storage and can begin
and end transactions as needed. We can probably provide one out of
the box for LDAP based on simple mapping metadata that can be
configured, but anything more complex has to be written by the user.
ok, once I finish export/import I can take a look how it looks with LDAP
and changelogs, but unfortunately I don't think that it's standardized
way to support this for all LDAP servers... Will also take a look at
pagination, which is surely supported by most of LDAP servers AFAIK. The
point for pagination is, that for example sync large LDAP with 100K
users is probably not going to happen in single Keycloak transaction,
but ChronJob would likely need to do it paginated in more transactions.
Marek
3572
days inactive
3577
days old
12 comments
3 participants
participants (3)
-
Bill Burke -
Marek Posolda -
Stian Thorgersen