5:39 a.m.
I'd like to have a hangout this week to talk about cache, clustering and splitting of
the model. I'd like to do this a.s.a.p. so we can decide on what we're going to do
for 1.0.final.
Last week I experimented with splitting the model into 3 parts: config, users and
sessions. I've got something close to working at the moment. The basic idea is to
create a new hybrid model provider that delegates to 3 different providers:
* Config SPI - realms, apps, clients, roles and scope mappings
* Could quite easily do a json file implementation. However, what about clustering?
Could we just reload the whole realm whenever it's changed?
* Not considering clustering, is there even any reason to store config in a database?
Dropping support for DBs and Mongo for config would make things significantly simpler.
* Can we load everything into mem on startup? How would that affect clustering?
* Should we add a revision to realms to make it easy to track consistency across servers
in a cluster?
* In the long run we could move code from managers into KeycloakSession and
HybridModelProvider (and also extract more shared code from the stores themselves)
* Does it make sense to have a read and read/write mode for config. For example admin
uses read/write config, while logins and such use the read only config
* We could add a batch mode to the admin console. An admin can perform a number of
changes that are kept in a draft version of the config on the server. Once the admin has
done all the changes, he can then choose to review the changes through a page in the admin
console and click push if he's happy with it. This could be taken further and have
some users that are allowed to perform changes, but not push the changes.
* Users SPI
* Stores users, credentials and role mappings
* I expect this is the only one users of Keycloak could want to implement themselves
* What implementations do we provide? DBs, Mongo, LDAP?, Files?. Does it make sense to
use PicketLink here (could provide jdbs, files, ldap, etc..)
* Sessions SPI
* Stores sessions (and probably login failures as well)
* In-mem implementation makes sense here. However, what about clustering?
* Do we still want JPA and Mongo implementations?
11:45 a.m.
On 6/30/2014 6:39 AM, Stian Thorgersen wrote:
I'd like to have a hangout this week to talk about cache,
clustering and splitting of the model. I'd like to do this a.s.a.p. so we can decide
on what we're going to do for 1.0.final.
Last week I experimented with splitting the model into 3 parts: config, users and
sessions. I've got something close to working at the moment. The basic idea is to
create a new hybrid model provider that delegates to 3 different providers:
realm, roles, scopes, apps, clients, etc. are *NOT* config. Please stop
calling it config. :)
* Config SPI - realms, apps, clients, roles and scope mappings
* Could quite easily do a json file implementation. However, what about clustering?
Could we just reload the whole realm whenever it's changed?
This ties into caching as well.
* Not considering clustering, is there even any reason to store
config in a database? Dropping support for DBs and Mongo for config would make things
significantly simpler.
I don't agree having a flat file would make things simpler.
* You'd have to write a flat-file database that could handle session
rollbacks.
* You'd have to require NFS or some other distributed file system for
clustering.
* You've just added another storage mechanism the admin has to worry about.
* Can we load everything into mem on startup? How would that
affect clustering?
That was already the plan with the caching layer I wrote. As it was,
the first access to a realm caches all non-user realm metadata.
* Should we add a revision to realms to make it easy to track
consistency across servers in a cluster?
* In the long run we could move code from managers into KeycloakSession and
HybridModelProvider (and also extract more shared code from the stores themselves)
95% of the model code is boilerplate. THere is very little code that is
duplicate.
* Does it make sense to have a read and read/write mode for
config. For example admin uses read/write config, while logins and such use the read only
config
* We could add a batch mode to the admin console. An admin can perform a number of
changes that are kept in a draft version of the config on the server. Once the admin has
done all the changes, he can then choose to review the changes through a page in the admin
console and click push if he's happy with it. This could be taken further and have
some users that are allowed to perform changes, but not push the changes.
IMO, those are very low priority compared to the other things we have to
accomplish over the next year.
* Users SPI
* Stores users, credentials and role mappings
* I expect this is the only one users of Keycloak could want to implement themselves
* What implementations do we provide? DBs, Mongo, LDAP?, Files?. Does it make sense to
use PicketLink here (could provide jdbs, files, ldap, etc..)
With this separation, do we still need the AuthProvider SPI?
* Sessions SPI
* Stores sessions (and probably login failures as well)
* In-mem implementation makes sense here. However, what about clustering?
* Do we still want JPA and Mongo implementations?
Without a JPA/Mongo implementation for User Sessions, you have to
introduce another moving part in a cluster environment: Infinispan, etc.
--
Bill Burke
JBoss, a division of Red Hat
http://bill.burkecentral.com
3:16 a.m.
On 30.6.2014 18:45, Bill Burke wrote:
On 6/30/2014 6:39 AM, Stian Thorgersen wrote:
> I'd like to have a hangout this week to talk about cache, clustering and
splitting of the model. I'd like to do this a.s.a.p. so we can decide on what
we're going to do for 1.0.final.
>
> Last week I experimented with splitting the model into 3 parts: config, users and
sessions. I've got something close to working at the moment. The basic idea is to
create a new hybrid model provider that delegates to 3 different providers:
>
realm, roles, scopes, apps, clients, etc. are *NOT* config. Please stop
calling it config. :)
> * Config SPI - realms, apps, clients, roles and scope mappings
> * Could quite easily do a json file implementation. However, what about
clustering? Could we just reload the whole realm whenever it's changed?
This ties into caching as well.
> * Not considering clustering, is there even any reason to store config in a
database? Dropping support for DBs and Mongo for config would make things significantly
simpler.
I don't agree having a flat file would make things simpler.
* You'd have to write a flat-file database that could handle session
rollbacks.
* You'd have to require NFS or some other distributed file system for
clustering.
* You've just added another storage mechanism the admin has to worry about.
I
wonder if we can reconsider again to use Infinispan:-)
It has support for both clustering and transactions. Should address the
issues you mentioned here. They have the SPIs to sync with persistence
layers
http://infinispan.org/docs/6.0.x/user_guide/user_guide.html#_persistence
and eventually we can provide our own SPI implementations.
The secure communication among cluster nodes is possible to achieve at
JGroups level by add support for ENCRYPT (encrypt cluster communication)
and AUTH (authenticate cluster nodes before join cluster). Some details
here http://www.jgroups.org/manual/html/protlist.html#Security . I
looked at ENCRYPT and unfortunately it seems that it's not possible to
secure the communication just with Keycloak itself, as it needs
symmetric SecretKey for encode/decode communication between cluster
nodes, when Keycloak provides private/public. SecretKey can be either
read from JCEKS keystore file or automatically generated by first
cluster node and then shared with other nodes with usage of asymmetric
private/public. It seems that with some tweaks we can use Keycloak keys
here.
I can investigate it more, if you see Infinispan as an option.
> * Can we load everything into mem on startup? How would that affect clustering?
That was already the plan with the caching layer I wrote. As it was,
the first access to a realm caches all non-user realm metadata.
> * Should we add a revision to realms to make it easy to track consistency across
servers in a cluster?
> * In the long run we could move code from managers into KeycloakSession and
HybridModelProvider (and also extract more shared code from the stores themselves)
95% of the model code is boilerplate. THere is very little code that is
duplicate.
> * Does it make sense to have a read and read/write mode for config. For example
admin uses read/write config, while logins and such use the read only config
> * We could add a batch mode to the admin console. An admin can perform a number
of changes that are kept in a draft version of the config on the server. Once the admin
has done all the changes, he can then choose to review the changes through a page in the
admin console and click push if he's happy with it. This could be taken further and
have some users that are allowed to perform changes, but not push the changes.
>
IMO, those are very low priority compared to the other things we have to
accomplish over the next year.
> * Users SPI
> * Stores users, credentials and role mappings
> * I expect this is the only one users of Keycloak could want to implement
themselves
> * What implementations do we provide? DBs, Mongo, LDAP?, Files?. Does it make
sense to use PicketLink here (could provide jdbs, files, ldap, etc..)
>
With this separation, do we still need the AuthProvider SPI?
I would say yes as
implementing AuthProvider is still much easier then
implementing whole users storage. And we already have some community
members who implemented their own AuthenticationProviders.
And for example LDAP is good just for authentication, but much more
tricky for save all other things related to UserModel (for example
attributes, requiredActions etc) due to static nature of LDAP schema.
Marek
> * Sessions SPI
> * Stores sessions (and probably login failures as well)
> * In-mem implementation makes sense here. However, what about clustering?
> * Do we still want JPA and Mongo implementations?
>
Without a JPA/Mongo implementation for User Sessions, you have to
introduce another moving part in a cluster environment: Infinispan, etc.
3:47 a.m.
----- Original Message -----
From: "Bill Burke" <bburke(a)redhat.com>
To: keycloak-dev(a)lists.jboss.org
Sent: Monday, 30 June, 2014 5:45:20 PM
Subject: Re: [keycloak-dev] Hangout this week to talk about cache and model splitting
On 6/30/2014 6:39 AM, Stian Thorgersen wrote:
> I'd like to have a hangout this week to talk about cache, clustering and
> splitting of the model. I'd like to do this a.s.a.p. so we can decide on
> what we're going to do for 1.0.final.
>
> Last week I experimented with splitting the model into 3 parts: config,
> users and sessions. I've got something close to working at the moment. The
> basic idea is to create a new hybrid model provider that delegates to 3
> different providers:
>
realm, roles, scopes, apps, clients, etc. are *NOT* config. Please stop
calling it config. :)
Realms SPI?
> * Config SPI - realms, apps, clients, roles and scope mappings
> * Could quite easily do a json file implementation. However, what about
> clustering? Could we just reload the whole realm whenever it's changed?
This ties into caching as well.
> * Not considering clustering, is there even any reason to store config
> in a database? Dropping support for DBs and Mongo for config would make
> things significantly simpler.
I don't agree having a flat file would make things simpler.
* You'd have to write a flat-file database that could handle session
rollbacks.
* You'd have to require NFS or some other distributed file system for
clustering.
* You've just added another storage mechanism the admin has to worry about.
I was thinking something a lot simpler. A single keycloak-realms.json file, and whenever
anything is changed the whole file is written again. Thinking about it some more it only
makes sense in smaller deployments, where there's only one or two realms, a few apps
and a few roles. So we'd still want DBs.
We could add this quite easily as we already need to have the capability of
importing/exporting the whole db to a json file. It could just have a limitation that it
doesn't do clustering, so would only be aimed for test/dev and small deployments.
> * Can we load everything into mem on startup? How would that affect
> clustering?
That was already the plan with the caching layer I wrote. As it was,
the first access to a realm caches all non-user realm metadata.
Basically what I was wondering here was if we could eagerly fetch everything for a realm
and just load everything in mem in one go. Also, does it make sense to load it at startup
rather than on first visit?
> * Should we add a revision to realms to make it easy to track
> consistency across servers in a cluster?
> * In the long run we could move code from managers into KeycloakSession
> and HybridModelProvider (and also extract more shared code from the
> stores themselves)
95% of the model code is boilerplate. THere is very little code that is
duplicate.
There's still 5% ;)
> * Does it make sense to have a read and read/write mode for config. For
> example admin uses read/write config, while logins and such use the
> read only config
> * We could add a batch mode to the admin console. An admin can perform a
> number of changes that are kept in a draft version of the config on the
> server. Once the admin has done all the changes, he can then choose to
> review the changes through a page in the admin console and click push
> if he's happy with it. This could be taken further and have some users
> that are allowed to perform changes, but not push the changes.
>
IMO, those are very low priority compared to the other things we have to
accomplish over the next year.
Totally agree, those where just some ideas that popped into my head.
> * Users SPI
> * Stores users, credentials and role mappings
> * I expect this is the only one users of Keycloak could want to
> implement themselves
> * What implementations do we provide? DBs, Mongo, LDAP?, Files?. Does it
> make sense to use PicketLink here (could provide jdbs, files, ldap,
> etc..)
>
With this separation, do we still need the AuthProvider SPI?
AuthProvider is trying to achieve something different than storing users, and the
combination of the two is quite flexible. Someone could still achieve what they can
achieve with AuthProvider by creating a UserProvider impl that delegates to another
UserProvider impl for storage, so maybe we could respin the AuthProvider SPI into a
UserProvider that can use another UserProvider for storing users, and one or more
UserProviders for authentication. That way we'd still have the capability, but with it
less hard-coded into the core code.
+--------------------+
Store | |
+------------> JpaUserProvider |
| | |
| +--------------------+
+------------------+
| |
| AuthUserProvider | +--------------------+
| | | |
+------------------+ | LDAPUserProvider |
+------------> |
Authenticate +--------------------+
Same again with Sync SPI, that could be done as a delegating UserProvider store.
+--------------------+
Store | |
+------------> JpaUserProvider |
| | |
| +--------------------+
+------------------+
| |
| SyncUserProvider | +--------------------+
| | | |
+------------------+ | LDAPUserProvider |
+------------> |
Sync +--------------------+
> * Sessions SPI
> * Stores sessions (and probably login failures as well)
> * In-mem implementation makes sense here. However, what about
> clustering?
> * Do we still want JPA and Mongo implementations?
>
Without a JPA/Mongo implementation for User Sessions, you have to
introduce another moving part in a cluster environment: Infinispan, etc.
Yep, for 1.0.final we should do JPA/Mongo. Then in the future consider Infinispan or
somethin'.
I've added a very simple in-mem session store though that can be used in non-clustered
mode and test/dev.
--
Bill Burke
JBoss, a division of Red Hat
http://bill.burkecentral.com
_______________________________________________
keycloak-dev mailing list
keycloak-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-dev
3831
days inactive
3832
days old
3 comments
3 participants
participants (3)
-
Bill Burke -
Marek Posolda -
Stian Thorgersen