need help on JPA schema changes
by Bill Burke
I refactored some things for both performance and for bulk delete of
users.
* use long @Id for all non-exposed entities, i.e. UserRoleMappingEntity,
etc. This is more efficient than strings
* Ditched all @ElementCollections from UserEntity. Bulk delete does not
cascade and there is no way to bulk delete @ElementCollections from
JPAQL! This caused me to add UserAttributeEntity and
UserRequiredActionEntity
* Added @ManyToOne UserEntity relationship to AuthenticationLinkEntity
so that it can be bulk deleted.
Questions:
I'm not sure about moving things like social links to UserAttributes.
SocialLinkEntity, UserRoleMappingEntity, UserRequiredActionEntity are
all in separate tables. Except for AuthenticationLinkEntity, which I
hope to deal away with on the AuthenticationProvider refactor, I think
UserEntity is fine the way it is.
Are we *ABSOLUTELY* sure we want to remove secondary key constraints
from UserRoleMappingEntity and UserEntity for role and realm
respectively? These constraints caught a lot of problems for me that I
wouldn't have found otherwise. I just don't think we'd ever store users
and realms in different stores.
--
Bill Burke
JBoss, a division of Red Hat
http://bill.burkecentral.com
10 years
Export/import into representations
by Marek Posolda
I've sent PR https://github.com/keycloak/keycloak/pull/523 for
export/import with same representation as admin realm.
- It's still possible to export into encrypted ZIP file (default
possibility), into directory or all realms into single JSON file.
- With export to directory, data of each realm are exported into
separate file and users are also exported into separate file. It's
configurable how many users will be in single file (By default it's 5000
users per file) and it's using separate transaction per each file (set
of users). So for example if you have realm with 20.000 users the realm
will be exported into 5 separate files (1 file like "demo-realm.json"
for realm data and 4 files like "demo-users-0.json", "demo-users-1.json"
etc for users).
- Export to encrypted ZIP is quite similar like export to directory
(realms and users in separate files inside ZIP), but all data are
canonicalized. This is default possibility and is intended for
production use
- Export into single JSON file allows to export all realms including
users into single JSON file (array of RealmRepresentations). This is
intended to be used mainly for development
- I've moved some code from "services" module as from there it's not
available to export/import. I've moved class ModelToRepresentation to
model/api and I've also extracted some code from managers (RealmManager,
ApplicationManager, OAuthClientManager, RolesManager) to
RepresentationToModel class.
- I've moved tests to testsuite/integration . Right now there is
ExportImportTest, which tests all 3 providers (zip, dir, singleFile).
- Question: Is it plan to move some manager classes like RealmManager,
ApplicationManager, OAuthClientManager and RolesManager to model/api as
well? These don't have dependencies on other stuff in "services" and it
will be useful for stuff like export/import to have them available. For
example export/import may need the code for setup masterAdmin
application (as if I import realm, I've need to check if particular
master application like "demo-realm" exists in master realm and create
it if doesn't, so I've needed to move some related code from
RealmManager for setup this too).
- Export/import is still checked at server startup with system properties.
Question: How big priority it is to support export from admin console?
TBH it shouldn't be hard to add it but it seems to me that this option
is useful more for development than production as during export from
admin console could be realm data edited in the meantime (For example if
some other user registers himself in the meantime when admin triggered
export). The solution might be JAX-RS or Http Filter, which will refuse
request to realm when export is in progress. Do we want to go that way?
Maybe just possibility to export current realm to single JSON file is
sufficient in admin console?
Let me know if you have some other ideas/proposals
Thanks,
Marek
10 years
ExportImportTest disabled, breaks build
by Bill Burke
When running the ExportImportTest with the build it causes other tests
to fail. I spent an hour or two trying to figure it out, I couldn't.
This is off of master *BEFORE* I did my big commit. Its not a caching
issue as I ran tests without the cache. It seems that users are not
being deleted. There's some other issues too even if I change the
usernames used in the ExportImportTest.
--
Bill Burke
JBoss, a division of Red Hat
http://bill.burkecentral.com
10 years
Remaining work and who does what for beta-4
by Stian Thorgersen
Now that the craziness of splitting the model is behind us I think we should figure out exactly what's left to do for beta-4, who's doing what and an estimated release date.
Here's the list of things I can come up with:
1. JPA UserRoleMappingEntity and UserEntity still have @ManyToOne to RoleEntity and RealmEntity respectively.
2. JPA and Mongo RealmEntity and UserEntity should be refactored to be attribute based as in the Hybrid model.
3. Credential validation, update needs to move from RealmModel to UserProvider.
4. Sync support in UserProvider
5. LDAP implementation for 4
6. Access code work (reducing code size, email link size, single-use code, etc.)
7. Pagination support for users
8. Improve delete of users
My vote is yes to all above!
Also, a couple of things we should consider:
9. Add generic configuration for providers, including the ability to configure any providers through the admin console
10. Have KeycloakSession bound to a specific realm
My suggestion is:
* Bill - works on sync support in UserSession
* Marek - works on LDAP implementation for sync
* Stian - I can either get started on 9 and 10, or scrape of the rest. Depends on whether or not we decide to do them or not.
If we're going to have a chance of releasing 1.0.final in August, I think we should aim at having beta-4 (with an actual code freeze!) towards the end of July. We'll need at least 1 month to do tests, security audit, update docs, etc.
Also, I'm away 13 days and Marek is away 8 days in August!
10 years
Admin console broken (fix coming!)
by Stian Thorgersen
Some idiot (aka me) has broken the admin console. Some changes I did to keycloak.js are causing a redirect loop. Fix coming asap!!
10 years
UserProvider merged
by Bill Burke
* ModelProvider renamed to RealmProvider
* All user related methods except for credential validation removed from
RealmModel
* KeycloakSession.users() gets you a UserProvider
* All classes now use UserProvider to work with users.
* KeycloakSession.model() renamed to realms()
* There are two cache providers now. One for realms, other for users.
TODO:
* JPA UserRoleMappingEntity and UserEntity still have @ManyToOne to
RoleEntity and RealmEntity respectively. Do we want to keep these
constraints?
* JPA and Mongo RealmEntity and UserEntity should be refactored to be
attribute based as in the Hybrid model. As Stian suggested, this will
allow us flexibility in the future.
* Credential validation, update needs to move from RealmModel to
UserProvider. I'll do this tomorrow.
--
Bill Burke
JBoss, a division of Red Hat
http://bill.burkecentral.com
10 years
Issue with CacheModel
by Marek Posolda
Hi,
During export/import I may want to remove some data (like realm) and
overwrite it with imported one. But actually I have some issues during
testing and looks like it's caused by the fact that not all the data are
properly invalidated during removal of entities.
For example when removing realm, there is just realm itself marked for
invalidation
https://github.com/keycloak/keycloak/blob/master/model/invalidation-cache...
. Actually with JPA model, the call to getDelegate().removeRealm(id)
also seems to invalidate application and application roles (due to the
call to getApplicationById() which uses CacheModel) but realm roles,
oauth clients and users are not invalidated. For Mongo model, there are
no data invalidated at all.
Question is if CacheModelProvider should rely on ModelProvider to
invalidate cached data during getDelegate().removeRealm ? Shouldn't it
be the responsibility of CacheModelProvider itself to find all cached
data related to particular realm (in this case all applications, oauth
clients, realm roles, app roles, users) and invalidate them?
Marek
10 years
Fixes for MySQL 5.1
by Stian Thorgersen
I've added some fixes for MySQL 5.1 to https://github.com/stianst/keycloak/commit/723a50197ce4489c30d1ca3604c0eb...
Bill - can I commit this now? Or do you want me to wait until after you've finished (I don't mind merging it once it is)?
Basically it doesn't support keys longer than 1000 bytes, which turns out to be varchar 200, not 255 as default. So I've set all id's that are uuid's to 36, and the others to 200 (user username only, and only used as a foreign key in login failures).
10 years