9:33 a.m.
We now have two different models for dealing with imports and two
different code paths too. Why does import/export have its own json
model under model/api/...entities? Why weren't the JSON representations
in keycloak-core/.../representations used?
We already have code that converts between
keycloak-core/...representations and Models that is updated and
maintained. We now have double the work to keep the export/import stuff
in sync too!
--
Bill Burke
JBoss, a division of Red Hat
http://bill.burkecentral.com
10:26 a.m.
I agree it would be better to use one set of JSON representations - that would also mean
that we can import the exported json through the admin console as well
----- Original Message -----
From: "Bill Burke" <bburke(a)redhat.com>
To: keycloak-dev(a)lists.jboss.org
Sent: Thursday, 22 May, 2014 3:33:59 PM
Subject: [keycloak-dev] why doesnt import/expot use reps?
We now have two different models for dealing with imports and two
different code paths too. Why does import/export have its own json
model under model/api/...entities? Why weren't the JSON representations
in keycloak-core/.../representations used?
We already have code that converts between
keycloak-core/...representations and Models that is updated and
maintained. We now have double the work to keep the export/import stuff
in sync too!
--
Bill Burke
JBoss, a division of Red Hat
http://bill.burkecentral.com
_______________________________________________
keycloak-dev mailing list
keycloak-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-dev
3:18 a.m.
I assume that main purpose of export/import is especially migration of
full DB from one environment to another, so it's a bit different than
just importing JSON file like testrealm.json with few data related to
one realm IMO.
My main worry is especially about performance. For example if you have
realm with million users and want to migrate it, the resulting
realm.json file will be very big and IMO it would be impossible to
import it with current approach used in RealmManager.importRealm, which
is doing whole import in 1 transaction and needs whole
RealmRepresentation to be read into memory with all the data and all
million users.
So that's why I used a bit different approach, which is doing import in
few steps and should scale well even with very big amount of data.
Also some data in representations can't be used as they are because it's
impossible to retrieve them from DB. For example
CredentialRepresentation assumes password in plain-text, but DB doesn't
contain password in plain-text. To workaround, I will need
CredentialRepresentation to support both plain-text password and also
hash+salt. Similarly for privateKey (if we ever have an SPI for secure
store of private key). Is it fine to change CredentialRepresentation
(and possibly other places) this way? Also I will need to add support
for "id" into representations as export/import is exporting everything
including ID of objects, but that's not a big issue though...
Also the stuff inside model/api is not used just by export/import, but
also by Mongo model. Mongo is storing it's data in JSON like format and
I am reusing same format for export/import. So we not to maintain more
things than before. If you want to add new configuration option with
getter+setter into Realm, you still have "just" 7 places to update :) (I
count RealmModel, 2xRealmEntity, 2xRealmAdapter, RealmRepresentation and
ModelToRepresentation)
I have already JIRA opened for investigation of using same format -
https://issues.jboss.org/browse/KEYCLOAK-487 . I can also investigate
the possibility to read data in stream instead of everything into memory
like RealmRepresentation is doing.
Marek
On 22.5.2014 16:33, Bill Burke wrote:
We now have two different models for dealing with imports and two
different code paths too. Why does import/export have its own json
model under model/api/...entities? Why weren't the JSON representations
in keycloak-core/.../representations used?
We already have code that converts between
keycloak-core/...representations and Models that is updated and
maintained. We now have double the work to keep the export/import stuff
in sync too!
3:15 a.m.
A few things I'd like to see from this:
* One JSON representation for everything (this includes exports, admin console and
examples)
* Use a serializing json provider when dumping the database
* Support dumping all realms or a specific realm to a single json file, also it would be
nice to export users only
* Support importing the exported realms through the admin console
* Support splitting users into a separate file - this can also support pagination to have
a specified amount of users per file
* JSON exports needs to have a version
* Database needs to have a version - so we can easily detect if the db is out of date and
kill the server if it is
* Transformation pipeline for JSON representation of version 1 to version 2 to version 3,
etc
* I think encrypting with bouncycastle would be better than winzipaes
This is an important feature, and should support the following use cases:
* Export the database to migrate to another Keycloak version
* Export the database for backup, or to extract the data from Keycloak (to prevent vendor
lock-in)
* Import from other sources - for example someone that has an existing user-database could
export their users to our format (we already have someone that's asked for this).
I'd imagine this would be done by supporting import a users only export into an
existing realm
* Export a specific realm for testing/demo purposes - this lets you create/configure a
realm through the console, then export for future use
Lower priority, but things I'd also like to see:
* Automatically migrating database on startup / done by checking version in db, export to
json, clear db, import from json - one caveat here is if we somehow manage to delete
everything :/
* What to do if realm being imported already exists (skip, merge, overwrite, users-only,
etc..)
* Import through the console / with single JSON representation we should already support
importing single-file, but we should also add support for separate users file(s) and
encrypted file
----- Original Message -----
From: "Marek Posolda" <mposolda(a)redhat.com>
To: "Bill Burke" <bburke(a)redhat.com>, keycloak-dev(a)lists.jboss.org
Sent: Wednesday, 28 May, 2014 9:18:50 AM
Subject: Re: [keycloak-dev] why doesnt import/expot use reps?
I assume that main purpose of export/import is especially migration of
full DB from one environment to another, so it's a bit different than
just importing JSON file like testrealm.json with few data related to
one realm IMO.
My main worry is especially about performance. For example if you have
realm with million users and want to migrate it, the resulting
realm.json file will be very big and IMO it would be impossible to
import it with current approach used in RealmManager.importRealm, which
is doing whole import in 1 transaction and needs whole
RealmRepresentation to be read into memory with all the data and all
million users.
So that's why I used a bit different approach, which is doing import in
few steps and should scale well even with very big amount of data.
Also some data in representations can't be used as they are because it's
impossible to retrieve them from DB. For example
CredentialRepresentation assumes password in plain-text, but DB doesn't
contain password in plain-text. To workaround, I will need
CredentialRepresentation to support both plain-text password and also
hash+salt. Similarly for privateKey (if we ever have an SPI for secure
store of private key). Is it fine to change CredentialRepresentation
(and possibly other places) this way? Also I will need to add support
for "id" into representations as export/import is exporting everything
including ID of objects, but that's not a big issue though...
Also the stuff inside model/api is not used just by export/import, but
also by Mongo model. Mongo is storing it's data in JSON like format and
I am reusing same format for export/import. So we not to maintain more
things than before. If you want to add new configuration option with
getter+setter into Realm, you still have "just" 7 places to update :) (I
count RealmModel, 2xRealmEntity, 2xRealmAdapter, RealmRepresentation and
ModelToRepresentation)
I have already JIRA opened for investigation of using same format -
https://issues.jboss.org/browse/KEYCLOAK-487 . I can also investigate
the possibility to read data in stream instead of everything into memory
like RealmRepresentation is doing.
Marek
On 22.5.2014 16:33, Bill Burke wrote:
> We now have two different models for dealing with imports and two
> different code paths too. Why does import/export have its own json
> model under model/api/...entities? Why weren't the JSON representations
> in keycloak-core/.../representations used?
>
> We already have code that converts between
> keycloak-core/...representations and Models that is updated and
> maintained. We now have double the work to keep the export/import stuff
> in sync too!
>
>
_______________________________________________
keycloak-dev mailing list
keycloak-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-dev
3858
days inactive
3866
days old
3 comments
3 participants
participants (3)
-
Bill Burke -
Marek Posolda -
Stian Thorgersen