[keycloak-dev] why doesnt import/expot use reps?

[Stian Thorgersen]

A few things I'd like to see from this:

* One JSON representation for everything (this includes exports, admin console and examples)
* Use a serializing json provider when dumping the database
* Support dumping all realms or a specific realm to a single json file, also it would be nice to export users only
* Support importing the exported realms through the admin console
* Support splitting users into a separate file - this can also support pagination to have a specified amount of users per file
* JSON exports needs to have a version
* Database needs to have a version - so we can easily detect if the db is out of date and kill the server if it is
* Transformation pipeline for JSON representation of version 1 to version 2 to version 3, etc
* I think encrypting with bouncycastle would be better than winzipaes

This is an important feature, and should support the following use cases:

* Export the database to migrate to another Keycloak version
* Export the database for backup, or to extract the data from Keycloak (to prevent vendor lock-in)
* Import from other sources - for example someone that has an existing user-database could export their users to our format (we already have someone that's asked for this). I'd imagine this would be done by supporting import a users only export into an existing realm
* Export a specific realm for testing/demo purposes - this lets you create/configure a realm through the console, then export for future use

Lower priority, but things I'd also like to see:

* Automatically migrating database on startup / done by checking version in db, export to json, clear db, import from json - one caveat here is if we somehow manage to delete everything :/
* What to do if realm being imported already exists (skip, merge, overwrite, users-only, etc..)
* Import through the console / with single JSON representation we should already support importing single-file, but we should also add support for separate users file(s) and encrypted file

----- Original Message -----
> From: "Marek Posolda" <mposolda at redhat.com>
> To: "Bill Burke" <bburke at redhat.com>, keycloak-dev at lists.jboss.org
> Sent: Wednesday, 28 May, 2014 9:18:50 AM
> Subject: Re: [keycloak-dev] why doesnt import/expot use reps?
> 
> I assume that main purpose of export/import is especially migration of
> full DB from one environment to another, so it's a bit different than
> just importing JSON file like testrealm.json with few data related to
> one realm IMO.
> 
> My main worry is especially about performance. For example if you have
> realm with million users and want to migrate it, the resulting
> realm.json file will be very big and IMO it would be impossible to
> import it with current approach used in RealmManager.importRealm, which
> is doing whole import in 1 transaction and needs whole
> RealmRepresentation to be read into memory with all the data and all
> million users.
> 
> So that's why I used a bit different approach, which is doing import in
> few steps and should scale well even with very big amount of data.
> 
> Also some data in representations can't be used as they are because it's
> impossible to retrieve them from DB. For example
> CredentialRepresentation assumes password in plain-text, but DB doesn't
> contain password in plain-text. To workaround, I will need
> CredentialRepresentation to support both plain-text password and also
> hash+salt. Similarly for privateKey (if we ever have an SPI for secure
> store of private key). Is it fine to change CredentialRepresentation
> (and possibly other places) this way? Also I will need to add support
> for "id" into representations as export/import is exporting everything
> including ID of objects, but that's not a big issue though...
> 
> Also the stuff inside model/api is not used just by export/import, but
> also by Mongo model. Mongo is storing it's data in JSON like format and
> I am reusing same format for export/import. So we not to maintain more
> things than before. If you want to add new configuration option with
> getter+setter into Realm, you still have "just" 7 places to update :) (I
> count RealmModel, 2xRealmEntity, 2xRealmAdapter, RealmRepresentation and
> ModelToRepresentation)
> 
> I have already JIRA opened for investigation of using same format -
> https://issues.jboss.org/browse/KEYCLOAK-487 . I can also investigate
> the possibility to read data in stream instead of everything into memory
> like RealmRepresentation is doing.
> 
> Marek
> 
> On 22.5.2014 16:33, Bill Burke wrote:
> > We now have two different models for dealing with imports and two
> > different code paths too.  Why does import/export have its own json
> > model under model/api/...entities?  Why weren't the JSON representations
> > in keycloak-core/.../representations used?
> >
> > We already have code that converts between
> > keycloak-core/...representations and Models that is updated and
> > maintained.  We now have double the work to keep the export/import stuff
> > in sync too!
> >
> >
> 
> _______________________________________________
> keycloak-dev mailing list
> keycloak-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>