[keycloak-dev] why doesnt import/expot use reps?

[Marek Posolda]

I assume that main purpose of export/import is especially migration of 
full DB from one environment to another, so it's a bit different than 
just importing JSON file like testrealm.json with few data related to 
one realm IMO.

My main worry is especially about performance. For example if you have 
realm with million users and want to migrate it, the resulting 
realm.json file will be very big and IMO it would be impossible to 
import it with current approach used in RealmManager.importRealm, which 
is doing whole import in 1 transaction and needs whole 
RealmRepresentation to be read into memory with all the data and all 
million users.

So that's why I used a bit different approach, which is doing import in 
few steps and should scale well even with very big amount of data.

Also some data in representations can't be used as they are because it's 
impossible to retrieve them from DB. For example 
CredentialRepresentation assumes password in plain-text, but DB doesn't 
contain password in plain-text. To workaround, I will need 
CredentialRepresentation to support both plain-text password and also 
hash+salt. Similarly for privateKey (if we ever have an SPI for secure 
store of private key). Is it fine to change CredentialRepresentation 
(and possibly other places) this way? Also I will need to add support 
for "id" into representations as export/import is exporting everything 
including ID of objects, but that's not a big issue though...

Also the stuff inside model/api is not used just by export/import, but 
also by Mongo model. Mongo is storing it's data in JSON like format and 
I am reusing same format for export/import. So we not to maintain more 
things than before. If you want to add new configuration option with 
getter+setter into Realm, you still have "just" 7 places to update :) (I 
count RealmModel, 2xRealmEntity, 2xRealmAdapter, RealmRepresentation and 
ModelToRepresentation)

I have already JIRA opened for investigation of using same format - 
https://issues.jboss.org/browse/KEYCLOAK-487 . I can also investigate 
the possibility to read data in stream instead of everything into memory 
like RealmRepresentation is doing.

Marek

On 22.5.2014 16:33, Bill Burke wrote:
> We now have two different models for dealing with imports and two
> different code paths too.  Why does import/export have its own json
> model under model/api/...entities?  Why weren't the JSON representations
> in keycloak-core/.../representations used?
>
> We already have code that converts between
> keycloak-core/...representations and Models that is updated and
> maintained.  We now have double the work to keep the export/import stuff
> in sync too!
>
>