[keycloak-dev] Export/import implementation

[Marek Posolda]

On 28.4.2014 14:52, Bill Burke wrote:
>
> On 4/28/2014 3:27 AM, Marek Posolda wrote:
>> I am planning to start soon on export/import. If I recall correctly, one
>> of the requirements is to export the content of whole DB content
>> (including IDs and password hashes) to JSON file, which can then be
>> later imported into other DB. This will allow to migrate between
>> environments and various DB types (For example from Mongo to MySQL and
>> viceversa).
>>
> IMO, a full export (of credentials) should require a secret given by the
> admin that will be used to encrypt the export.  The export should only
> be saved locally to disk and not available over the network.
Yes, so the exported JSON file will be always saved just to local 
filesystem with KC server and never sent over the network. With respect 
to this, I am not sure if securing the JSON file is big issue as the 
file is anyway available just to someone with physical access to the 
machine with KC server? However for better protection and ensuring that 
nobody can read it, I will encrypt the content of this file, likely with 
KDF as proposed by Bruno.
>
>> I have some question though
>>
>> 1) I assume that DB should be cleared before full import from JSON file?
>> Or do we want to update existing data without deleting the previous
>> content? I assume that this is used for migration, so it's not about
>> updating but completely delete and recreate existing DB, correct?
>>
>> 2) How to implement it. I can see two approaches
>>
>> a) Use model API to retrieve content of the DB into JSON file during
>> export. Similarly during import use model API to sync objects back from
>> JSON into model DB.
>>
>> b) Add some methods to KeycloakSession interface like:
>>
>> ObjectNode export();
>>
>> void import(ObjectNode node);
>>
>> and implement export/import separately for each model.
>>
>> Approach (b) might be better for performance as it allows to directly
>> use low-level queries specific to JPA, Mongo or other model
>> implementations to export/import stuff more effectively in batch,
>> however it will require changes in model implementations and probably
>> adding more stuff into dependencies. So I am more convinced to use (a).
>> Thoughts?
>>
> "a", IMO.  Easier to maintain.
oops, now thinking again that current model API doesn't allow to 
retrieve password hash or save password hash into DB? It allows to 
validate plain-text password or save plain-text password but there is no 
support for load or save directly the value of password hash. I wonder 
if it's ok to add some methods to RealmModel, which will allow to 
load/save password hashes?

If you don't want this, I will maybe need to use low-level stuff and 
stick with option (b) .
>
>> 3) How will be export/import triggered?
>>
>> I think that for security reasons, we want to always export into local
>> file with KC server and similarly always import from local file. Is it
>> correct? I can see approaches like:
>>
> You can already import a full json description from the admin console.
>
>> a) Use KC admin console. By default, just "admin" will be able to
>> export/import stuff . Data will be always exported/imported into JSON
>> file local to server. So it will be possible to trigger export/import
>> remotely from admin console, but just use local JSON file. The import
>> would be tricky though as import will recreate all data (including admin
>> realm and admin user) so it would need to cancel logout sessions
>> including the session of admin user who triggered import.
>>
>> b) Use some script, which will trigger JVM process for export/import
>> data. Script can be executed locally from CMD. I can imagine something
>> like this (executed from the directory with AS7/wildfly server):
>>
>> ./bin/keycloak-import -Dkeycloak.model=jpa
>> -Ddata-to-import=/tmp/file-with-full-kc-data-to-import-which-were-previously-exported.json
>>
>> Assumption is that distribution contains deployed KC inside
>> standalone/deployments/auth-server.war. Script will be able to run JVM
>> which will have access to needed KC jars on classpath and access to
>> persistence.xml . In AS7/Wildfly environment the persistence.xml bundled
>> inside auth-server.war is using datasource though, so this JVM process
>> would also need to parse datasource from
>> standalone/configuration/standalone.xml as it won't be executed in
>> managed AS7/Wildfly environment.
>>
>> c) Use something similar to approach (b) but execute export/import from
>> AS7/Wildfly CLI or admin console. The advantage is that it is triggered
>> from the managed (AS7 or Wildfly) so has access to server resources like
>> datasource referenced from persistence.xml
>>
> I don't see any reason you couldn't trigger an import from the admin
> console.  Especially if we require a secret used to encrypt the export.
>
>
Sure, I can. However as already mentioned, all the KC data (including 
admin realm and current admin user who is triggering import) are going 
to be deleted and re-imported from JSON file, so there are many things, 
which need to be addressed if we want to support it on the "online" 
system like:
- Filter, which will pause all requests until export/import is done
- All existing states in memory (TokenManager, SocialRequestManager) 
needs to be cleared after import
- All existing sessions should be logged out (including the session of 
current admin user) after import

There might be more issues related to this and user experience might not 
be good. So it seems to be much easier to support importing data either 
on KC startup as proposed by Stian or at "offline" time when KC server 
is not running at all, which can be done with script?

Exporting data from KC DB into JSON file seems to be safer, but there is 
still a need to pause the server with HTTP or JAX-RS filter, as if users 
are allowed to change data during export (like changing password or 
changing their emails etc), it might result in broken exported data.

I don't know exactly how are production systems deal with 
backup/migration of data, don't have much experience with this. But I 
assume that administrators are usually not doing it on the "online" data.

Marek