[keycloak-dev] Export/import implementation

[Stian Thorgersen]

----- Original Message -----
> From: "Marek Posolda" <mposolda at redhat.com>
> To: keycloak-dev at lists.jboss.org
> Sent: Monday, 28 April, 2014 8:27:45 AM
> Subject: [keycloak-dev] Export/import implementation
> 
> I am planning to start soon on export/import. If I recall correctly, one
> of the requirements is to export the content of whole DB content
> (including IDs and password hashes) to JSON file, which can then be
> later imported into other DB. This will allow to migrate between
> environments and various DB types (For example from Mongo to MySQL and
> viceversa).
> 
> I have some question though
> 
> 1) I assume that DB should be cleared before full import from JSON file?
> Or do we want to update existing data without deleting the previous
> content? I assume that this is used for migration, so it's not about
> updating but completely delete and recreate existing DB, correct?

Yes, I think first pass should be full DB export and import. If there is time, or at some point in the future, we can consider being able to import/export individual realms, and even some form of a merge on import

> 
> 2) How to implement it. I can see two approaches
> 
> a) Use model API to retrieve content of the DB into JSON file during
> export. Similarly during import use model API to sync objects back from
> JSON into model DB.
> 
> b) Add some methods to KeycloakSession interface like:
> 
> ObjectNode export();
> 
> void import(ObjectNode node);
> 
> and implement export/import separately for each model.
> 
> Approach (b) might be better for performance as it allows to directly
> use low-level queries specific to JPA, Mongo or other model
> implementations to export/import stuff more effectively in batch,
> however it will require changes in model implementations and probably
> adding more stuff into dependencies. So I am more convinced to use (a).
> Thoughts?

I'd say let's go with option a. If performance does become a real headache in the future we can reconsider

> 
> 
> 3) How will be export/import triggered?
> 
> I think that for security reasons, we want to always export into local
> file with KC server and similarly always import from local file. Is it
> correct? I can see approaches like:
> 
> a) Use KC admin console. By default, just "admin" will be able to
> export/import stuff . Data will be always exported/imported into JSON
> file local to server. So it will be possible to trigger export/import
> remotely from admin console, but just use local JSON file. The import
> would be tricky though as import will recreate all data (including admin
> realm and admin user) so it would need to cancel logout sessions
> including the session of admin user who triggered import.
> 
> b) Use some script, which will trigger JVM process for export/import
> data. Script can be executed locally from CMD. I can imagine something
> like this (executed from the directory with AS7/wildfly server):
> 
> ./bin/keycloak-import -Dkeycloak.model=jpa
> -Ddata-to-import=/tmp/file-with-full-kc-data-to-import-which-were-previously-exported.json
> 
> Assumption is that distribution contains deployed KC inside
> standalone/deployments/auth-server.war. Script will be able to run JVM
> which will have access to needed KC jars on classpath and access to
> persistence.xml . In AS7/Wildfly environment the persistence.xml bundled
> inside auth-server.war is using datasource though, so this JVM process
> would also need to parse datasource from
> standalone/configuration/standalone.xml as it won't be executed in
> managed AS7/Wildfly environment.
> 
> c) Use something similar to approach (b) but execute export/import from
> AS7/Wildfly CLI or admin console. The advantage is that it is triggered
> from the managed (AS7 or Wildfly) so has access to server resources like
> datasource referenced from persistence.xml

First pass would be to make users import/export with a script, or something similar. I don't think we need any support through the admin console straight up.

An alternative to script/CLI would be to just make the "server" itself do it. You'd just start the server with -Dexport=<my file> and it would export the db, then call System.exit(). While exporting we need to make sure the server doesn't allow any requests so we need a way to "pause" the server. Once you've upgraded to a new version you'd just start it with -Dimport<my file> the first time and it would import the db. Same thing on import, the server would be paused until the import is completed. We can probably import the pause with a JAX-RS filter.

It would be great if we could get the server to automatically use the export/import when an upgrade has occurred. We'd save a model "version" in the database, and whenever the server starts and detects an invalid version it would export the data to the json file, clear the database and re-import the data. We might need a prompt for users before clearing the db though. This may be complicated/time consuming to add though, but I think it would be worth considering at least.


> 
> Any other possibility I am missing?
> 
> Thanks,
> Marek
> _______________________________________________
> keycloak-dev mailing list
> keycloak-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>