[keycloak-dev] Export/import implementation

[Stian Thorgersen]

----- Original Message -----
> From: "Marek Posolda" <mposolda at redhat.com>
> To: "Stian Thorgersen" <stian at redhat.com>
> Cc: keycloak-dev at lists.jboss.org
> Sent: Monday, 28 April, 2014 10:28:57 AM
> Subject: Re: [keycloak-dev] Export/import implementation
> 
> On 28.4.2014 10:39, Stian Thorgersen wrote:
> >
> > ----- Original Message -----
> >> From: "Marek Posolda" <mposolda at redhat.com>
> >> To: keycloak-dev at lists.jboss.org
> >> Sent: Monday, 28 April, 2014 8:27:45 AM
> >> Subject: [keycloak-dev] Export/import implementation
> >>
> >> I am planning to start soon on export/import. If I recall correctly, one
> >> of the requirements is to export the content of whole DB content
> >> (including IDs and password hashes) to JSON file, which can then be
> >> later imported into other DB. This will allow to migrate between
> >> environments and various DB types (For example from Mongo to MySQL and
> >> viceversa).
> >>
> >> I have some question though
> >>
> >> 1) I assume that DB should be cleared before full import from JSON file?
> >> Or do we want to update existing data without deleting the previous
> >> content? I assume that this is used for migration, so it's not about
> >> updating but completely delete and recreate existing DB, correct?
> > Yes, I think first pass should be full DB export and import. If there is
> > time, or at some point in the future, we can consider being able to
> > import/export individual realms, and even some form of a merge on import
> >
> >> 2) How to implement it. I can see two approaches
> >>
> >> a) Use model API to retrieve content of the DB into JSON file during
> >> export. Similarly during import use model API to sync objects back from
> >> JSON into model DB.
> >>
> >> b) Add some methods to KeycloakSession interface like:
> >>
> >> ObjectNode export();
> >>
> >> void import(ObjectNode node);
> >>
> >> and implement export/import separately for each model.
> >>
> >> Approach (b) might be better for performance as it allows to directly
> >> use low-level queries specific to JPA, Mongo or other model
> >> implementations to export/import stuff more effectively in batch,
> >> however it will require changes in model implementations and probably
> >> adding more stuff into dependencies. So I am more convinced to use (a).
> >> Thoughts?
> > I'd say let's go with option a. If performance does become a real headache
> > in the future we can reconsider
> >
> >>
> >> 3) How will be export/import triggered?
> >>
> >> I think that for security reasons, we want to always export into local
> >> file with KC server and similarly always import from local file. Is it
> >> correct? I can see approaches like:
> >>
> >> a) Use KC admin console. By default, just "admin" will be able to
> >> export/import stuff . Data will be always exported/imported into JSON
> >> file local to server. So it will be possible to trigger export/import
> >> remotely from admin console, but just use local JSON file. The import
> >> would be tricky though as import will recreate all data (including admin
> >> realm and admin user) so it would need to cancel logout sessions
> >> including the session of admin user who triggered import.
> >>
> >> b) Use some script, which will trigger JVM process for export/import
> >> data. Script can be executed locally from CMD. I can imagine something
> >> like this (executed from the directory with AS7/wildfly server):
> >>
> >> ./bin/keycloak-import -Dkeycloak.model=jpa
> >> -Ddata-to-import=/tmp/file-with-full-kc-data-to-import-which-were-previously-exported.json
> >>
> >> Assumption is that distribution contains deployed KC inside
> >> standalone/deployments/auth-server.war. Script will be able to run JVM
> >> which will have access to needed KC jars on classpath and access to
> >> persistence.xml . In AS7/Wildfly environment the persistence.xml bundled
> >> inside auth-server.war is using datasource though, so this JVM process
> >> would also need to parse datasource from
> >> standalone/configuration/standalone.xml as it won't be executed in
> >> managed AS7/Wildfly environment.
> >>
> >> c) Use something similar to approach (b) but execute export/import from
> >> AS7/Wildfly CLI or admin console. The advantage is that it is triggered
> >> from the managed (AS7 or Wildfly) so has access to server resources like
> >> datasource referenced from persistence.xml
> > First pass would be to make users import/export with a script, or something
> > similar. I don't think we need any support through the admin console
> > straight up.
> >
> > An alternative to script/CLI would be to just make the "server" itself do
> > it. You'd just start the server with -Dexport=<my file> and it would
> > export the db, then call System.exit(). While exporting we need to make
> > sure the server doesn't allow any requests so we need a way to "pause" the
> > server. Once you've upgraded to a new version you'd just start it with
> > -Dimport<my file> the first time and it would import the db. Same thing on
> > import, the server would be paused until the import is completed. We can
> > probably import the pause with a JAX-RS filter.
> yes, I think it will work and KC would be able to find DS as it's
> executed in managed environment and filter would take care of the pauses
> during import/export. So I will likely go this way. Thanks.
> 
> I was also thinking about adding filter if we want to support
> export/import from console, however supporting at with console would be
> more complicated as it would require pause of already running server,
> clearing all AccessCodeEntry tickets in progress, logout all existing
> sessions etc. If we want to support it just at startup of the server,
> things are much easier.
> >
> > It would be great if we could get the server to automatically use the
> > export/import when an upgrade has occurred. We'd save a model "version" in
> > the database, and whenever the server starts and detects an invalid
> > version it would export the data to the json file, clear the database and
> > re-import the data. We might need a prompt for users before clearing the
> > db though. This may be complicated/time consuming to add though, but I
> > think it would be worth considering at least.
> I will look once I have all other things done. However I think that some
> manual work would be almost always needed from users during version
> update. For example there might be some new configuration options of the
> realm configuration and other configuration options might be removed.
> Some might be changed to support different type (For example some
> "boolean" property might be changed to support enum with 3 values etc).
> Maybe we can provide also tool to update previously exported JSON file
> to newer version of model? I know that in GateIn there is usage of this:
> https://windup.jboss.org/ . I don't know details and not sure if it's
> good also for our scenario though.

I was thinking the import should be able to import an export from any previous versions. The json export should contain a version number then it goes through a chain of converters to get to the current version. That way each new version only needs to know how to convert from the previous version, but you can still import any version.

> 
> Marek
> 
> >
> >
> >> Any other possibility I am missing?
> >>
> >> Thanks,
> >> Marek
> >> _______________________________________________
> >> keycloak-dev mailing list
> >> keycloak-dev at lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/keycloak-dev
> >>
> 
>