[keycloak-dev] Export/import implementation

Wed May 7 04:14:08 EDT 2014

Nice :)

----- Original Message -----
> From: "Marek Posolda" <mposolda at redhat.com>
> To: "Bill Burke" <bburke at redhat.com>, keycloak-dev at lists.jboss.org
> Sent: Tuesday, 6 May, 2014 9:50:51 PM
> Subject: Re: [keycloak-dev] Export/import implementation
> 
> I've sent PR with export/import (migration) implementation.
> https://github.com/keycloak/keycloak/pull/366
> 
> Some notes:
> 
> * It's possible to trigger export/import during startup with usage of
> system properties. It's triggered from KeycloakApplication constructor
> similarly like all other initialization tasks (ApplianceBootstrap etc).
> Hence there was no need to create any "pause" filter as export/import is
> done before KC can serve requests
> 
> * 2 IO providers for export/import data. You can either choose from:
> ** "dir" provider (data are exported into JSON files in some directory
> in your filesystem. All data including password hashes are in
> plain-text, so this provider is useful mainly for testing)
> ** "zip" provider (data are exported into ZIP file encrypted by provided
> password. Data are exported directly from Java OutputStream into ZIP,
> nothing is written to disk unencrypted. Usage of small 3rd party library
> winzipaes). Zip provider is the default one. The ID of provider,
> location of ZIP file and password for ZIP encryption are configurable
> with system properties.
> 
> * I've used model approach as we agreed, so I needed to add few methods
> into model API. Main additions are 2 new methods
> "getCredentialsDirectly" and "updateCredentialsDirectly" for save and
> load user passwords with knowing of their hash+salt . Also I've added
> new methods, which allow to create objects by their name and ID. For
> example:
> addApplication(String id, String name);
> addRole(String id, String name);
> ...
> 
> The previous form with creating new objects just by name is indeed kept,
> so methods like:
> addApplication(String name);
> addRole(String name);
> ...
> are still available. In this case ID is autogenerated like before.
> 
> * As a consequence, I did some refactoring of mongo module and removed
> dependency on picketlink-common from it. I've forked some helper
> reflection classes from it into model-api as those classes are used by
> both mongo-model and export-import. I still need to do more work related
> to picketlink isolation, so will likely continue tomorrow.
> 
> For now, I did not anything related to versions and migration
> converters. I believe that this can be postponed after Beta1 and maybe
> also after 1.0-Final? I think we can develop support for version updates
> later once we need it (so somewhere between 1.0-Final and 1.1-Final
> releases)

Yes, that's fine

> 
> Marek
> 
> 
> On 28.4.2014 17:43, Marek Posolda wrote:
> > On 28.4.2014 14:52, Bill Burke wrote:
> >> On 4/28/2014 3:27 AM, Marek Posolda wrote:
> >>> I am planning to start soon on export/import. If I recall correctly, one
> >>> of the requirements is to export the content of whole DB content
> >>> (including IDs and password hashes) to JSON file, which can then be
> >>> later imported into other DB. This will allow to migrate between
> >>> environments and various DB types (For example from Mongo to MySQL and
> >>> viceversa).
> >>>
> >> IMO, a full export (of credentials) should require a secret given by the
> >> admin that will be used to encrypt the export.  The export should only
> >> be saved locally to disk and not available over the network.
> > Yes, so the exported JSON file will be always saved just to local
> > filesystem with KC server and never sent over the network. With respect
> > to this, I am not sure if securing the JSON file is big issue as the
> > file is anyway available just to someone with physical access to the
> > machine with KC server? However for better protection and ensuring that
> > nobody can read it, I will encrypt the content of this file, likely with
> > KDF as proposed by Bruno.
> >>> I have some question though
> >>>
> >>> 1) I assume that DB should be cleared before full import from JSON file?
> >>> Or do we want to update existing data without deleting the previous
> >>> content? I assume that this is used for migration, so it's not about
> >>> updating but completely delete and recreate existing DB, correct?
> >>>
> >>> 2) How to implement it. I can see two approaches
> >>>
> >>> a) Use model API to retrieve content of the DB into JSON file during
> >>> export. Similarly during import use model API to sync objects back from
> >>> JSON into model DB.
> >>>
> >>> b) Add some methods to KeycloakSession interface like:
> >>>
> >>> ObjectNode export();
> >>>
> >>> void import(ObjectNode node);
> >>>
> >>> and implement export/import separately for each model.
> >>>
> >>> Approach (b) might be better for performance as it allows to directly
> >>> use low-level queries specific to JPA, Mongo or other model
> >>> implementations to export/import stuff more effectively in batch,
> >>> however it will require changes in model implementations and probably
> >>> adding more stuff into dependencies. So I am more convinced to use (a).
> >>> Thoughts?
> >>>
> >> "a", IMO.  Easier to maintain.
> > oops, now thinking again that current model API doesn't allow to
> > retrieve password hash or save password hash into DB? It allows to
> > validate plain-text password or save plain-text password but there is no
> > support for load or save directly the value of password hash. I wonder
> > if it's ok to add some methods to RealmModel, which will allow to
> > load/save password hashes?
> >
> > If you don't want this, I will maybe need to use low-level stuff and
> > stick with option (b) .
> >>> 3) How will be export/import triggered?
> >>>
> >>> I think that for security reasons, we want to always export into local
> >>> file with KC server and similarly always import from local file. Is it
> >>> correct? I can see approaches like:
> >>>
> >> You can already import a full json description from the admin console.
> >>
> >>> a) Use KC admin console. By default, just "admin" will be able to
> >>> export/import stuff . Data will be always exported/imported into JSON
> >>> file local to server. So it will be possible to trigger export/import
> >>> remotely from admin console, but just use local JSON file. The import
> >>> would be tricky though as import will recreate all data (including admin
> >>> realm and admin user) so it would need to cancel logout sessions
> >>> including the session of admin user who triggered import.
> >>>
> >>> b) Use some script, which will trigger JVM process for export/import
> >>> data. Script can be executed locally from CMD. I can imagine something
> >>> like this (executed from the directory with AS7/wildfly server):
> >>>
> >>> ./bin/keycloak-import -Dkeycloak.model=jpa
> >>> -Ddata-to-import=/tmp/file-with-full-kc-data-to-import-which-were-previously-exported.json
> >>>
> >>> Assumption is that distribution contains deployed KC inside
> >>> standalone/deployments/auth-server.war. Script will be able to run JVM
> >>> which will have access to needed KC jars on classpath and access to
> >>> persistence.xml . In AS7/Wildfly environment the persistence.xml bundled
> >>> inside auth-server.war is using datasource though, so this JVM process
> >>> would also need to parse datasource from
> >>> standalone/configuration/standalone.xml as it won't be executed in
> >>> managed AS7/Wildfly environment.
> >>>
> >>> c) Use something similar to approach (b) but execute export/import from
> >>> AS7/Wildfly CLI or admin console. The advantage is that it is triggered
> >>> from the managed (AS7 or Wildfly) so has access to server resources like
> >>> datasource referenced from persistence.xml
> >>>
> >> I don't see any reason you couldn't trigger an import from the admin
> >> console.  Especially if we require a secret used to encrypt the export.
> >>
> >>
> > Sure, I can. However as already mentioned, all the KC data (including
> > admin realm and current admin user who is triggering import) are going
> > to be deleted and re-imported from JSON file, so there are many things,
> > which need to be addressed if we want to support it on the "online"
> > system like:
> > - Filter, which will pause all requests until export/import is done
> > - All existing states in memory (TokenManager, SocialRequestManager)
> > needs to be cleared after import
> > - All existing sessions should be logged out (including the session of
> > current admin user) after import
> >
> > There might be more issues related to this and user experience might not
> > be good. So it seems to be much easier to support importing data either
> > on KC startup as proposed by Stian or at "offline" time when KC server
> > is not running at all, which can be done with script?
> >
> > Exporting data from KC DB into JSON file seems to be safer, but there is
> > still a need to pause the server with HTTP or JAX-RS filter, as if users
> > are allowed to change data during export (like changing password or
> > changing their emails etc), it might result in broken exported data.
> >
> > I don't know exactly how are production systems deal with
> > backup/migration of data, don't have much experience with this. But I
> > assume that administrators are usually not doing it on the "online" data.
> >
> > Marek
> > _______________________________________________
> > keycloak-dev mailing list
> > keycloak-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/keycloak-dev
> 
> _______________________________________________
> keycloak-dev mailing list
> keycloak-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>