[keycloak-dev] Export/import implementation

Tue May 6 16:50:51 EDT 2014

I've sent PR with export/import (migration) implementation. 
https://github.com/keycloak/keycloak/pull/366

Some notes:

* It's possible to trigger export/import during startup with usage of 
system properties. It's triggered from KeycloakApplication constructor 
similarly like all other initialization tasks (ApplianceBootstrap etc). 
Hence there was no need to create any "pause" filter as export/import is 
done before KC can serve requests

* 2 IO providers for export/import data. You can either choose from:
** "dir" provider (data are exported into JSON files in some directory 
in your filesystem. All data including password hashes are in 
plain-text, so this provider is useful mainly for testing)
** "zip" provider (data are exported into ZIP file encrypted by provided 
password. Data are exported directly from Java OutputStream into ZIP, 
nothing is written to disk unencrypted. Usage of small 3rd party library 
winzipaes). Zip provider is the default one. The ID of provider, 
location of ZIP file and password for ZIP encryption are configurable 
with system properties.

* I've used model approach as we agreed, so I needed to add few methods 
into model API. Main additions are 2 new methods 
"getCredentialsDirectly" and "updateCredentialsDirectly" for save and 
load user passwords with knowing of their hash+salt . Also I've added 
new methods, which allow to create objects by their name and ID. For 
example:
addApplication(String id, String name);
addRole(String id, String name);
...

The previous form with creating new objects just by name is indeed kept, 
so methods like:
addApplication(String name);
addRole(String name);
...
are still available. In this case ID is autogenerated like before.

* As a consequence, I did some refactoring of mongo module and removed 
dependency on picketlink-common from it. I've forked some helper 
reflection classes from it into model-api as those classes are used by 
both mongo-model and export-import. I still need to do more work related 
to picketlink isolation, so will likely continue tomorrow.

For now, I did not anything related to versions and migration 
converters. I believe that this can be postponed after Beta1 and maybe 
also after 1.0-Final? I think we can develop support for version updates 
later once we need it (so somewhere between 1.0-Final and 1.1-Final 
releases)

Marek

On 28.4.2014 17:43, Marek Posolda wrote:
> On 28.4.2014 14:52, Bill Burke wrote:
>> On 4/28/2014 3:27 AM, Marek Posolda wrote:
>>> I am planning to start soon on export/import. If I recall correctly, one
>>> of the requirements is to export the content of whole DB content
>>> (including IDs and password hashes) to JSON file, which can then be
>>> later imported into other DB. This will allow to migrate between
>>> environments and various DB types (For example from Mongo to MySQL and
>>> viceversa).
>>>
>> IMO, a full export (of credentials) should require a secret given by the
>> admin that will be used to encrypt the export.  The export should only
>> be saved locally to disk and not available over the network.
> Yes, so the exported JSON file will be always saved just to local
> filesystem with KC server and never sent over the network. With respect
> to this, I am not sure if securing the JSON file is big issue as the
> file is anyway available just to someone with physical access to the
> machine with KC server? However for better protection and ensuring that
> nobody can read it, I will encrypt the content of this file, likely with
> KDF as proposed by Bruno.
>>> I have some question though
>>>
>>> 1) I assume that DB should be cleared before full import from JSON file?
>>> Or do we want to update existing data without deleting the previous
>>> content? I assume that this is used for migration, so it's not about
>>> updating but completely delete and recreate existing DB, correct?
>>>
>>> 2) How to implement it. I can see two approaches
>>>
>>> a) Use model API to retrieve content of the DB into JSON file during
>>> export. Similarly during import use model API to sync objects back from
>>> JSON into model DB.
>>>
>>> b) Add some methods to KeycloakSession interface like:
>>>
>>> ObjectNode export();
>>>
>>> void import(ObjectNode node);
>>>
>>> and implement export/import separately for each model.
>>>
>>> Approach (b) might be better for performance as it allows to directly
>>> use low-level queries specific to JPA, Mongo or other model
>>> implementations to export/import stuff more effectively in batch,
>>> however it will require changes in model implementations and probably
>>> adding more stuff into dependencies. So I am more convinced to use (a).
>>> Thoughts?
>>>
>> "a", IMO.  Easier to maintain.
> oops, now thinking again that current model API doesn't allow to
> retrieve password hash or save password hash into DB? It allows to
> validate plain-text password or save plain-text password but there is no
> support for load or save directly the value of password hash. I wonder
> if it's ok to add some methods to RealmModel, which will allow to
> load/save password hashes?
>
> If you don't want this, I will maybe need to use low-level stuff and
> stick with option (b) .
>>> 3) How will be export/import triggered?
>>>
>>> I think that for security reasons, we want to always export into local
>>> file with KC server and similarly always import from local file. Is it
>>> correct? I can see approaches like:
>>>
>> You can already import a full json description from the admin console.
>>
>>> a) Use KC admin console. By default, just "admin" will be able to
>>> export/import stuff . Data will be always exported/imported into JSON
>>> file local to server. So it will be possible to trigger export/import
>>> remotely from admin console, but just use local JSON file. The import
>>> would be tricky though as import will recreate all data (including admin
>>> realm and admin user) so it would need to cancel logout sessions
>>> including the session of admin user who triggered import.
>>>
>>> b) Use some script, which will trigger JVM process for export/import
>>> data. Script can be executed locally from CMD. I can imagine something
>>> like this (executed from the directory with AS7/wildfly server):
>>>
>>> ./bin/keycloak-import -Dkeycloak.model=jpa
>>> -Ddata-to-import=/tmp/file-with-full-kc-data-to-import-which-were-previously-exported.json
>>>
>>> Assumption is that distribution contains deployed KC inside
>>> standalone/deployments/auth-server.war. Script will be able to run JVM
>>> which will have access to needed KC jars on classpath and access to
>>> persistence.xml . In AS7/Wildfly environment the persistence.xml bundled
>>> inside auth-server.war is using datasource though, so this JVM process
>>> would also need to parse datasource from
>>> standalone/configuration/standalone.xml as it won't be executed in
>>> managed AS7/Wildfly environment.
>>>
>>> c) Use something similar to approach (b) but execute export/import from
>>> AS7/Wildfly CLI or admin console. The advantage is that it is triggered
>>> from the managed (AS7 or Wildfly) so has access to server resources like
>>> datasource referenced from persistence.xml
>>>
>> I don't see any reason you couldn't trigger an import from the admin
>> console.  Especially if we require a secret used to encrypt the export.
>>
>>
> Sure, I can. However as already mentioned, all the KC data (including
> admin realm and current admin user who is triggering import) are going
> to be deleted and re-imported from JSON file, so there are many things,
> which need to be addressed if we want to support it on the "online"
> system like:
> - Filter, which will pause all requests until export/import is done
> - All existing states in memory (TokenManager, SocialRequestManager)
> needs to be cleared after import
> - All existing sessions should be logged out (including the session of
> current admin user) after import
>
> There might be more issues related to this and user experience might not
> be good. So it seems to be much easier to support importing data either
> on KC startup as proposed by Stian or at "offline" time when KC server
> is not running at all, which can be done with script?
>
> Exporting data from KC DB into JSON file seems to be safer, but there is
> still a need to pause the server with HTTP or JAX-RS filter, as if users
> are allowed to change data during export (like changing password or
> changing their emails etc), it might result in broken exported data.
>
> I don't know exactly how are production systems deal with
> backup/migration of data, don't have much experience with this. But I
> assume that administrators are usually not doing it on the "online" data.
>
> Marek
> _______________________________________________
> keycloak-dev mailing list
> keycloak-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-dev