[keycloak-dev] Export/import implementation

[Bill Burke]

Great work Marek!

On 5/6/2014 4:50 PM, Marek Posolda wrote:
> I've sent PR with export/import (migration) implementation.
> https://github.com/keycloak/keycloak/pull/366
>
> Some notes:
>
> * It's possible to trigger export/import during startup with usage of
> system properties. It's triggered from KeycloakApplication constructor
> similarly like all other initialization tasks (ApplianceBootstrap etc).
> Hence there was no need to create any "pause" filter as export/import is
> done before KC can serve requests
>
> * 2 IO providers for export/import data. You can either choose from:
> ** "dir" provider (data are exported into JSON files in some directory
> in your filesystem. All data including password hashes are in
> plain-text, so this provider is useful mainly for testing)
> ** "zip" provider (data are exported into ZIP file encrypted by provided
> password. Data are exported directly from Java OutputStream into ZIP,
> nothing is written to disk unencrypted. Usage of small 3rd party library
> winzipaes). Zip provider is the default one. The ID of provider,
> location of ZIP file and password for ZIP encryption are configurable
> with system properties.
>
> * I've used model approach as we agreed, so I needed to add few methods
> into model API. Main additions are 2 new methods
> "getCredentialsDirectly" and "updateCredentialsDirectly" for save and
> load user passwords with knowing of their hash+salt . Also I've added
> new methods, which allow to create objects by their name and ID. For
> example:
> addApplication(String id, String name);
> addRole(String id, String name);
> ...
>
> The previous form with creating new objects just by name is indeed kept,
> so methods like:
> addApplication(String name);
> addRole(String name);
> ...
> are still available. In this case ID is autogenerated like before.
>
> * As a consequence, I did some refactoring of mongo module and removed
> dependency on picketlink-common from it. I've forked some helper
> reflection classes from it into model-api as those classes are used by
> both mongo-model and export-import. I still need to do more work related
> to picketlink isolation, so will likely continue tomorrow.
>
> For now, I did not anything related to versions and migration
> converters. I believe that this can be postponed after Beta1 and maybe
> also after 1.0-Final? I think we can develop support for version updates
> later once we need it (so somewhere between 1.0-Final and 1.1-Final
> releases)
>
> Marek
>
>
> On 28.4.2014 17:43, Marek Posolda wrote:
>> On 28.4.2014 14:52, Bill Burke wrote:
>>> On 4/28/2014 3:27 AM, Marek Posolda wrote:
>>>> I am planning to start soon on export/import. If I recall correctly,
>>>> one
>>>> of the requirements is to export the content of whole DB content
>>>> (including IDs and password hashes) to JSON file, which can then be
>>>> later imported into other DB. This will allow to migrate between
>>>> environments and various DB types (For example from Mongo to MySQL and
>>>> viceversa).
>>>>
>>> IMO, a full export (of credentials) should require a secret given by the
>>> admin that will be used to encrypt the export.  The export should only
>>> be saved locally to disk and not available over the network.
>> Yes, so the exported JSON file will be always saved just to local
>> filesystem with KC server and never sent over the network. With respect
>> to this, I am not sure if securing the JSON file is big issue as the
>> file is anyway available just to someone with physical access to the
>> machine with KC server? However for better protection and ensuring that
>> nobody can read it, I will encrypt the content of this file, likely with
>> KDF as proposed by Bruno.
>>>> I have some question though
>>>>
>>>> 1) I assume that DB should be cleared before full import from JSON
>>>> file?
>>>> Or do we want to update existing data without deleting the previous
>>>> content? I assume that this is used for migration, so it's not about
>>>> updating but completely delete and recreate existing DB, correct?
>>>>
>>>> 2) How to implement it. I can see two approaches
>>>>
>>>> a) Use model API to retrieve content of the DB into JSON file during
>>>> export. Similarly during import use model API to sync objects back from
>>>> JSON into model DB.
>>>>
>>>> b) Add some methods to KeycloakSession interface like:
>>>>
>>>> ObjectNode export();
>>>>
>>>> void import(ObjectNode node);
>>>>
>>>> and implement export/import separately for each model.
>>>>
>>>> Approach (b) might be better for performance as it allows to directly
>>>> use low-level queries specific to JPA, Mongo or other model
>>>> implementations to export/import stuff more effectively in batch,
>>>> however it will require changes in model implementations and probably
>>>> adding more stuff into dependencies. So I am more convinced to use (a).
>>>> Thoughts?
>>>>
>>> "a", IMO.  Easier to maintain.
>> oops, now thinking again that current model API doesn't allow to
>> retrieve password hash or save password hash into DB? It allows to
>> validate plain-text password or save plain-text password but there is no
>> support for load or save directly the value of password hash. I wonder
>> if it's ok to add some methods to RealmModel, which will allow to
>> load/save password hashes?
>>
>> If you don't want this, I will maybe need to use low-level stuff and
>> stick with option (b) .
>>>> 3) How will be export/import triggered?
>>>>
>>>> I think that for security reasons, we want to always export into local
>>>> file with KC server and similarly always import from local file. Is it
>>>> correct? I can see approaches like:
>>>>
>>> You can already import a full json description from the admin console.
>>>
>>>> a) Use KC admin console. By default, just "admin" will be able to
>>>> export/import stuff . Data will be always exported/imported into JSON
>>>> file local to server. So it will be possible to trigger export/import
>>>> remotely from admin console, but just use local JSON file. The import
>>>> would be tricky though as import will recreate all data (including
>>>> admin
>>>> realm and admin user) so it would need to cancel logout sessions
>>>> including the session of admin user who triggered import.
>>>>
>>>> b) Use some script, which will trigger JVM process for export/import
>>>> data. Script can be executed locally from CMD. I can imagine something
>>>> like this (executed from the directory with AS7/wildfly server):
>>>>
>>>> ./bin/keycloak-import -Dkeycloak.model=jpa
>>>> -Ddata-to-import=/tmp/file-with-full-kc-data-to-import-which-were-previously-exported.json
>>>>
>>>>
>>>> Assumption is that distribution contains deployed KC inside
>>>> standalone/deployments/auth-server.war. Script will be able to run JVM
>>>> which will have access to needed KC jars on classpath and access to
>>>> persistence.xml . In AS7/Wildfly environment the persistence.xml
>>>> bundled
>>>> inside auth-server.war is using datasource though, so this JVM process
>>>> would also need to parse datasource from
>>>> standalone/configuration/standalone.xml as it won't be executed in
>>>> managed AS7/Wildfly environment.
>>>>
>>>> c) Use something similar to approach (b) but execute export/import from
>>>> AS7/Wildfly CLI or admin console. The advantage is that it is triggered
>>>> from the managed (AS7 or Wildfly) so has access to server resources
>>>> like
>>>> datasource referenced from persistence.xml
>>>>
>>> I don't see any reason you couldn't trigger an import from the admin
>>> console.  Especially if we require a secret used to encrypt the export.
>>>
>>>
>> Sure, I can. However as already mentioned, all the KC data (including
>> admin realm and current admin user who is triggering import) are going
>> to be deleted and re-imported from JSON file, so there are many things,
>> which need to be addressed if we want to support it on the "online"
>> system like:
>> - Filter, which will pause all requests until export/import is done
>> - All existing states in memory (TokenManager, SocialRequestManager)
>> needs to be cleared after import
>> - All existing sessions should be logged out (including the session of
>> current admin user) after import
>>
>> There might be more issues related to this and user experience might not
>> be good. So it seems to be much easier to support importing data either
>> on KC startup as proposed by Stian or at "offline" time when KC server
>> is not running at all, which can be done with script?
>>
>> Exporting data from KC DB into JSON file seems to be safer, but there is
>> still a need to pause the server with HTTP or JAX-RS filter, as if users
>> are allowed to change data during export (like changing password or
>> changing their emails etc), it might result in broken exported data.
>>
>> I don't know exactly how are production systems deal with
>> backup/migration of data, don't have much experience with this. But I
>> assume that administrators are usually not doing it on the "online" data.
>>
>> Marek
>> _______________________________________________
>> keycloak-dev mailing list
>> keycloak-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>

-- 
Bill Burke
JBoss, a division of Red Hat
http://bill.burkecentral.com