[keycloak-dev] Batch import/export

[Marek Posolda]

Yes, DirExportProvider provides pagination in separate transactions. But 
it's doing everything sequentially on single cluster node (no cluster 
awareness). If host crashes after 50.000 users exported, you need to 
start again.

In shortcut, what we don't yet have is: concurrency, cluster scalability 
+ failover, possibility to see progress of long running task in admin 
console

Marek

On 23/10/15 15:09, Stian Thorgersen wrote:
> We already have paginated export of users though?
>
> On 23 October 2015 at 14:22, Marek Posolda <mposolda at redhat.com 
> <mailto:mposolda at redhat.com>> wrote:
>
>     Sorry for late response. Is the scope of this task also to improve
>     big long-running export/import ? We already have someone with
>     75.000 users in DB.
>
>     It seems that export/import is kind of task, which can be easily
>     "paginated" (Export of user1 - user500 is page1, export of
>     user501-user1000 is page2 etc). We may have the table where you
>     can see the progress of long-running export/import job and how
>     many users (pages) are already exported.
>
>     Besides that we may support:
>     - Concurrency (more worker threads. Each worker doing an export of
>     different page)
>     - Cluster scalability (Each cluster node takes some pages and
>     helps with the overall export task)
>     - Cluster failover (When some cluster node crashes during export
>     job, the whole job is not canceled but continue on other cluster
>     nodes without problem)
>
>     I've already addressed the concurrency, scalability and failover
>     with the InfinispanUserSessionInitializer, which is used at
>     startup for pre-load offline sessions from DB. It's based on
>     infinispan Distributed Executor service. I think that with some
>     minor changes (maybe even without them) it can be reused for any
>     long running "paginatable" job (export/import, sync of big number
>     of users from federationProvider etc)
>
>     Marek
>
>     On 21/10/15 14:44, Stian Thorgersen wrote:
>>     I've just added client registration services in 1.6 which should
>>     be more useful for that. There's also a Java library. Basically
>>     admins and service accounts with the role create-client can
>>     create new clients, while clients themselves have permissions to
>>     update their config.
>>
>>     You could have a client template that you change the client-id
>>     only then use this endpoint to register the client.
>>
>>     On 21 October 2015 at 14:40, Thomas Raehalme
>>     <thomas.raehalme at aitiofinland.com
>>     <mailto:thomas.raehalme at aitiofinland.com>> wrote:
>>
>>         If you deploy the same application multiple times for
>>         different customers you could have a configuration template
>>         containing all the common bits and pieces, but have Keycloak
>>         generate keys and secrets when you import the configuration.
>>
>>         Best regards,
>>         Thomas
>>
>>
>>         On Wed, Oct 21, 2015 at 3:33 PM, Stian Thorgersen
>>         <sthorger at redhat.com <mailto:sthorger at redhat.com>> wrote:
>>
>>             Can you elaborate a bit on that?
>>
>>             On 21 October 2015 at 14:29, Thomas Raehalme
>>             <thomas.raehalme at aitiofinland.com
>>             <mailto:thomas.raehalme at aitiofinland.com>> wrote:
>>
>>
>>
>>                 On Wed, Oct 21, 2015 at 3:27 PM, Stian Thorgersen
>>                 <sthorger at redhat.com <mailto:sthorger at redhat.com>> wrote:
>>
>>                     I'd like to get import/export done properly. The
>>                     addition of being able to add bits and pieces to
>>                     import in a directory would be really helpful on
>>                     Docker/OpenShift/etc..
>>
>>
>>                 I had similar things in mind when I suggested the
>>                 re-generation of keys and secrets. You could define a
>>                 template which you'd use in a process for new
>>                 deployments.
>>
>>                 Best regards,
>>                 Thomas
>>
>>
>>
>>
>>
>>         -- 
>>         *Thomas Raehalme*
>>         /CTO, teknologiajohtaja/
>>         Mobile +358 40 545 0605 <tel:%2B358%2040%20545%200605>
>>
>>         *Aitio Finland Oy*
>>         Väinönkatu 26 A
>>         40100 JYVÄSKYLÄ, Finland
>>         Tel. +358 10 322 0040 <tel:%2B358%2010%20322%200040>
>>         www.aitiofinland.com <http://www.aitiofinland.com>
>>
>>         *Codecenter on nyt Aitio – me kun ei vain koodata!*
>>
>>
>>
>>
>>     _______________________________________________
>>     keycloak-dev mailing list
>>     keycloak-dev at lists.jboss.org <mailto:keycloak-dev at lists.jboss.org>
>>     https://lists.jboss.org/mailman/listinfo/keycloak-dev
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/keycloak-dev/attachments/20151023/5787c241/attachment.html