[keycloak-dev] Batch import/export

[Stian Thorgersen]

The good old import/export stuff does pagination. I don't think the admin
console export you're working on needs pagination.

On 23 October 2015 at 15:11, Stan Silvert <ssilvert at redhat.com> wrote:

> On 10/23/2015 9:09 AM, Stian Thorgersen wrote:
>
> We already have paginated export of users though?
>
> I haven't started on export.  So far, I've done batch import of users,
> clients, and identity providers.
>
>
> On 23 October 2015 at 14:22, Marek Posolda <mposolda at redhat.com> wrote:
>
>> Sorry for late response. Is the scope of this task also to improve big
>> long-running export/import ? We already have someone with 75.000 users in
>> DB.
>>
>> It seems that export/import is kind of task, which can be easily
>> "paginated" (Export of user1 - user500 is page1, export of user501-user1000
>> is page2 etc). We may have the table where you can see the progress of
>> long-running export/import job and how many users (pages) are already
>> exported.
>>
>> Besides that we may support:
>> - Concurrency (more worker threads. Each worker doing an export of
>> different page)
>> - Cluster scalability (Each cluster node takes some pages and helps with
>> the overall export task)
>> - Cluster failover (When some cluster node crashes during export job, the
>> whole job is not canceled but continue on other cluster nodes without
>> problem)
>>
>> I've already addressed the concurrency, scalability and failover with the InfinispanUserSessionInitializer,
>> which is used at startup for pre-load offline sessions from DB. It's based
>> on infinispan Distributed Executor service. I think that with some minor
>> changes (maybe even without them) it can be reused for any long running
>> "paginatable" job (export/import, sync of big number of users from
>> federationProvider etc)
>>
>> Marek
>>
>> On 21/10/15 14:44, Stian Thorgersen wrote:
>>
>> I've just added client registration services in 1.6 which should be more
>> useful for that. There's also a Java library. Basically admins and service
>> accounts with the role create-client can create new clients, while clients
>> themselves have permissions to update their config.
>>
>> You could have a client template that you change the client-id only then
>> use this endpoint to register the client.
>>
>> On 21 October 2015 at 14:40, Thomas Raehalme <
>> thomas.raehalme at aitiofinland.com> wrote:
>>
>>> If you deploy the same application multiple times for different
>>> customers you could have a configuration template containing all the common
>>> bits and pieces, but have Keycloak generate keys and secrets when you
>>> import the configuration.
>>>
>>> Best regards,
>>> Thomas
>>>
>>>
>>> On Wed, Oct 21, 2015 at 3:33 PM, Stian Thorgersen <sthorger at redhat.com>
>>> wrote:
>>>
>>>> Can you elaborate a bit on that?
>>>>
>>>> On 21 October 2015 at 14:29, Thomas Raehalme <
>>>> thomas.raehalme at aitiofinland.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Wed, Oct 21, 2015 at 3:27 PM, Stian Thorgersen <sthorger at redhat.com
>>>>> > wrote:
>>>>>
>>>>>> I'd like to get import/export done properly. The addition of being
>>>>>> able to add bits and pieces to import in a directory would be really
>>>>>> helpful on Docker/OpenShift/etc..
>>>>>>
>>>>>
>>>>> I had similar things in mind when I suggested the re-generation of
>>>>> keys and secrets. You could define a template which you'd use in a process
>>>>> for new deployments.
>>>>>
>>>>> Best regards,
>>>>> Thomas
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> *Thomas Raehalme*
>>> *CTO, teknologiajohtaja*
>>> Mobile +358 40 545 0605
>>>
>>> *Aitio Finland Oy*
>>> Väinönkatu 26 A
>>> 40100 JYVÄSKYLÄ, Finland
>>> Tel. +358 10 322 0040
>>> www.aitiofinland.com
>>>
>>> *Codecenter on nyt Aitio – me kun ei vain koodata!*
>>>
>>>
>>
>>
>> _______________________________________________
>> keycloak-dev mailing listkeycloak-dev at lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/keycloak-dev
>>
>>
>>
>
>
> _______________________________________________
> keycloak-dev mailing listkeycloak-dev at lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/keycloak-dev
>
>
>
> _______________________________________________
> keycloak-dev mailing list
> keycloak-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/keycloak-dev/attachments/20151023/48742e09/attachment-0001.html