[keycloak-user] Very slow export/import of realms

[Marek Posolda]

Thanks for reporting this. Could you please create JIRA and describe 
your findings? We didn't yet try to test export/import with bigger 
number of users, so didn't spotted this yet.

I think the only reason for the redundant flush/detach calls is, that 
"something" didn't work as expected (maybe even just during development 
phase) and flush helped to solve the issue. This doesn't mean that it's 
necessarily correct. Now we are close to the "feature freeze" and we 
will probably look at performance tuning. So we will likely look at this 
issue too.

Btv. are you using default H2 database or something else? Could you 
please mention this in JIRA too?

Thanks,
Marek

On 28/01/16 19:22, Gabriel Lavoie wrote:
> Hi,
>     I am currently having issues with very slow export/import of 
> realms with a large number of users (10K). Both operations take ~10 
> minutes each.
>
> After digging in the KeyCloak code, I've found out that a lot of 
> "flush" is done at the Hibernate/JPA level (at least 4-5 times per user).
>
> _Export:_
> For the export, there are 4 very slow queries in 
> ExportUtils.exportUser() 
> (https://github.com/keycloak/keycloak/blob/7895eb6a3dd7e4abc16dd401066bd08ed8075ffb/services/src/main/java/org/keycloak/exportimport/util/ExportUtils.java):
> * Set<FederatedIdentityModel> socialLinks = 
> session.users().getFederatedIdentities(user, realm);
> * Set<RoleModel> roles = user.getRoleMappings();
> * List<UserConsentModel> consents = user.getConsents();
> * for(GroupModel group : user.getGroups()) {
>
> They seem to be caused by Hibernate that forces a flush by default in 
> "getResultList()". As a workaround I told Hibernate to flush only on 
> commit of the transaction through "-Dorg.hibernate.flushMode=COMMIT" 
> and the export time came down to _~20 seconds_.
>
> Could there be any issue of changing the flush mode only for the 
> export? Data shouldn't change in the database at this moment and 
> queries shouldn't return stale data.
>
> _Import:_
> For the import it seems that changing the flushMode isn't sufficient. 
> It should likely not be done. However, I found a few places in the 
> keycloak-model-jpa module where entities are created, persisted, then 
> em.flush() and em.detatch() is called right away without the entity 
> being returned (ex: UserAdapter.grantRole() - 
> https://github.com/keycloak/keycloak/blob/master/model/jpa/src/main/java/org/keycloak/models/jpa/UserAdapter.java). 
>
>
> If I remove all those flush/detach calls, the import process goes down 
> to _~50 seconds_. What is the reason for flushing every time an entity 
> is created rather than letting JPA/Hibernate do it when necessary?
>
> Thank you,
>
> Gabriel
> -- 
> Gabriel Lavoie
> glavoie at gmail.com <mailto:glavoie at gmail.com>
>
>
> _______________________________________________
> keycloak-user mailing list
> keycloak-user at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/keycloak-user/attachments/20160128/6e719210/attachment.html