[keycloak-user] Very slow export/import of realms

[Gabriel Lavoie]

Hi,
    I am currently having issues with very slow export/import of realms
with a large number of users (10K). Both operations take ~10 minutes each.

After digging in the KeyCloak code, I've found out that a lot of "flush" is
done at the Hibernate/JPA level (at least 4-5 times per user).

*Export:*
For the export, there are 4 very slow queries in ExportUtils.exportUser() (
https://github.com/keycloak/keycloak/blob/7895eb6a3dd7e4abc16dd401066bd08ed8075ffb/services/src/main/java/org/keycloak/exportimport/util/ExportUtils.java
):
* Set<FederatedIdentityModel> socialLinks =
session.users().getFederatedIdentities(user, realm);
* Set<RoleModel> roles = user.getRoleMappings();
* List<UserConsentModel> consents = user.getConsents();
* for (GroupModel group : user.getGroups()) {

They seem to be caused by Hibernate that forces a flush by default in
"getResultList()". As a workaround I told Hibernate to flush only on commit
of the transaction through "-Dorg.hibernate.flushMode=COMMIT" and the
export time came down to *~20 seconds*.

Could there be any issue of changing the flush mode only for the export?
Data shouldn't change in the database at this moment and queries shouldn't
return stale data.

*Import:*
For the import it seems that changing the flushMode isn't sufficient. It
should likely not be done. However, I found a few places in the
keycloak-model-jpa module where entities are created, persisted, then
em.flush() and em.detatch() is called right away without the entity being
returned (ex: UserAdapter.grantRole() -
https://github.com/keycloak/keycloak/blob/master/model/jpa/src/main/java/org/keycloak/models/jpa/UserAdapter.java
).

If I remove all those flush/detach calls, the import process goes down to *~50
seconds*. What is the reason for flushing every time an entity is created
rather than letting JPA/Hibernate do it when necessary?

Thank you,

Gabriel
-- 
Gabriel Lavoie
glavoie at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/keycloak-user/attachments/20160128/d23ae2aa/attachment.html