<div dir="ltr"><div>Hi,</div><div> I am currently having issues with very slow export/import of realms with a large number of users (10K). Both operations take ~10 minutes each. </div><div><br></div><div>After digging in the KeyCloak code, I've found out that a lot of "flush" is done at the Hibernate/JPA level (at least 4-5 times per user).</div><div><br></div><div><font face="arial, helvetica, sans-serif"><u>Export:</u></font></div><div><font face="arial, helvetica, sans-serif">For the export, there are 4 very slow queries in ExportUtils.exportUser() (<a href="https://github.com/keycloak/keycloak/blob/7895eb6a3dd7e4abc16dd401066bd08ed8075ffb/services/src/main/java/org/keycloak/exportimport/util/ExportUtils.java">https://github.com/keycloak/keycloak/blob/7895eb6a3dd7e4abc16dd401066bd08ed8075ffb/services/src/main/java/org/keycloak/exportimport/util/ExportUtils.java</a></font><span style="font-family:arial,helvetica,sans-serif">):</span></div><div><font face="arial, helvetica, sans-serif">* Set<FederatedIdentityModel> socialLinks = session.users().getFederatedIdentities(user, realm);</font></div><div><font face="arial, helvetica, sans-serif">* Set<RoleModel> roles = user.getRoleMappings();</font></div><div><font face="arial, helvetica, sans-serif">* <span style="color:rgb(0,0,0)">List<UserConsentModel> consents = user.getConsents();</span></font></div><div><font face="arial, helvetica, sans-serif" color="#000000">* for<span style="font-weight:bold"> </span>(GroupModel group : user.getGroups()) {</font></div><div><font face="arial, helvetica, sans-serif" color="#000000"><br></font></div><div><font color="#000000" face="arial, helvetica, sans-serif">They seem to be caused by Hibernate that forces a flush by default in "getResultList()". As a workaround I told Hibernate to flush only on commit of the transaction through "-Dorg.hibernate.flushMode=COMMIT" and the export time came down to <u>~20 seconds</u>. </font></div><div><font color="#000000" face="arial, helvetica, sans-serif"><br></font></div><div><font color="#000000" face="arial, helvetica, sans-serif">Could there be any issue of changing the flush mode only for the export? Data shouldn't change in the database at this moment and queries shouldn't return stale data.</font></div><div><font color="#000000" face="arial, helvetica, sans-serif"><br></font></div><div><font color="#000000" face="arial, helvetica, sans-serif"><u>Import:</u></font></div><div>For the import it seems that changing the flushMode isn't sufficient. It should likely not be done. However, I found a few places in the keycloak-model-jpa module where entities are created, persisted, then em.flush() and em.detatch() is called right away without the entity being returned (ex: UserAdapter.grantRole() - <a href="https://github.com/keycloak/keycloak/blob/master/model/jpa/src/main/java/org/keycloak/models/jpa/UserAdapter.java">https://github.com/keycloak/keycloak/blob/master/model/jpa/src/main/java/org/keycloak/models/jpa/UserAdapter.java</a>). </div><div><br></div><div>If I remove all those flush/detach calls, the import process goes down to <u>~50 seconds</u>. What is the reason for flushing every time an entity is created rather than letting JPA/Hibernate do it when necessary? </div><div><br></div><div>Thank you,</div><div><br></div><div>Gabriel</div>-- <br><div class="gmail_signature">Gabriel Lavoie<br><a href="mailto:glavoie@gmail.com" target="_blank">glavoie@gmail.com</a></div>
</div>