<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Thanks for reporting this. Could you
please create JIRA and describe your findings? We didn't yet try
to test export/import with bigger number of users, so didn't
spotted this yet.<br>
<br>
I think the only reason for the redundant flush/detach calls is,
that "something" didn't work as expected (maybe even just during
development phase) and flush helped to solve the issue. This
doesn't mean that it's necessarily correct. Now we are close to
the "feature freeze" and we will probably look at performance
tuning. So we will likely look at this issue too.<br>
<br>
Btv. are you using default H2 database or something else? Could
you please mention this in JIRA too? <br>
<br>
Thanks,<br>
Marek<br>
<br>
On 28/01/16 19:22, Gabriel Lavoie wrote:<br>
</div>
<blockquote
cite="mid:CAKp1Nw5av5QwtFb3fBOgYz9YXegkx-1TQQ0m9O-58K+npBUhPg@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>Hi,</div>
<div> I am currently having issues with very slow
export/import of realms with a large number of users (10K).
Both operations take ~10 minutes each. </div>
<div><br>
</div>
<div>After digging in the KeyCloak code, I've found out that a
lot of "flush" is done at the Hibernate/JPA level (at least
4-5 times per user).</div>
<div><br>
</div>
<div><font face="arial, helvetica, sans-serif"><u>Export:</u></font></div>
<div><font face="arial, helvetica, sans-serif">For the export,
there are 4 very slow queries in ExportUtils.exportUser() (<a
moz-do-not-send="true"
href="https://github.com/keycloak/keycloak/blob/7895eb6a3dd7e4abc16dd401066bd08ed8075ffb/services/src/main/java/org/keycloak/exportimport/util/ExportUtils.java"><a class="moz-txt-link-freetext" href="https://github.com/keycloak/keycloak/blob/7895eb6a3dd7e4abc16dd401066bd08ed8075ffb/services/src/main/java/org/keycloak/exportimport/util/ExportUtils.java">https://github.com/keycloak/keycloak/blob/7895eb6a3dd7e4abc16dd401066bd08ed8075ffb/services/src/main/java/org/keycloak/exportimport/util/ExportUtils.java</a></a></font><span
style="font-family:arial,helvetica,sans-serif">):</span></div>
<div><font face="arial, helvetica, sans-serif">*
Set<FederatedIdentityModel> socialLinks =
session.users().getFederatedIdentities(user, realm);</font></div>
<div><font face="arial, helvetica, sans-serif">*
Set<RoleModel> roles = user.getRoleMappings();</font></div>
<div><font face="arial, helvetica, sans-serif">* <span
style="color:rgb(0,0,0)">List<UserConsentModel>
consents = user.getConsents();</span></font></div>
<div><font face="arial, helvetica, sans-serif" color="#000000">* for<span
style="font-weight:bold"> </span>(GroupModel group :
user.getGroups()) {</font></div>
<div><font face="arial, helvetica, sans-serif" color="#000000"><br>
</font></div>
<div><font face="arial, helvetica, sans-serif" color="#000000">They
seem to be caused by Hibernate that forces a flush by
default in "getResultList()". As a workaround I told
Hibernate to flush only on commit of the transaction through
"-Dorg.hibernate.flushMode=COMMIT" and the export time came
down to <u>~20 seconds</u>. </font></div>
<div><font face="arial, helvetica, sans-serif" color="#000000"><br>
</font></div>
<div><font face="arial, helvetica, sans-serif" color="#000000">Could
there be any issue of changing the flush mode only for the
export? Data shouldn't change in the database at this moment
and queries shouldn't return stale data.</font></div>
<div><font face="arial, helvetica, sans-serif" color="#000000"><br>
</font></div>
<div><font face="arial, helvetica, sans-serif" color="#000000"><u>Import:</u></font></div>
<div>For the import it seems that changing the flushMode isn't
sufficient. It should likely not be done. However, I found a
few places in the keycloak-model-jpa module where entities are
created, persisted, then em.flush() and em.detatch() is called
right away without the entity being returned (ex:
UserAdapter.grantRole() - <a moz-do-not-send="true"
href="https://github.com/keycloak/keycloak/blob/master/model/jpa/src/main/java/org/keycloak/models/jpa/UserAdapter.java">https://github.com/keycloak/keycloak/blob/master/model/jpa/src/main/java/org/keycloak/models/jpa/UserAdapter.java</a>). </div>
<div><br>
</div>
<div>If I remove all those flush/detach calls, the import
process goes down to <u>~50 seconds</u>. What is the reason
for flushing every time an entity is created rather than
letting JPA/Hibernate do it when necessary? </div>
<div><br>
</div>
<div>Thank you,</div>
<div><br>
</div>
<div>Gabriel</div>
-- <br>
<div class="gmail_signature">Gabriel Lavoie<br>
<a moz-do-not-send="true" href="mailto:glavoie@gmail.com"
target="_blank">glavoie@gmail.com</a></div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
keycloak-user mailing list
<a class="moz-txt-link-abbreviated" href="mailto:keycloak-user@lists.jboss.org">keycloak-user@lists.jboss.org</a>
<a class="moz-txt-link-freetext" href="https://lists.jboss.org/mailman/listinfo/keycloak-user">https://lists.jboss.org/mailman/listinfo/keycloak-user</a></pre>
</blockquote>
<br>
</body>
</html>