<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">On 18/02/16 08:07, Stian Thorgersen
wrote:<br>
</div>
<blockquote
cite="mid:CAJgngAeTE3LV2jibPxfF1x3XKO3iGviCGsAU-QBqEUuHd6mxfw@mail.gmail.com"
type="cite">
<div dir="ltr">Having two many joins (fetching everything about a
realm in one query) is probably going to be bad for performance,
especially if there are loads of clients and roles. There can
also be large difference between different vendors.</div>
</blockquote>
Maybe clients and roles (and groups?) shouldn't be loaded eagerly
with realm, but I guess most of the other stuff can be (ie.
federation providers, identity providers, SMTP configurations etc).
Also for clients we can eagerly load most of the stuff (ie.
redirectUris, scopes ). ATM we eagerly preload whole realm in
CachedRealm constructor, but we have fetchType.LAZY everywhere. IMO
this is not optimal and would mean lot of SQL at startup during
realm preload (I did not doublecheck myself).<br>
<br>
Marek<br>
<br>
<blockquote
cite="mid:CAJgngAeTE3LV2jibPxfF1x3XKO3iGviCGsAU-QBqEUuHd6mxfw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div><br>
</div>
<div>Another thing in the future we should separate clients out
into a separate store. There could be thousands of clients or
even more. So they should be treated in a similar fashion to
users. Does that have impact on how we improve/refactor/fix
caching now? </div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On 17 February 2016 at 21:56, Bill
Burke <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:bburke@redhat.com" target="_blank">bburke@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"><span class=""> <br>
<br>
<div>On 2/17/2016 3:09 PM, Marek Posolda wrote:<br>
</div>
<blockquote type="cite">
<div>On 17/02/16 18:11, Bill Burke wrote:<br>
</div>
<blockquote type="cite">
<pre>Currently, adding or deleting a client, or updating a realm causes
invalidation/eviction of the realm and all clients in that realm. To
make matters worse, the next time the realm is accessed, it queries and
loads each client and its relationships. Why do we do this? When a
realm invalidation happens, the cache has no idea if the realm is just
being updated or removed entirely from the DB. With a realm removal, you
also need to evict the cache of all clients within the realm. So, a
cached realm MUST have a list of all clients within it. As a result,
the more clients that get added, keycloak gets slower and slower.
Eventually though the cache stabilizes after inserts/update/deletes
subside and we get back to normal performance, but you can see a nasty
blip for a little bit.</pre>
</blockquote>
We don't need to eagerly preload all clients when
realm is loaded. Infinispan has streaming/predicate
API and we are using it in many places (see
InfinispanUserSessionProvider and all the stuff in
package org.keycloak.models.sessions.infinispan.<span
style="background-color:#e4e4ff">stream</span> ). So
when entry is invalidated, we can have cacheListener,
which will query infinispan to return all cached
clients of the realm and remove them. I can see that
we already have listener. So possibly we can just
change this line to use predicate query: <a
moz-do-not-send="true"
href="https://github.com/keycloak/keycloak/blob/master/model/infinispan/src/main/java/org/keycloak/models/cache/infinispan/locking/LockingCacheRealmProviderFactory.java#L141"
target="_blank"><a class="moz-txt-link-freetext" href="https://github.com/keycloak/keycloak/blob/master/model/infinispan/src/main/java/org/keycloak/models/cache/infinispan/locking/LockingCacheRealmProviderFactory.java#L141">https://github.com/keycloak/keycloak/blob/master/model/infinispan/src/main/java/org/keycloak/models/cache/infinispan/locking/LockingCacheRealmProviderFactory.java#L141</a></a><br>
<br>
<br>
</blockquote>
<br>
</span> Lol, i was sort of kind of doing that doing
cache.values(). This makes it much easier with much less
refactoring.<span class=""><br>
<br>
<blockquote type="cite"> Other option is, that we can
try to optimize eager preload and see if we can reduce
number of SQL queries at startup. For example, it
seems that CachedClient constructor sends SQL query to
preload just realm roles. And then each client sends
another SQL query to preload his client roles (method
CachedClient.cacheRoles ). Couldn't we instead send
one SQL query to load all roles of realm (both realm
and client roles)? Maybe we can even send one big SQL
query to load realm with all it's objects <span><span>
:-) </span></span>Possibly some FetchType.EAGER
in Hibernate entities could help here.<br>
<br>
</blockquote>
<br>
</span> yeah, we should start looking into stuff like
this. We can at least reduce the number of sql calls
for caching a client or realm individually. I'm not sure
how feasible it is though or how much better it is.
Realms and clients contain more than a few one to many
relationships which would make the query result HUGE if I
remember how joins work.<span class=""><br>
<br>
<pre cols="72">--
Bill Burke
JBoss, a division of Red Hat
<a moz-do-not-send="true" href="http://bill.burkecentral.com" target="_blank">http://bill.burkecentral.com</a></pre>
</span></div>
<br>
_______________________________________________<br>
keycloak-dev mailing list<br>
<a moz-do-not-send="true"
href="mailto:keycloak-dev@lists.jboss.org">keycloak-dev@lists.jboss.org</a><br>
<a moz-do-not-send="true"
href="https://lists.jboss.org/mailman/listinfo/keycloak-dev"
rel="noreferrer" target="_blank">https://lists.jboss.org/mailman/listinfo/keycloak-dev</a><br>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</body>
</html>