<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Hi,<br>
<br>
what's the keycloak version used? Could you try latest keycloak
and check if performance is still the issue?<br>
<br>
Marek<br>
<br>
On 08/06/16 01:30, Fabricio Milone wrote:<br>
</div>
<blockquote
cite="mid:CAOjtoUMY8Lh2gN9ocYfmKZcCCgH7Vtsrh+Gg03x-7GiWtx-chA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div style="font-size:12.8px">Hi all,</div>
<div style="font-size:12.8px"><br>
</div>
<div style="font-size:12.8px">I sent this email yesterday with 5
or more attachments, so I think it was blocked or something...
here I go again :)</div>
<div style="font-size:12.8px"><br>
</div>
<div style="font-size:12.8px">I've been running load tests on
our application during the last few weeks, and having some
performance issues when my custom federator is enabled.</div>
<div style="font-size:12.8px"><br>
</div>
<div style="font-size:12.8px">The performance issue does not
exist when the federator is disabled.</div>
<div style="font-size:12.8px"> </div>
<div style="font-size:12.8px"><b>Configuration</b>: </div>
<div style="font-size:12.8px"><br>
</div>
<div style="font-size:12.8px">I have a cluster of 2 instances of
Keycloak, with a standalone DB, we've verified the DB isn't an
issue when the federator is disabled. Both instances have a
quad core CPU and they are in the same network. We’ve left the
memory at 512MB. The test script, database and API that
connects to the federator are in separate machines.</div>
<div style="font-size:12.8px"> </div>
<div style="font-size:12.8px"><b>Federator</b>:</div>
<div style="font-size:12.8px"><br>
</div>
<div style="font-size:12.8px">We have a simple custom federator
that makes calls to a very performant api, which has been
tested and is ok. Additionally, we've tested stubbing the API
so the performance is not a problem there. This federator is
using a jaxb marshaller to create a request, again tested in
isolation and is performing well.</div>
<div style="font-size:12.8px"><br>
</div>
<div style="font-size:12.8px">As the federator is doing a lot of
calls to the API (3 per login request), I've implemented a
httpclient that uses a PoolingHttpClientConnectionManager with
1000 connections available to use, instead of using the
standard apache httpclient from http components. That hasn't
improved a bit the performance of the system.</div>
<div style="font-size:12.8px"> </div>
<div style="font-size:12.8px"><b>Tests</b>:</div>
<div style="font-size:12.8px"> </div>
<div style="font-size:12.8px">It is a gatling scala script that
could generate around ~300 (or more) requests/second to the
direct grants login endpoint using random usernames from a
list (all of them already registered using KC). The script is
doing a round robin across both instances of Keycloak with an
even distribution to each KC instance.</div>
<div style="font-size:12.8px"> </div>
<div style="font-size:12.8px">The idea is simulate a load of 300
to 1500 concurrent users trying to login into our systems.</div>
<div style="font-size:12.8px"> </div>
<div style="font-size:12.8px"><b>Problem</b>:</div>
<div style="font-size:12.8px"><br>
</div>
<div style="font-size:12.8px">If I run the tests without using a
federation I can see a very good performance, but when I try
to run the tests with the custom federation code, the
performance drops from ~150 requests/second to 22 req/sec
using both instances.</div>
<div style="font-size:12.8px"> </div>
<div style="font-size:12.8px"> </div>
<div style="font-size:12.8px">Memory wise, it seems to be ok.
I've never seen an error related to memory with this
configuration, also if you take a look at the attached
visualVM screenshot you'll see that memory is not a problem or
it seems not to be.</div>
<div style="font-size:12.8px"> </div>
<div style="font-size:12.8px">CPU utilisation is very low to my
mind, I'd expect more than 80% of usage or something like
that.</div>
<div style="font-size:12.8px"> </div>
<div style="font-size:12.8px">There is a method that is leading
the CPU samples on VisualVM called Semaphore.tryAcquire(). Not
quite sure what's that for, still investigating.</div>
<div style="font-size:12.8px"><br>
</div>
<div style="font-size:12.8px">
<div style="font-size:12.8px">I can see that a lot of new
threads are being created when the test starts, as it
creates around 60requests/second to the direct grants login
call, but it seems to be a bottleneck at some point.</div>
<div style="font-size:12.8px"><br>
</div>
<div style="font-size:12.8px">So I'm wondering if there is
some configuration I'm missing on Keycloak side that could
be affecting the cluster performance when a federator is
enabled. Maybe something related to jpa connections,
infinispan configuration or even wildfly.<br>
</div>
<div style="font-size:12.8px"><br>
</div>
<div style="font-size:12.8px">I'd really appreciate your help
on this one as I'm out of ideas.</div>
<div style="font-size:12.8px"><br>
</div>
<div style="font-size:12.8px">I've attached some screenshots
of visualVM and tests results from my last run today.</div>
<div style="font-size:12.8px"><br>
</div>
<div style="font-size:12.8px"><br>
</div>
<div style="font-size:12.8px">Sorry for the long email and
please let me know if you need further information.</div>
<div style="font-size:12.8px"><br>
</div>
<div style="font-size:12.8px">Thank you in advance,</div>
<div style="font-size:12.8px"><br>
</div>
<div style="font-size:12.8px">Regards,</div>
<div style="font-size:12.8px">Fab</div>
</div>
<div><br>
</div>
-- <br>
<div class="gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">
<div><span
style="font-size:12.8px;font-family:Verdana,Arial,Helvetica,sans-serif"><b><font
color="#000000">Fabricio Milone</font></b></span></div>
<div><span
style="font-size:12.8px;font-family:Verdana,Arial,Helvetica,sans-serif"><font
color="#000000">Developer</font></span></div>
<span
style="font-size:12.8px;font-family:Verdana,Arial,Helvetica,sans-serif"><b><font
color="#009900">
<div><span
style="font-size:12.8px;font-family:Verdana,Arial,Helvetica,sans-serif"><b><font
color="#009900"><br>
</font></b></span></div>
Shine Consulting </font></b></span><span
style="font-size:12.8px;font-family:Verdana,Arial,Helvetica,sans-serif">
<p style="font-size:13.3px;color:rgb(0,153,0);margin:0pt"><span
style="color:rgb(0,0,0)">30/600 Bourke Street</span></p>
<p style="font-size:13.3px;color:rgb(0,153,0);margin:0pt"><span
style="color:rgb(0,0,0)">Melbourne VIC 3000</span></p>
<p style="font-size:13.3px;color:rgb(0,153,0);margin:0pt"><span
style="color:rgb(0,0,0)">T: 03 8488 9939</span></p>
<p style="font-size:13.3px;color:rgb(0,153,0);margin:0pt"><span
style="color:rgb(0,0,0)">M: 04 3200 4006</span></p>
<p style="font-size:13.3px;color:rgb(0,153,0);margin:0pt"><span
style="color:rgb(0,0,0)"><br>
</span></p>
</span><span
style="font-size:13.3px;font-family:Verdana,Arial,Helvetica,sans-serif"><span
style="font-size:13.3px">
<p style="margin:0pt"><a moz-do-not-send="true"
href="http://www.shinetech.com/"
style="color:rgb(51,51,51)" target="_blank">www.shinetech.com</a><font
color="#333333"> </font><i
style="color:rgb(51,51,51)"><b>a</b></i><font
color="#333333"> passion for excellence</font></p>
</span></span></div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
keycloak-user mailing list
<a class="moz-txt-link-abbreviated" href="mailto:keycloak-user@lists.jboss.org">keycloak-user@lists.jboss.org</a>
<a class="moz-txt-link-freetext" href="https://lists.jboss.org/mailman/listinfo/keycloak-user">https://lists.jboss.org/mailman/listinfo/keycloak-user</a></pre>
</blockquote>
<br>
</body>
</html>