<div dir="ltr">Can you create a JIRA for this issue? If you could also make the source for your stub provider available that would be helpful.</div><div class="gmail_extra"><br><div class="gmail_quote">On 13 June 2016 at 12:35, Thomas Connolly <span dir="ltr"><<a href="mailto:thomas_connolly@yahoo.com" target="_blank">thomas_connolly@yahoo.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="color:#000;background-color:#fff;font-family:HelveticaNeue-Light,Helvetica Neue Light,Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-serif;font-size:10px"><div><span>Hi Marek</span></div><span class=""><div><span><br></span></div><div dir="ltr"><span style="font-family:HelveticaNeue,"Helvetica Neue",Helvetica,Arial,"Lucida Grande",sans-serif;font-size:16px">> Thanks, AFAIK we didn't tried much performance testing with federationProviders enabled. It's on todo list though. Also we plan some refactoring of userStorage + userFederation, so we will likely go into it later.</span><br clear="none" style="font-family:HelveticaNeue,"Helvetica Neue",Helvetica,Arial,"Lucida Grande",sans-serif;font-size:16px"><br></div></span><div dir="ltr">Yes and we've found it is a major bottleneck in our system testing (using stub to remove internal back end dependencies). </div><div dir="ltr">Can you suggest any short term measures to improve performance, we're blocked from pushing this to production at the moment?</div><div dir="ltr">This is a major feature of the system I'm guessing this affects the LDAP / AD integration / federator performance too.</div><div dir="ltr">Do you have any timeframe around the priority to address this?</div><span class=""><div dir="ltr"><br clear="none"><span style="font-family:HelveticaNeue,"Helvetica Neue",Helvetica,Arial,"Lucida Grande",sans-serif;font-size:16px">> For your case, the performance bottleneck can be in your federationProvider implementation, so I am not sure if it's the issue in Keycloak or rather issue in your implementation. </span><br clear="none" style="font-family:HelveticaNeue,"Helvetica Neue",Helvetica,Arial,"Lucida Grande",sans-serif;font-size:16px"><br clear="none" style="font-family:HelveticaNeue,"Helvetica Neue",Helvetica,Arial,"Lucida Grande",sans-serif;font-size:16px"></div></span><div>As indicated we've created a stub implementation, code included below, to demonstrate there is an issue calling a federator in KC.</div><div><br></div><div>/** Code snippet **/</div>
<div><span>public</span> <span>class</span> StubFederationProvider <span>implements</span> UserFederationProvider {</div>
<div><span> </span></div>
<div> <span>private</span> <span>static</span> <span>final</span> Logger <span>logger</span> = Logger.getLogger(StubFederationProvider.<span>class</span>);</div>
<div> <span>protected</span> KeycloakSession <span>session</span>;</div>
<div> <span>protected</span> UserFederationProviderModel <span>model</span>;</div>
<div><br></div>
<div><span> </span><span>public</span> StubFederationProvider(KeycloakSession <span>session</span>, UserFederationProviderModel <span>model</span>){</div>
<div> <span>this</span>.<span>session</span> = <span>session</span>;</div>
<div> <span>this</span>.<span>model</span> = <span>model</span>;</div>
<div> }</div>
<div><br></div>
<div> <span>public</span> UserFederationProviderModel getModel() {</div>
<div> <span>return</span> <span>model</span>;</div>
<div> }</div>
<div><br></div>
<div><span> </span>@Override</div>
<div><span> </span><span>public</span> UserModel getUserByUsername(RealmModel <span>realm</span>, String <span>username</span>) {</div>
<div><br></div>
<div><span> </span><span> </span>UserModel <span>userModel</span> = addUserModelToUserStorage(<span>realm</span>, <span>username</span>);</div>
<div><span> </span><span> </span><span>userModel</span>.setEnabled(<span>true</span>);</div>
<div><span> </span><span> </span><span>userModel</span>.setFederationLink(<span>model</span>.getId());</div>
<div><br></div>
<div><span><span> </span><span> </span></span><span>return</span><span> </span>userModel<span>;</span></div>
<div><span> </span>}</div>
<div><br></div>
<div><span> </span><span>protected</span> UserModel addUserModelToUserStorage(RealmModel <span>realm</span>, String <span>username</span>) {</div>
<div><span> </span><span> </span><span>return</span> <span>session</span>.userStorage().addUser(<span>realm</span>, <span>username</span>);</div>
<div><span> </span>}</div>
<div><br></div>
<div><span> </span>@Override</div>
<div> <span>public</span> UserModel getUserByEmail(RealmModel <span>realm</span>, String <span>email</span>) {</div>
<div> <span>return</span> <span>null</span>;</div>
<div> }</div>
<div><br></div>
<div><span> </span>@Override</div>
<div> <span>public</span> List<UserModel> searchByAttributes(Map<String, String> <span>attributes</span>, RealmModel <span>realm</span>, <span>int</span> <span>maxResults</span>) {</div>
<div> <span>return</span> Collections.emptyList();</div>
<div> }</div>
<div><br></div>
<div><span> </span>@Override</div>
<div><span> </span><span>public</span> List<UserModel> getGroupMembers(RealmModel <span>realm</span>, GroupModel <span>group</span>, <span>int</span> <span>firstResult</span>, <span>int</span> <span>maxResults</span>) {</div>
<div><span><span> </span><span> </span></span>return<span> </span>null<span>;</span></div>
<div><span> </span>}</div>
<div><br></div>
<div><span> </span>@Override</div>
<div> <span>public</span> <span>void</span> preRemove(RealmModel <span>realm</span>) {</div>
<div><span> </span>// complete We don't care about the realm being removed</div>
<div> }</div>
<div><br></div>
<div><span> </span>@Override</div>
<div> <span>public</span> <span>void</span> preRemove(RealmModel <span>realm</span>, RoleModel <span>role</span>) {</div>
<div><span> </span>// complete we dont'care if a role is removed</div>
<div><br></div>
<div> }</div>
<div><br></div>
<div><span> </span>@Override</div>
<div><span> </span><span>public</span> <span>void</span> preRemove(RealmModel <span>realmModel</span>, GroupModel <span>groupModel</span>) {</div>
<div><span><span> </span><span> </span></span>// complete we dont'care if a role is removed</div>
<div><span> </span>}</div>
<div><br></div>
<div>@Override<br></div>
<div><span> </span><span>public</span> <span>boolean</span> isValid(RealmModel <span>realm</span>, UserModel <span>local</span>) {</div>
<div><br></div>
<div><span> </span><span> </span><span>return</span> userExists(<span>local</span>.getUsername());</div>
<div><span> </span>}</div>
<div><span> </span></div>
<div> <span>/**</span></div>
<div> * Returns supported credentials by this <span>federator</span>. PASSWORD is always supported but TOTP is optional for each user.</div>
<div> *</div>
<div> * <span>@param</span> user</div>
<div> * <span>@return</span></div>
<div> */</div>
<div><span> </span>@Override</div>
<div> <span>public</span> Set<String> getSupportedCredentialTypes(UserModel <span>user</span>) {</div>
<div><span> </span><span> </span>Set<String> <span>supportedCredentialTypes</span> = <span>new</span> HashSet<>();</div>
<div><br></div>
<div><span> </span><span> </span><span>supportedCredentialTypes</span>.add(UserCredentialModel.<span>PASSWORD</span>);</div>
<div><br></div>
<div><span><span> </span><span> </span></span>// check for any <span>otp</span> configured on this user</div>
<div><span> </span><span> </span><span>if</span> (<span>user</span>.isOtpEnabled()) {</div>
<div><span> </span><span> </span><span> </span><span>supportedCredentialTypes</span>.add(UserCredentialModel.<span>TOTP</span>);</div>
<div><span> </span><span> </span><span> </span><span>supportedCredentialTypes</span>.add(UserCredentialModel.<span>HOTP</span>);</div>
<div><span> </span><span> </span>}</div>
<div><br></div>
<div><span> </span><span>return</span><span> </span>supportedCredentialTypes<span>;</span></div>
<div> }</div>
<div><br></div>
<div><span> </span>@Override</div>
<div> <span>public</span> <span>boolean</span> validCredentials(RealmModel <span>realm</span>, UserModel <span>user</span>, List<UserCredentialModel> <span>input</span>) {</div>
<div> <span>for</span> (UserCredentialModel <span>cred</span> : <span>input</span>) {</div>
<div> <span>if</span> (<span>cred</span>.getType().equals(UserCredentialModel.<span>PASSWORD</span>)) {</div>
<div> <span> </span><span>return</span> validate(<span>user</span>, <span>cred</span>.getValue());</div>
<div> } <span>else</span> <span>if</span> (<span>cred</span>.getType().equals(UserCredentialModel.<span>TOTP</span>)) {</div>
<div><span> </span><span> </span><span> </span><span> </span><span>return</span> CredentialValidation.validTOTP(<span>realm</span>, <span>user</span>, <span>cred</span>.getValue());</div>
<div><span> </span><span> </span><span> </span>} <span>else</span> <span>if</span> (<span>cred</span>.getType().equals(UserCredentialModel.<span>HOTP</span>)) {</div>
<div><span> </span><span> </span><span> </span><span> </span><span>return</span> CredentialValidation.validHOTP(<span>realm</span>, <span>user</span>, <span>cred</span>.getValue());</div>
<div><span> </span><span> </span><span> </span>}</div>
<div> }</div>
<div><span> </span>return<span> </span>false<span>;</span></div>
<div> }</div>
<div><br></div>
<div><span> </span>@Override</div>
<div> <span>public</span> <span>boolean</span> validCredentials(RealmModel <span>realm</span>, UserModel <span>user</span>, UserCredentialModel... <span>input</span>) {</div>
<div> <span>for</span> (UserCredentialModel <span>cred</span> : <span>input</span>) {</div>
<div> <span>if</span> (<span>cred</span>.getType().equals(UserCredentialModel.<span>PASSWORD</span>)) {</div>
<div> <span> </span><span>return</span> validate(<span>user</span>, <span>cred</span>.getValue());</div>
<div> } <span>else</span> <span>if</span> (<span>cred</span>.getType().equals(UserCredentialModel.<span>TOTP</span>)) {</div>
<div><span> </span><span> </span><span> </span><span> </span><span>return</span> CredentialValidation.validTOTP(<span>realm</span>, <span>user</span>, <span>cred</span>.getValue());</div>
<div><span> </span><span> </span><span> </span>} <span>else</span> <span>if</span> (<span>cred</span>.getType().equals(UserCredentialModel.<span>HOTP</span>)) {</div>
<div><span> </span><span> </span><span> </span><span> </span><span>return</span> CredentialValidation.validHOTP(<span>realm</span>, <span>user</span>, <span>cred</span>.getValue());</div>
<div><span> </span><span> </span><span> </span>}</div>
<div> }</div>
<div><span> </span>return<span> </span>false<span>;</span></div>
<div> }</div>
<div><br></div>
<div><span> </span>@Override</div>
<div> <span>public</span> <span>void</span> close() {</div>
<div><br></div>
<div> }</div>
<div> </div>
<div> <span>/**</span></div>
<div> * <span>Keycloak</span> will call this method if it finds an imported UserModel. Here we <span>proxy</span> the UserModel with</div>
<div> * a <span>Readonly</span> <span>proxy</span> which will <span>barf</span> if password is updated.</div>
<div> *</div>
<div> * <span>@param</span> local</div>
<div> * <span>@return</span></div>
<div> */</div>
<div><span> </span>@Override</div>
<div><span> </span><span>public</span> UserModel validateAndProxy(RealmModel <span>realm</span>, UserModel <span>local</span>) {</div>
<div><span> </span><span> </span><span>if</span> (isValid(<span>realm</span>, <span>local</span>)) {</div>
<div><span> </span><span> </span><span> </span>getUserDetails(<span>local</span>);</div>
<div><span> </span><span> </span><span> </span><span>return</span> <span>new</span> StubUserModelProxy(<span>local</span>, <span>this</span>);</div>
<div><span> </span><span> </span>} <span>else</span> {</div>
<div><span><span> </span><span> </span><span> </span></span>return<span> </span>null<span>;</span></div>
<div><span> </span><span> </span>}</div>
<div> }</div>
<div><br></div>
<div><span> </span>@Override</div>
<div> <span>public</span> <span>boolean</span> synchronizeRegistrations() {</div>
<div> <span>return</span> <span>true</span>;</div>
<div> }</div>
<div><br></div>
<div> <span>/**</span></div>
<div> * Called if this federation provider has priority and supports synchronized registrations.</div>
<div> *</div>
<div> * <span>@param</span> realm</div>
<div> * <span>@param</span> user</div>
<div> * <span>@return</span></div>
<div> */</div>
<div><span> </span>@Override</div>
<div> <span>public</span> UserModel register(RealmModel <span>realm</span>, UserModel <span>user</span>) {</div>
<div><br></div>
<div><span>user</span>.setSingleAttribute(<span>"status"</span>, <span>"OK"</span>);</div>
<div><span><span> </span><span> </span></span>return<span> </span><span>user</span><span>;</span></div>
<div> }</div>
<div><br></div>
<div><span> </span>@Override</div>
<div> <span>public</span> <span>boolean</span> removeUser(RealmModel <span>realm</span>, UserModel <span>user</span>) {</div>
<div><span><span> </span><span> </span></span>// Not supported. Used as a part of the Workaround to <a href="https://issues.jboss.org/browse/KEYCLOAK-1075" target="_blank">https://issues.jboss.org/browse/KEYCLOAK-1075</a></div>
<div><span><span> </span><span> </span></span>return<span> </span>true<span>;</span></div>
<div> }</div>
<div><br></div>
<div><span> </span>/**</div>
<div><span> </span> * Supported credentials by this <span>federator</span>. PASSWORD is a supported type. TOTP depends on the user.</div>
<div><span> </span> *</div>
<div><span> </span> * <span>@return</span> supportedCredentialTypes</div>
<div><span> </span> */</div>
<div><span> </span>@Override</div>
<div><span> </span><span>public</span> Set<String> getSupportedCredentialTypes() {</div>
<div><span> </span><span> </span>Set<String> <span>supportedCredentialTypes</span> = <span>new</span> HashSet<>();</div>
<div><span> </span><span> </span><span>supportedCredentialTypes</span>.add(UserCredentialModel.<span>PASSWORD</span>);</div>
<div><span> </span><span> </span><span>supportedCredentialTypes</span>.add(UserCredentialModel.<span>TOTP</span>);</div>
<div><span> </span><span> </span><span>supportedCredentialTypes</span>.add(UserCredentialModel.<span>HOTP</span>);</div>
<div><span><span> </span><span> </span></span><span>return</span><span> </span>supportedCredentialTypes<span>;</span></div>
<div><span> </span>}</div>
<div><br></div>
<div><span> </span>@Override</div>
<div><span> </span><span>public</span> CredentialValidationOutput validCredentials(RealmModel <span>realm</span>, UserCredentialModel <span>credential</span>) {</div>
<div> <span>throw</span> <span>new</span> IllegalStateException(<span>"validCredentials not supported"</span>);</div>
<div><span> </span>}</div>
<div><br></div>
<div><span> </span><span>private</span> <span>boolean</span> userExists(String <span>username</span>)<span> </span>{</div>
<div><span> <span> </span></span>return<span> </span>true<span>;</span></div>
<div> }</div>
<div><span> </span></div>
<div><span> </span><span>private</span> <span>void</span> getUserDetails(UserModel <span>user</span>) {</div>
<div><span> </span><span> </span><span>user</span>.setFirstName(<span>"first name"</span>);</div>
<div><span> </span><span> </span><span>user</span>.setLastName(<span>"last name"</span>);</div>
<div><span> </span>}</div>
<div><br></div>
<div><span> </span><span>public</span> <span>boolean</span> validate(UserModel <span>user</span>, String <span>password</span>) {</div>
<div><span><span> </span><span> </span></span>return<span> </span>true<span>;</span></div>
<div><span> </span>}</div>
<div><br></div>
<div dir="ltr">}</div><div dir="ltr"><br></div><div dir="ltr">/** End Snippet **/ <br></div><div><br></div><div>Regards
Tom Connolly.</div><div><br><div class="hm HOEnZb"><br></div></div><div style="display:block"><div class="hm HOEnZb"> </div><div style="font-family:HelveticaNeue-Light,Helvetica Neue Light,Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-serif;font-size:10px"><div class="hm HOEnZb"> </div><div style="font-family:HelveticaNeue,Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-serif;font-size:16px"><div class="hm HOEnZb"> </div><div dir="ltr"><div class="hm HOEnZb"> </div><font size="2" face="Arial"><div class="hm HOEnZb"> <hr size="1"> <b><span style="font-weight:bold">From:</span></b> Marek Posolda <<a href="mailto:mposolda@redhat.com" target="_blank">mposolda@redhat.com</a>><br> <b><span style="font-weight:bold">To:</span></b> Thomas Connolly <<a href="mailto:thomas_connolly@yahoo.com" target="_blank">thomas_connolly@yahoo.com</a>>; "<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a>" <<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a>> <br> <b><span style="font-weight:bold">Sent:</span></b> Monday, June 13, 2016 5:47 PM</div><div><div class="h5"><br> <b><span style="font-weight:bold">Subject:</span></b> Re: [keycloak-user] Performance issues with Federation provider enabled<br> </div></div></font> </div><div><div class="h5"> <div><br><div><div>
<div>Thanks, AFAIK we didn't tried much
performance testing with federationProviders enabled. It's on todo
list though. Also we plan some refactoring of userStorage +
userFederation, so we will likely go into it later.<br clear="none">
<br clear="none">
For your case, the performance bottleneck can be in your
federationProvider implementation, so I am not sure if it's the
issue in Keycloak or rather issue in your implementation. <br clear="none">
<br clear="none">
One thing to note (maybe it's not an issue in your case, but just
adding it to be sure you're aware): UserFederationProvider.close
is currently not called. So if you are rely on this method to free
any important resources related to your implementation, you
shouldn't as it doesn't work right now. We are working on
improving this for next version.<br clear="none">
<br clear="none">
Marek<br clear="none">
<br clear="none">
On 13/06/16 07:57, Thomas Connolly wrote:<br clear="none">
</div>
<blockquote type="cite">
<div><div style="color:#000;background-color:#fff;font-family:HelveticaNeue-Light,Helvetica Neue Light,Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-serif;font-size:10px">Hi Marek<br clear="none">
<br clear="none">
I'm working with Fabricio on the federation performance issues
with Keycloak.<br clear="none">
<br clear="none">
In answer to your question we are using the latest KC 1.9.7
version (we upgraded this week from 1.9.2).<br clear="none">
<br clear="none">
To give you some indication of the running a gatling direct
access login test (results below).<br clear="none">
<br clear="none">
As you can see below in (1) using KC out of the box. Great
performance - we saw 110 tx per sec on a 4 core system.<br clear="none">
<div>In scenario (2)
using a stubbed federator (simply an echo plugin not
connecting to any back end services), performance is
unacceptable.</div>
<br clear="none">
1) Not using the federator - Stub federator (disabled) - while
29 tx per second we could easily get to a stable 110 tx per
second.<br clear="none">
300 Users (hitting single server)<br clear="none">
---- Global Information
--------------------------------------------------------<br clear="none">
> request count
9185 (OK=9185 KO=0 )<br clear="none">
> min response time
18 (OK=18 KO=- )<br clear="none">
> max response time
723 (OK=723 KO=- )<br clear="none">
> mean response time
27 (OK=27 KO=- )<br clear="none">
> std deviation
44 (OK=44 KO=- )<br clear="none">
> response time 50th percentile
20 (OK=20 KO=- )<br clear="none">
> response time 75th percentile
21 (OK=21 KO=- )<br clear="none">
> mean requests/sec
29.626 (OK=29.626 KO=- )<br clear="none">
---- Response Time Distribution
------------------------------------------------<br clear="none">
> t < 800 ms
9185 (100%)<br clear="none">
> 800 ms < t < 1200
ms 0 ( 0%)<br clear="none">
> t > 1200
ms 0 ( 0%)<br clear="none">
> failed
0 ( 0%)<br clear="none">
<br clear="none">
2) Stub federator (enabled)- if we brought test down to 12 tx
per second (about 90 users) the response times dropped to <
1200 ms response times, however not even close to meeting out
acceptance creteria.<br clear="none">
300 Users (hitting single server) <br clear="none">
---- Global Information
--------------------------------------------------------<br clear="none">
> request count
8496 (OK=8496 KO=0 )<br clear="none">
> min response time
511 (OK=511 KO=- )<br clear="none">
> max response time
11191 (OK=11191 KO=- )<br clear="none">
> mean response time
6832 (OK=6832 KO=- )<br clear="none">
> std deviation
2329 (OK=2329 KO=- )<br clear="none">
> response time 50th percentile
7194 (OK=7194 KO=- )<br clear="none">
> response time 75th percentile
8690 (OK=8690 KO=- )<br clear="none">
> mean requests/sec
27.404 (OK=27.404 KO=- )<br clear="none">
---- Response Time Distribution
------------------------------------------------<br clear="none">
> t < 800 ms
154 ( 2%)<br clear="none">
> 800 ms < t < 1200
ms 85 ( 1%)<br clear="none">
> t > 1200 ms
8257 ( 97%)<br clear="none">
> failed
0 ( 0%)<br clear="none">
<br clear="none">
This is currently a show stopper for us and is blocking our path
to production.<br clear="none">
Do you run similar tests and how can we help you optimise the
performance?<br clear="none">
<br clear="none">
Regards<br clear="none">
Tom.<br clear="none">
<br clear="none">
<br clear="none">
Date: Wed, 8 Jun 2016 12:28:19 +0200<br clear="none">
From: Marek Posolda <a rel="nofollow" shape="rect" href="mailto:mposolda@redhat.com" target="_blank"><mposolda@redhat.com></a><br clear="none">
Subject: Re: [keycloak-user] Performance issues with Federation<br clear="none">
provider enabled<br clear="none">
To: Fabricio Milone <a rel="nofollow" shape="rect" href="mailto:fabricio.milone@shinetech.com" target="_blank"><fabricio.milone@shinetech.com></a>,
keycloak-user<br clear="none">
<a rel="nofollow" shape="rect" href="mailto:keycloak-user@lists.jboss.org" target="_blank"><keycloak-user@lists.jboss.org></a><br clear="none">
Message-ID: <a rel="nofollow" shape="rect" href="mailto:5757F343.1040803@redhat.com" target="_blank"><5757F343.1040803@redhat.com></a><br clear="none">
Content-Type: text/plain; charset="windows-1252"<br clear="none">
<br clear="none">
Hi,<br clear="none">
<br clear="none">
what's the keycloak version used? Could you try latest keycloak
and <br clear="none">
check if performance is still the issue?<br clear="none">
<br clear="none">
Marek<br clear="none">
<br clear="none">
On 08/06/16 01:30, Fabricio Milone wrote:<br clear="none">
> Hi all,<br clear="none">
><br clear="none">
> I sent this email yesterday with 5 or more attachments, so
I think it <br clear="none">
> was blocked or something... here I go again :)<br clear="none">
><br clear="none">
> I've been running load tests on our application during the
last few <br clear="none">
> weeks, and having some performance issues when my custom
federator is <br clear="none">
> enabled.<br clear="none">
><br clear="none">
> The performance issue does not exist when the federator is
disabled.<br clear="none">
> *Configuration*:<br clear="none">
><br clear="none">
> I have a cluster of 2 instances of Keycloak, with a
standalone DB, <br clear="none">
> we've verified the DB isn't an issue when the federator is
disabled. <br clear="none">
> Both instances have a quad core CPU and they are in the
same network. <br clear="none">
> We?ve left the memory at 512MB. The test script, database
and API that <br clear="none">
> connects to the federator are in separate machines.<br clear="none">
> *Federator*:<br clear="none">
><br clear="none">
> We have a simple custom federator that makes calls to a
very <br clear="none">
> performant api, which has been tested and is ok.
Additionally, we've <br clear="none">
> tested stubbing the API so the performance is not a problem
there. <br clear="none">
> This federator is using a jaxb marshaller to create a
request, again <br clear="none">
> tested in isolation and is performing well.<br clear="none">
><br clear="none">
> As the federator is doing a lot of calls to the API (3 per
login <br clear="none">
> request), I've implemented a httpclient that uses a <br clear="none">
> PoolingHttpClientConnectionManager with 1000 connections
available to <br clear="none">
> use, instead of using the standard apache httpclient from
http <br clear="none">
> components. That hasn't improved a bit the performance of
the system.<br clear="none">
> *Tests*:<br clear="none">
> It is a gatling scala script that could generate around
~300 (or more) <br clear="none">
> requests/second to the direct grants login endpoint using
random <br clear="none">
> usernames from a list (all of them already registered using
KC). The <br clear="none">
> script is doing a round robin across both instances of
Keycloak with <br clear="none">
> an even distribution to each KC instance.<br clear="none">
> The idea is simulate a load of 300 to 1500 concurrent users
trying to <br clear="none">
> login into our systems.<br clear="none">
> *Problem*:<br clear="none">
><br clear="none">
> If I run the tests without using a federation I can see a
very good <br clear="none">
> performance, but when I try to run the tests with the
custom <br clear="none">
> federation code, the performance drops from ~150
requests/second to 22 <br clear="none">
> req/sec using both instances.<br clear="none">
> Memory wise, it seems to be ok. I've never seen an error
related to <br clear="none">
> memory with this configuration, also if you take a look at
the <br clear="none">
> attached visualVM screenshot you'll see that memory is not
a problem <br clear="none">
> or it seems not to be.<br clear="none">
> CPU utilisation is very low to my mind, I'd expect more
than 80% of <br clear="none">
> usage or something like that.<br clear="none">
> There is a method that is leading the CPU samples on
VisualVM called <br clear="none">
> Semaphore.tryAcquire(). Not quite sure what's that for,
still <br clear="none">
> investigating.<br clear="none">
><br clear="none">
> I can see that a lot of new threads are being created when
the test <br clear="none">
> starts, as it creates around 60requests/second to the
direct grants <br clear="none">
> login call, but it seems to be a bottleneck at some point.<br clear="none">
><br clear="none">
> So I'm wondering if there is some configuration I'm missing
on <br clear="none">
> Keycloak side that could be affecting the cluster
performance when a <br clear="none">
> federator is enabled. Maybe something related to jpa
connections, <br clear="none">
> infinispan configuration or even wildfly.<br clear="none">
><br clear="none">
> I'd really appreciate your help on this one as I'm out of
ideas.<br clear="none">
><br clear="none">
> I've attached some screenshots of visualVM and tests
results from my <br clear="none">
> last run today.<br clear="none">
><br clear="none">
><br clear="none">
> Sorry for the long email and please let me know if you need
further <br clear="none">
> information.<br clear="none">
><br clear="none">
> Thank you in advance,<br clear="none">
><br clear="none">
> Regards,<br clear="none">
> Fab<br clear="none">
><br clear="none">
> -- <br clear="none">
> *Fabricio Milone*<br clear="none">
<div dir="ltr">> Developer</div>
</div></div>
<br clear="none">
<fieldset></fieldset>
<br clear="none">
<pre>_______________________________________________
keycloak-user mailing list
<a rel="nofollow" shape="rect" href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a>
<a rel="nofollow" shape="rect" href="https://lists.jboss.org/mailman/listinfo/keycloak-user" target="_blank">https://lists.jboss.org/mailman/listinfo/keycloak-user</a></pre>
</blockquote>
<br clear="none">
</div></div><br><br></div> </div></div></div> </div> </div></div></div><br>_______________________________________________<br>
keycloak-user mailing list<br>
<a href="mailto:keycloak-user@lists.jboss.org">keycloak-user@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/keycloak-user" rel="noreferrer" target="_blank">https://lists.jboss.org/mailman/listinfo/keycloak-user</a><br></blockquote></div><br></div>