<html>
  <head>
    <meta content="text/html; charset=windows-1252"
      http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <div class="moz-cite-prefix">Thanks, AFAIK we didn't tried much
      performance testing with federationProviders enabled. It's on todo
      list though. Also we plan some refactoring of userStorage +
      userFederation, so we will likely go into it later.<br>
      <br>
      For your case, the performance bottleneck can be in your
      federationProvider implementation, so I am not sure if it's the
      issue in Keycloak or rather issue in your implementation. <br>
      <br>
      One thing to note (maybe it's not an issue in your case, but just
      adding it to be sure you're aware): UserFederationProvider.close
      is currently not called. So if you are rely on this method to free
      any important resources related to your implementation, you
      shouldn't as it doesn't work right now. We are working on
      improving this for next version.<br>
      <br>
      Marek<br>
      <br>
      On 13/06/16 07:57, Thomas Connolly wrote:<br>
    </div>
    <blockquote
      cite="mid:315000034.1945497.1465797441058.JavaMail.yahoo@mail.yahoo.com"
      type="cite">
      <div style="color:#000; background-color:#fff;
        font-family:HelveticaNeue-Light, Helvetica Neue Light, Helvetica
        Neue, Helvetica, Arial, Lucida Grande,
        sans-serif;font-size:10px">Hi Marek<br
          id="yui_3_16_0_ym19_1_1465797102861_4190">
        <br id="yui_3_16_0_ym19_1_1465797102861_4191">
        I'm working with Fabricio on the federation performance issues
        with Keycloak.<br id="yui_3_16_0_ym19_1_1465797102861_4192">
        <br id="yui_3_16_0_ym19_1_1465797102861_4193">
        In answer to your question we are using the latest KC 1.9.7
        version (we upgraded this week from 1.9.2).<br
          id="yui_3_16_0_ym19_1_1465797102861_4194">
        <br id="yui_3_16_0_ym19_1_1465797102861_4195">
        To give you some indication of the running a gatling direct
        access login test (results below).<br
          id="yui_3_16_0_ym19_1_1465797102861_4196">
        <br id="yui_3_16_0_ym19_1_1465797102861_4197">
        As you can see below in (1) using KC out of the box. Great
        performance - we saw 110 tx per sec on a 4 core system.<br
          id="yui_3_16_0_ym19_1_1465797102861_4198">
        <div id="yui_3_16_0_ym19_1_1465797102861_4449">In scenario (2)
          using a stubbed federator (simply an echo plugin not
          connecting to any back end services), performance is
          unacceptable.</div>
        <br id="yui_3_16_0_ym19_1_1465797102861_4200">
        1) Not using the federator - Stub federator (disabled) - while
        29 tx per second we could easily get to a stable 110 tx per
        second.<br id="yui_3_16_0_ym19_1_1465797102861_4201">
            300 Users (hitting single server)<br
          id="yui_3_16_0_ym19_1_1465797102861_4202">
            ---- Global Information
        --------------------------------------------------------<br
          id="yui_3_16_0_ym19_1_1465797102861_4203">
            &gt; request count                                      
        9185 (OK=9185   KO=0     )<br
          id="yui_3_16_0_ym19_1_1465797102861_4204">
            &gt; min response time                                    
        18 (OK=18     KO=-     )<br
          id="yui_3_16_0_ym19_1_1465797102861_4205">
            &gt; max response time                                   
        723 (OK=723    KO=-     )<br
          id="yui_3_16_0_ym19_1_1465797102861_4206">
            &gt; mean response time                                   
        27 (OK=27     KO=-     )<br
          id="yui_3_16_0_ym19_1_1465797102861_4207">
            &gt; std deviation                                        
        44 (OK=44     KO=-     )<br
          id="yui_3_16_0_ym19_1_1465797102861_4208">
            &gt; response time 50th percentile                        
        20 (OK=20     KO=-     )<br
          id="yui_3_16_0_ym19_1_1465797102861_4209">
            &gt; response time 75th percentile                        
        21 (OK=21     KO=-     )<br
          id="yui_3_16_0_ym19_1_1465797102861_4210">
            &gt; mean requests/sec                                
        29.626 (OK=29.626 KO=-     )<br
          id="yui_3_16_0_ym19_1_1465797102861_4211">
            ---- Response Time Distribution
        ------------------------------------------------<br
          id="yui_3_16_0_ym19_1_1465797102861_4212">
            &gt; t &lt; 800 ms                                         
        9185 (100%)<br id="yui_3_16_0_ym19_1_1465797102861_4213">
            &gt; 800 ms &lt; t &lt; 1200
        ms                                   0 (  0%)<br
          id="yui_3_16_0_ym19_1_1465797102861_4214">
            &gt; t &gt; 1200
        ms                                            0 (  0%)<br
          id="yui_3_16_0_ym19_1_1465797102861_4215">
            &gt; failed                                                
        0 (  0%)<br id="yui_3_16_0_ym19_1_1465797102861_4216">
        <br id="yui_3_16_0_ym19_1_1465797102861_4217">
        2) Stub federator (enabled)- if we brought test down to 12 tx
        per second (about 90 users) the response times dropped to &lt;
        1200 ms response times, however not even close to meeting out
        acceptance creteria.<br
          id="yui_3_16_0_ym19_1_1465797102861_4218">
            300 Users (hitting single server) <br
          id="yui_3_16_0_ym19_1_1465797102861_4219">
            ---- Global Information
        --------------------------------------------------------<br
          id="yui_3_16_0_ym19_1_1465797102861_4220">
            &gt; request count                                      
        8496 (OK=8496   KO=0     )<br
          id="yui_3_16_0_ym19_1_1465797102861_4221">
            &gt; min response time                                   
        511 (OK=511    KO=-     )<br
          id="yui_3_16_0_ym19_1_1465797102861_4222">
            &gt; max response time                                 
        11191 (OK=11191  KO=-     )<br
          id="yui_3_16_0_ym19_1_1465797102861_4223">
            &gt; mean response time                                 
        6832 (OK=6832   KO=-     )<br
          id="yui_3_16_0_ym19_1_1465797102861_4224">
            &gt; std deviation                                      
        2329 (OK=2329   KO=-     )<br
          id="yui_3_16_0_ym19_1_1465797102861_4225">
            &gt; response time 50th percentile                      
        7194 (OK=7194   KO=-     )<br
          id="yui_3_16_0_ym19_1_1465797102861_4226">
            &gt; response time 75th percentile                      
        8690 (OK=8690   KO=-     )<br
          id="yui_3_16_0_ym19_1_1465797102861_4227">
            &gt; mean requests/sec                                
        27.404 (OK=27.404 KO=-     )<br
          id="yui_3_16_0_ym19_1_1465797102861_4228">
            ---- Response Time Distribution
        ------------------------------------------------<br
          id="yui_3_16_0_ym19_1_1465797102861_4229">
            &gt; t &lt; 800 ms                                          
        154 (  2%)<br id="yui_3_16_0_ym19_1_1465797102861_4230">
            &gt; 800 ms &lt; t &lt; 1200
        ms                                  85 (  1%)<br
          id="yui_3_16_0_ym19_1_1465797102861_4231">
            &gt; t &gt; 1200 ms                                        
        8257 ( 97%)<br id="yui_3_16_0_ym19_1_1465797102861_4232">
            &gt; failed                                                
        0 (  0%)<br id="yui_3_16_0_ym19_1_1465797102861_4233">
        <br id="yui_3_16_0_ym19_1_1465797102861_4234">
        This is currently a show stopper for us and is blocking our path
        to production.<br id="yui_3_16_0_ym19_1_1465797102861_4235">
        Do you run similar tests and how can we help you optimise the
        performance?<br id="yui_3_16_0_ym19_1_1465797102861_4236">
        <br id="yui_3_16_0_ym19_1_1465797102861_4237">
        Regards<br id="yui_3_16_0_ym19_1_1465797102861_4238">
        Tom.<br id="yui_3_16_0_ym19_1_1465797102861_4239">
        <br id="yui_3_16_0_ym19_1_1465797102861_4240">
        <br id="yui_3_16_0_ym19_1_1465797102861_4241">
        Date: Wed, 8 Jun 2016 12:28:19 +0200<br
          id="yui_3_16_0_ym19_1_1465797102861_4242">
        From: Marek Posolda <a class="moz-txt-link-rfc2396E" href="mailto:mposolda@redhat.com">&lt;mposolda@redhat.com&gt;</a><br
          id="yui_3_16_0_ym19_1_1465797102861_4243">
        Subject: Re: [keycloak-user] Performance issues with Federation<br
          id="yui_3_16_0_ym19_1_1465797102861_4244">
            provider enabled<br
          id="yui_3_16_0_ym19_1_1465797102861_4245">
        To: Fabricio Milone <a class="moz-txt-link-rfc2396E" href="mailto:fabricio.milone@shinetech.com">&lt;fabricio.milone@shinetech.com&gt;</a>,   
        keycloak-user<br id="yui_3_16_0_ym19_1_1465797102861_4246">
            <a class="moz-txt-link-rfc2396E" href="mailto:keycloak-user@lists.jboss.org">&lt;keycloak-user@lists.jboss.org&gt;</a><br
          id="yui_3_16_0_ym19_1_1465797102861_4247">
        Message-ID: <a class="moz-txt-link-rfc2396E" href="mailto:5757F343.1040803@redhat.com">&lt;5757F343.1040803@redhat.com&gt;</a><br
          id="yui_3_16_0_ym19_1_1465797102861_4248">
        Content-Type: text/plain; charset="windows-1252"<br
          id="yui_3_16_0_ym19_1_1465797102861_4249">
        <br id="yui_3_16_0_ym19_1_1465797102861_4250">
        Hi,<br id="yui_3_16_0_ym19_1_1465797102861_4251">
        <br id="yui_3_16_0_ym19_1_1465797102861_4252">
        what's the keycloak version used? Could you try latest keycloak
        and <br id="yui_3_16_0_ym19_1_1465797102861_4253">
        check if performance is still the issue?<br
          id="yui_3_16_0_ym19_1_1465797102861_4254">
        <br id="yui_3_16_0_ym19_1_1465797102861_4255">
        Marek<br id="yui_3_16_0_ym19_1_1465797102861_4256">
        <br id="yui_3_16_0_ym19_1_1465797102861_4257">
        On 08/06/16 01:30, Fabricio Milone wrote:<br
          id="yui_3_16_0_ym19_1_1465797102861_4258">
        &gt; Hi all,<br id="yui_3_16_0_ym19_1_1465797102861_4259">
        &gt;<br id="yui_3_16_0_ym19_1_1465797102861_4260">
        &gt; I sent this email yesterday with 5 or more attachments, so
        I think it <br id="yui_3_16_0_ym19_1_1465797102861_4261">
        &gt; was blocked or something... here I go again :)<br
          id="yui_3_16_0_ym19_1_1465797102861_4262">
        &gt;<br id="yui_3_16_0_ym19_1_1465797102861_4263">
        &gt; I've been running load tests on our application during the
        last few <br id="yui_3_16_0_ym19_1_1465797102861_4264">
        &gt; weeks, and having some performance issues when my custom
        federator is <br id="yui_3_16_0_ym19_1_1465797102861_4265">
        &gt; enabled.<br id="yui_3_16_0_ym19_1_1465797102861_4266">
        &gt;<br id="yui_3_16_0_ym19_1_1465797102861_4267">
        &gt; The performance issue does not exist when the federator is
        disabled.<br id="yui_3_16_0_ym19_1_1465797102861_4268">
        &gt; *Configuration*:<br
          id="yui_3_16_0_ym19_1_1465797102861_4269">
        &gt;<br id="yui_3_16_0_ym19_1_1465797102861_4270">
        &gt; I have a cluster of 2 instances of Keycloak, with a
        standalone DB, <br id="yui_3_16_0_ym19_1_1465797102861_4271">
        &gt; we've verified the DB isn't an issue when the federator is
        disabled. <br id="yui_3_16_0_ym19_1_1465797102861_4272">
        &gt; Both instances have a quad core CPU and they are in the
        same network. <br id="yui_3_16_0_ym19_1_1465797102861_4273">
        &gt; We?ve left the memory at 512MB. The test script, database
        and API that <br id="yui_3_16_0_ym19_1_1465797102861_4274">
        &gt; connects to the federator are in separate machines.<br
          id="yui_3_16_0_ym19_1_1465797102861_4275">
        &gt; *Federator*:<br id="yui_3_16_0_ym19_1_1465797102861_4276">
        &gt;<br id="yui_3_16_0_ym19_1_1465797102861_4277">
        &gt; We have a simple custom federator that makes calls to a
        very <br id="yui_3_16_0_ym19_1_1465797102861_4278">
        &gt; performant api, which has been tested and is ok.
        Additionally, we've <br
          id="yui_3_16_0_ym19_1_1465797102861_4279">
        &gt; tested stubbing the API so the performance is not a problem
        there. <br id="yui_3_16_0_ym19_1_1465797102861_4280">
        &gt; This federator is using a jaxb marshaller to create a
        request, again <br id="yui_3_16_0_ym19_1_1465797102861_4281">
        &gt; tested in isolation and is performing well.<br
          id="yui_3_16_0_ym19_1_1465797102861_4282">
        &gt;<br id="yui_3_16_0_ym19_1_1465797102861_4283">
        &gt; As the federator is doing a lot of calls to the API (3 per
        login <br id="yui_3_16_0_ym19_1_1465797102861_4284">
        &gt; request), I've implemented a httpclient that uses a <br
          id="yui_3_16_0_ym19_1_1465797102861_4285">
        &gt; PoolingHttpClientConnectionManager with 1000 connections
        available to <br id="yui_3_16_0_ym19_1_1465797102861_4286">
        &gt; use, instead of using the standard apache httpclient from
        http <br id="yui_3_16_0_ym19_1_1465797102861_4287">
        &gt; components. That hasn't improved a bit the performance of
        the system.<br id="yui_3_16_0_ym19_1_1465797102861_4288">
        &gt; *Tests*:<br id="yui_3_16_0_ym19_1_1465797102861_4289">
        &gt; It is a gatling scala script that could generate around
        ~300 (or more) <br id="yui_3_16_0_ym19_1_1465797102861_4290">
        &gt; requests/second to the direct grants login endpoint using
        random <br id="yui_3_16_0_ym19_1_1465797102861_4291">
        &gt; usernames from a list (all of them already registered using
        KC). The <br id="yui_3_16_0_ym19_1_1465797102861_4292">
        &gt; script is doing a round robin across both instances of
        Keycloak with <br id="yui_3_16_0_ym19_1_1465797102861_4293">
        &gt; an even distribution to each KC instance.<br
          id="yui_3_16_0_ym19_1_1465797102861_4294">
        &gt; The idea is simulate a load of 300 to 1500 concurrent users
        trying to <br id="yui_3_16_0_ym19_1_1465797102861_4295">
        &gt; login into our systems.<br
          id="yui_3_16_0_ym19_1_1465797102861_4296">
        &gt; *Problem*:<br id="yui_3_16_0_ym19_1_1465797102861_4297">
        &gt;<br id="yui_3_16_0_ym19_1_1465797102861_4298">
        &gt; If I run the tests without using a federation I can see a
        very good <br id="yui_3_16_0_ym19_1_1465797102861_4299">
        &gt; performance, but when I try to run the tests with the
        custom <br id="yui_3_16_0_ym19_1_1465797102861_4300">
        &gt; federation code, the performance drops from ~150
        requests/second to 22 <br
          id="yui_3_16_0_ym19_1_1465797102861_4301">
        &gt; req/sec using both instances.<br
          id="yui_3_16_0_ym19_1_1465797102861_4302">
        &gt; Memory wise, it seems to be ok. I've never seen an error
        related to <br id="yui_3_16_0_ym19_1_1465797102861_4303">
        &gt; memory with this configuration, also if you take a look at
        the <br id="yui_3_16_0_ym19_1_1465797102861_4304">
        &gt; attached visualVM screenshot you'll see that memory is not
        a problem <br id="yui_3_16_0_ym19_1_1465797102861_4305">
        &gt; or it seems not to be.<br
          id="yui_3_16_0_ym19_1_1465797102861_4306">
        &gt; CPU utilisation is very low to my mind, I'd expect more
        than 80% of <br id="yui_3_16_0_ym19_1_1465797102861_4307">
        &gt; usage or something like that.<br
          id="yui_3_16_0_ym19_1_1465797102861_4308">
        &gt; There is a method that is leading the CPU samples on
        VisualVM called <br id="yui_3_16_0_ym19_1_1465797102861_4309">
        &gt; Semaphore.tryAcquire(). Not quite sure what's that for,
        still <br id="yui_3_16_0_ym19_1_1465797102861_4310">
        &gt; investigating.<br id="yui_3_16_0_ym19_1_1465797102861_4311">
        &gt;<br id="yui_3_16_0_ym19_1_1465797102861_4312">
        &gt; I can see that a lot of new threads are being created when
        the test <br id="yui_3_16_0_ym19_1_1465797102861_4313">
        &gt; starts, as it creates around 60requests/second to the
        direct grants <br id="yui_3_16_0_ym19_1_1465797102861_4314">
        &gt; login call, but it seems to be a bottleneck at some point.<br
          id="yui_3_16_0_ym19_1_1465797102861_4315">
        &gt;<br id="yui_3_16_0_ym19_1_1465797102861_4316">
        &gt; So I'm wondering if there is some configuration I'm missing
        on <br id="yui_3_16_0_ym19_1_1465797102861_4317">
        &gt; Keycloak side that could be affecting the cluster
        performance when a <br
          id="yui_3_16_0_ym19_1_1465797102861_4318">
        &gt; federator is enabled. Maybe something related to jpa
        connections, <br id="yui_3_16_0_ym19_1_1465797102861_4319">
        &gt; infinispan configuration or even wildfly.<br
          id="yui_3_16_0_ym19_1_1465797102861_4320">
        &gt;<br id="yui_3_16_0_ym19_1_1465797102861_4321">
        &gt; I'd really appreciate your help on this one as I'm out of
        ideas.<br id="yui_3_16_0_ym19_1_1465797102861_4322">
        &gt;<br id="yui_3_16_0_ym19_1_1465797102861_4323">
        &gt; I've attached some screenshots of visualVM and tests
        results from my <br id="yui_3_16_0_ym19_1_1465797102861_4324">
        &gt; last run today.<br
          id="yui_3_16_0_ym19_1_1465797102861_4325">
        &gt;<br id="yui_3_16_0_ym19_1_1465797102861_4326">
        &gt;<br id="yui_3_16_0_ym19_1_1465797102861_4327">
        &gt; Sorry for the long email and please let me know if you need
        further <br id="yui_3_16_0_ym19_1_1465797102861_4328">
        &gt; information.<br id="yui_3_16_0_ym19_1_1465797102861_4329">
        &gt;<br id="yui_3_16_0_ym19_1_1465797102861_4330">
        &gt; Thank you in advance,<br
          id="yui_3_16_0_ym19_1_1465797102861_4331">
        &gt;<br id="yui_3_16_0_ym19_1_1465797102861_4332">
        &gt; Regards,<br id="yui_3_16_0_ym19_1_1465797102861_4333">
        &gt; Fab<br id="yui_3_16_0_ym19_1_1465797102861_4334">
        &gt;<br id="yui_3_16_0_ym19_1_1465797102861_4335">
        &gt; -- <br id="yui_3_16_0_ym19_1_1465797102861_4336">
        &gt; *Fabricio Milone*<br
          id="yui_3_16_0_ym19_1_1465797102861_4337">
        <div dir="ltr">&gt; Developer</div>
      </div>
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <br>
      <pre wrap="">_______________________________________________
keycloak-user mailing list
<a class="moz-txt-link-abbreviated" href="mailto:keycloak-user@lists.jboss.org">keycloak-user@lists.jboss.org</a>
<a class="moz-txt-link-freetext" href="https://lists.jboss.org/mailman/listinfo/keycloak-user">https://lists.jboss.org/mailman/listinfo/keycloak-user</a></pre>
    </blockquote>
    <br>
  </body>
</html>