<div dir="ltr">Hi All,<div><br></div><div>We have upgraded the keycloak version to 1.9.0. </div><div>I just carried out a load test on our identity server and it seems to have reduced the failures to a great extent.</div><div>However, when I execute 50 threads per second, there are some intermittent failures (2-3 failures for 50 threads). I further noticed that the frequency is higher for realm roles than for client roles.</div><div><br></div><div>Regards,</div><div>Malmi</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Feb 6, 2016 at 8:33 AM, Malmi Samarasinghe <span dir="ltr">&lt;<a href="mailto:malmi.suh@gmail.com" target="_blank">malmi.suh@gmail.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Many Thanks to your assistance regarding the issue. </div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Feb 5, 2016 at 7:12 PM, Bill Burke <span dir="ltr">&lt;<a href="mailto:bburke@redhat.com" target="_blank">bburke@redhat.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
  
    
  
  <div bgcolor="#FFFFFF" text="#000000">
    1.9.0.Final will have it...<div><div><br>
    <br>
    <div>On 2/5/2016 7:50 AM, Malmi Samarasinghe
      wrote:<br>
    </div>
    <blockquote type="cite">
      <div dir="ltr">Hi Stian,
        <div><br>
        </div>
        <div>Thank you very much for looking in to the issue. We tried
          with around 6 role creations per second, and I tried switching
          off realm cache and it had negative impact on the performance
          of other API s.</div>
        <div><br>
        </div>
        <div>Really appreciate if you could suggest us a rough timeline
          for a fix date. </div>
        <div><br>
        </div>
        <div>Regards,</div>
        <div>Malmi</div>
      </div>
      <div class="gmail_extra"><br>
        <div class="gmail_quote">On Fri, Feb 5, 2016 at 3:20 PM, Stian
          Thorgersen <span dir="ltr">&lt;<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>&gt;</span>
          wrote:<br>
          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div dir="ltr">Either don&#39;t create roles concurrently or
              disable cache.
              <div><br>
              </div>
              <div>How frequently are you creating roles? Just wondering
                because if you do it will significantly impact the
                benefits of the cache as we invalidate a large amount of
                the cache when roles are added/removed.</div>
              <div><br>
              </div>
              <div>The problem you are seeing is most likely down to a
                race condition when the realm role list (or client role
                lists) are re-loaded after they are invalidated. I
                haven&#39;t had much time to look at it yet, so I don&#39;t know
                the exact cause or a solution.</div>
            </div>
            <div>
              <div>
                <div class="gmail_extra"><br>
                  <div class="gmail_quote">On 5 February 2016 at 09:57,
                    Malmi Samarasinghe <span dir="ltr">&lt;<a href="mailto:malmi.suh@gmail.com" target="_blank"></a><a href="mailto:malmi.suh@gmail.com" target="_blank">malmi.suh@gmail.com</a>&gt;</span>
                    wrote:<br>
                    <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                      <div dir="ltr">Hi Stian,
                        <div><br>
                        </div>
                        <div>We have this in production is there any
                          intermediary fix that we can do or any
                          workaround?</div>
                        <div><br>
                        </div>
                        <div>Regards,</div>
                        <div>Malmi</div>
                      </div>
                      <div>
                        <div>
                          <div class="gmail_extra"><br>
                            <div class="gmail_quote">On Fri, Feb 5, 2016
                              at 2:11 PM, Stian Thorgersen <span dir="ltr">&lt;<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>&gt;</span>
                              wrote:<br>
                              <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                <div dir="ltr">Confirmed this bug <a href="https://issues.jboss.org/browse/KEYCLOAK-2458" target="_blank"></a><a href="https://issues.jboss.org/browse/KEYCLOAK-2458" target="_blank">https://issues.jboss.org/browse/KEYCLOAK-2458</a></div>
                                <div>
                                  <div>
                                    <div class="gmail_extra"><br>
                                      <div class="gmail_quote">On 5
                                        February 2016 at 06:53, Malmi
                                        Samarasinghe <span dir="ltr">&lt;<a href="mailto:malmi.suh@gmail.com" target="_blank"></a><a href="mailto:malmi.suh@gmail.com" target="_blank">malmi.suh@gmail.com</a>&gt;</span>
                                        wrote:<br>
                                        <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                          <div dir="ltr">Hi Stian/Bill,
                                            <div><br>
                                            </div>
                                            <div>I just wanted to
                                              highlight that this issue
                                              only occurred when realm
                                              cache enabled option is
                                              ON. </div>
                                            <div><br>
                                            </div>
                                            <div>Regards,</div>
                                            <div>Malmi</div>
                                          </div>
                                          <div>
                                            <div>
                                              <div class="gmail_extra"><br>
                                                <div class="gmail_quote">On
                                                  Thu, Feb 4, 2016 at
                                                  8:38 PM, Malmi
                                                  Samarasinghe <span dir="ltr">&lt;<a href="mailto:malmi.suh@gmail.com" target="_blank"></a><a href="mailto:malmi.suh@gmail.com" target="_blank">malmi.suh@gmail.com</a>&gt;</span>
                                                  wrote:<br>
                                                  <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                                    <div dir="ltr">
                                                      <div>Hi Stian</div>
                                                      <div><br>
                                                      </div>
                                                      I have multiple
                                                      threads creating
                                                      different roles.
                                                      Basically one
                                                      thread will
                                                      execute all three
                                                      apis one after
                                                      another.
                                                      <div><br>
                                                      </div>
                                                      <div>Regards,</div>
                                                      <div>Malmi</div>
                                                    </div>
                                                    <div>
                                                      <div>
                                                        <div class="gmail_extra"><br>
                                                          <div class="gmail_quote">On
                                                          Thu, Feb 4,
                                                          2016 at 5:23
                                                          PM, Stian
                                                          Thorgersen <span dir="ltr">&lt;<a href="mailto:sthorger@redhat.com" target="_blank"></a><a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>&gt;</span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                                          <div dir="ltr">When
                                                          you say
                                                          method1 is
                                                          executed in
                                                          multiple
                                                          threads, do
                                                          you mean one
                                                          thread creates
                                                          the role and
                                                          another
                                                          retrieves it?
                                                          Or do you have
                                                          multiple
                                                          threads
                                                          creating
                                                          different
                                                          roles?</div>
                                                          <div>
                                                          <div>
                                                          <div class="gmail_extra"><br>
                                                          <div class="gmail_quote">On
                                                          4 February
                                                          2016 at 12:31,
                                                          Malmi
                                                          Samarasinghe <span dir="ltr">&lt;<a href="mailto:malmi.suh@gmail.com" target="_blank"></a><a href="mailto:malmi.suh@gmail.com" target="_blank">malmi.suh@gmail.com</a>&gt;</span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                                          <div dir="ltr">Hi
                                                          Bill,
                                                          <div><br>
                                                          </div>
                                                          <div>Please
                                                          find the work
                                                          flow that we
                                                          have
                                                          implemented</div>
                                                          <div>create
                                                          user : POST
                                                          : admin/realms/{realm}/users</div>
                                                          <div><br>
                                                          </div>
                                                          <div><b>Method1</b>
                                                          wrapps the
                                                          following API
                                                          calls</div>
                                                          <div>Create
                                                          Realm role :
                                                          POST :
                                                          admin/realms/{realm}/roles</div>
                                                          <div>Retrieve
                                                          Role : GET
                                                          : admin/realms/{realm}/roles/{roleName}</div>
                                                          <div>Assign
                                                          Role : POST :
admin/realms/leapset/users/{0}/role-mappings/realm</div>
                                                          <div><br>
                                                          </div>
                                                          <div>Same for
                                                          the client
                                                          roles as well.</div>
                                                          <div><br>
                                                          </div>
                                                          <div><b>Method1
                                                          </b>is
                                                          executed in
                                                          multiple
                                                          threads and
                                                          assign reams
                                                          role API
                                                          starts failing
                                                          with <span style="font-size:12.8px">404
                                                          (keycloak log
                                                          states role
                                                          not found)</span></div>
                                                          <div><br>
                                                          </div>
                                                          <div>Regards,</div>
                                                          <div>Malmi</div>
                                                          </div>
                                                          <div>
                                                          <div>
                                                          <div class="gmail_extra"><br>
                                                          <div class="gmail_quote">On
                                                          Thu, Feb 4,
                                                          2016 at 9:00
                                                          AM, Bill Burke
                                                          <span dir="ltr">&lt;<a href="mailto:bburke@redhat.com" target="_blank"></a><a href="mailto:bburke@redhat.com" target="_blank">bburke@redhat.com</a>&gt;</span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                                          <div bgcolor="#FFFFFF" text="#000000">
                                                          Can you give
                                                          me what REST
                                                          invocations
                                                          you are doing?
                                                          How do you
                                                          find the
                                                          role?  How do
                                                          you create the
                                                          role? etc...<span><br>
                                                          <br>
                                                          <div>On
                                                          2/3/2016 9:45
                                                          PM, Malmi
                                                          Samarasinghe
                                                          wrote:<br>
                                                          </div>
                                                          </span>
                                                          <div>
                                                          <div>
                                                          <blockquote type="cite">
                                                          <div dir="ltr">Hi
                                                          Bill,
                                                          <div><br>
                                                          </div>
                                                          <div>We tried
                                                          the above fix
                                                          on top of
                                                          1.7.0 by
                                                          applying the
                                                          changes from
                                                          the commits
                                                          attached to
                                                          the <a href="https://issues.jboss.org/browse/KEYCLOAK-2327" target="_blank"></a><a href="https://issues.jboss.org/browse/KEYCLOAK-2327" target="_blank">https://issues.jboss.org/browse/KEYCLOAK-2327</a> and

                                                          deployed, and
                                                          it seems to
                                                          have the same
                                                          issue. If you
                                                          have any
                                                          further update
                                                          on this please
                                                          let us know.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>Regards,</div>
                                                          <div>Malmi </div>
                                                          </div>
                                                          <div class="gmail_extra"><br>
                                                          <div class="gmail_quote">On
                                                          Mon, Feb 1,
                                                          2016 at 4:02
                                                          PM, Stian
                                                          Thorgersen <span dir="ltr">&lt;<a href="mailto:sthorger@redhat.com" target="_blank"></a><a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>&gt;</span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                                          <div dir="ltr">This
                                                          could be
                                                          related to <a href="https://issues.jboss.org/browse/KEYCLOAK-2327" target="_blank"></a><a href="https://issues.jboss.org/browse/KEYCLOAK-2327" target="_blank">https://issues.jboss.org/browse/KEYCLOAK-2327</a>.
                                                          <div><br>
                                                          </div>
                                                          <div>It&#39;s
                                                          already fixed
                                                          in master, so
                                                          if you can try
                                                          it out that
                                                          would be
                                                          great. We
                                                          should also
                                                          have a
                                                          1.8.1.Final
                                                          release this
                                                          week with the
                                                          fix in as
                                                          well.</div>
                                                          </div>
                                                          <div>
                                                          <div>
                                                          <div class="gmail_extra"><br>
                                                          <div class="gmail_quote">On
                                                          30 January
                                                          2016 at 05:16,
                                                          Malmi
                                                          Samarasinghe <span dir="ltr">&lt;<a href="mailto:malmi.suh@gmail.com" target="_blank"></a><a href="mailto:malmi.suh@gmail.com" target="_blank">malmi.suh@gmail.com</a>&gt;</span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                                          <p dir="ltr">Hi
                                                          Bill,</p>
                                                          <p dir="ltr">We
                                                          are using
                                                          keycloak 1.7.0
                                                          and rdbms
                                                          (mysql)</p>
                                                          <p dir="ltr">Regards,<br>
                                                          Malmi
                                                          Samarasinghe</p>
                                                          <div>
                                                          <div>
                                                          <div class="gmail_quote">On
                                                          Jan 29, 2016
                                                          7:41 PM, &quot;Bill
                                                          Burke&quot; &lt;<a href="mailto:bburke@redhat.com" target="_blank"></a><a href="mailto:bburke@redhat.com" target="_blank">bburke@redhat.com</a>&gt;

                                                          wrote:<br type="attribution">
                                                          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                                          <div bgcolor="#FFFFFF" text="#000000">
                                                          Which version
                                                          of keycloak? 
                                                          RDBMS or
                                                          Mongo?<br>
                                                          <br>
                                                          <div>On
                                                          1/29/2016
                                                          12:35 AM,
                                                          Malmi
                                                          Samarasinghe
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote type="cite">
                                                          <div dir="ltr">Hi
                                                          Everyone,
                                                          <div><br>
                                                          </div>
                                                          <div>In my
                                                          application we
                                                          create
                                                          retrieve and
                                                          assign role
                                                          subsequently
                                                          and it seems
                                                          that even for
                                                          a small load
                                                          (2-3 threads)
                                                          with realm
                                                          cache enabled
                                                          option, assign
                                                          realm role
                                                          call fails due
                                                          to role not
                                                          exist error
                                                          and 404 is
                                                          returned from
                                                          keycloak.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>With the
                                                          realm cache
                                                          disabled
                                                          option the
                                                          load works
                                                          fine.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>Please
                                                          get back to me
                                                          if you have
                                                          any
                                                          information on
                                                          any other
                                                          option we can
                                                          follow to get
                                                          this issue
                                                          sorted or on
                                                          what action
                                                          the realm
                                                          cache will be
                                                          persisted to
                                                          DB.<br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div>Regards,</div>
                                                          <div>Malmi</div>
                                                          </div>
                                                          <br>
                                                          <fieldset></fieldset>
                                                          <br>
                                                          <pre>_______________________________________________
keycloak-user mailing list
<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a>
<a href="https://lists.jboss.org/mailman/listinfo/keycloak-user" target="_blank">https://lists.jboss.org/mailman/listinfo/keycloak-user</a></pre>
                                                          </blockquote>
                                                          <br>
                                                          <pre cols="72">-- 
Bill Burke
JBoss, a division of Red Hat
<a href="http://bill.burkecentral.com" target="_blank">http://bill.burkecentral.com</a></pre>
                                                          </div>
                                                          <br>
_______________________________________________<br>
                                                          keycloak-user
                                                          mailing list<br>
                                                          <a href="mailto:keycloak-user@lists.jboss.org" target="_blank"></a><a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a><br>
                                                          <a href="https://lists.jboss.org/mailman/listinfo/keycloak-user" rel="noreferrer" target="_blank"></a><a href="https://lists.jboss.org/mailman/listinfo/keycloak-user" target="_blank">https://lists.jboss.org/mailman/listinfo/keycloak-user</a><br>
                                                          </blockquote>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          <br>
_______________________________________________<br>
                                                          keycloak-user
                                                          mailing list<br>
                                                          <a href="mailto:keycloak-user@lists.jboss.org" target="_blank"></a><a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a><br>
                                                          <a href="https://lists.jboss.org/mailman/listinfo/keycloak-user" rel="noreferrer" target="_blank"></a><a href="https://lists.jboss.org/mailman/listinfo/keycloak-user" target="_blank">https://lists.jboss.org/mailman/listinfo/keycloak-user</a><br>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </blockquote>
                                                          <br>
                                                          <pre cols="72">-- 
Bill Burke
JBoss, a division of Red Hat
<a href="http://bill.burkecentral.com" target="_blank">http://bill.burkecentral.com</a></pre>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                        </div>
                                                      </div>
                                                    </div>
                                                  </blockquote>
                                                </div>
                                                <br>
                                              </div>
                                            </div>
                                          </div>
                                        </blockquote>
                                      </div>
                                      <br>
                                    </div>
                                  </div>
                                </div>
                              </blockquote>
                            </div>
                            <br>
                          </div>
                        </div>
                      </div>
                    </blockquote>
                  </div>
                  <br>
                </div>
              </div>
            </div>
          </blockquote>
        </div>
        <br>
      </div>
    </blockquote>
    <br>
    <pre cols="72">-- 
Bill Burke
JBoss, a division of Red Hat
<a href="http://bill.burkecentral.com" target="_blank">http://bill.burkecentral.com</a></pre>
  </div></div></div>

</blockquote></div><br></div>
</div></div></blockquote></div><br></div>