<div dir="ltr"><div><div><div><div><div><div><div>Bill,<br><br></div>This particular load test consists of four API calls, starts with authentication:<br><br>1) <span style="color:rgb(51,51,51)">Invoke {host}</span>/auth/realms/xxxx<span style="color:rgb(51,51,51)">/protocol</span>/openid-connect/token with &quot;<code class="">username={username}&amp;password={password}&amp;grant_type=password</code>&quot; as the body - we get the Access Token from this and use it for subsequent API calls<br></div>2) Get user by User ID<br></div>3) Get Realm Roles of above user<br></div>4) Get Client Roles of a single given client for above user<br><br></div>We
 run this test suite with concurrent threads (same superuser for 
authentication, but steps 2,3,4 for different users). But with our 2 
highly available M3-Medium server setup, it is impossible to go beyond 
10-15 threads w/o the processor hitting 100% usage.<br><br></div><div>During the tests, we see millions (figuratively) of invalidation logs like this:<br>[2015-11-30 14:32:08.0823], DEBUG, org.infinispan.interceptors.InvalidationInterceptor Timer-2 - Cache [dev-idm-a1] replicating InvalidateCommand{keys=[40113545-5069-47c4-bf1c-8f58a303caf6]}<br></div><div><br><br></div>Regards,<br></div>Lohitha.<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Nov 30, 2015 at 8:17 PM, Bill Burke <span dir="ltr">&lt;<a href="mailto:bburke@redhat.com" target="_blank">bburke@redhat.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I just did a loop test on GET https://{host}/auth/admin/realms/xxxxx/users/xxxx.  The same userid over and over as you described.  I see zero invalidations.  Are you sure you aren&#39;t doing any updates?<span class=""><br>
<br>
On 11/30/2015 9:33 AM, Stian Thorgersen wrote:<br>
</span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
If you can try building master and enable trace logging for<br>
&quot;org.keycloak.models.cache.infinispan.InfinispanCacheUserProviderFactory&quot;.<br>
It will print log statements when entries are removed/invalidated in the<br>
cache.<br>
<br>
On 30 November 2015 at 15:19, Bill Burke &lt;<a href="mailto:bburke@redhat.com" target="_blank">bburke@redhat.com</a><br></span><span class="">
&lt;mailto:<a href="mailto:bburke@redhat.com" target="_blank">bburke@redhat.com</a>&gt;&gt; wrote:<br>
<br>
    Can you describe your load tests please?  What kind of authentication<br>
    are you doing?  The only thing that would cause invalidations is HOTP.<br>
<br>
    On 11/30/2015 9:13 AM, Lohitha Chiranjeewa wrote:<br>
    &gt; Bill, the problem here is that our load tests cannot go beyond a paltry<br>
    &gt; 15-20 concurrent threads with the default set up (invalidation caches<br>
    &gt; with SYNC mode - two HA AWS M3-Medium servers). Hence we&#39;re trying to<br>
    &gt; find ways to improve on the performance and the avg response time under<br>
    &gt; a load.<br>
    &gt;<br>
    &gt; On Mon, Nov 30, 2015 at 7:16 PM, Bill Burke &lt;<a href="mailto:bburke@redhat.com" target="_blank">bburke@redhat.com</a> &lt;mailto:<a href="mailto:bburke@redhat.com" target="_blank">bburke@redhat.com</a>&gt;<br></span><div><div class="h5">
    &gt; &lt;mailto:<a href="mailto:bburke@redhat.com" target="_blank">bburke@redhat.com</a> &lt;mailto:<a href="mailto:bburke@redhat.com" target="_blank">bburke@redhat.com</a>&gt;&gt;&gt; wrote:<br>
    &gt;<br>
    &gt;     BTW, besides being more scalable, we invalidate rather than replicate to<br>
    &gt;     avoid transmitting sensitive security data over the network.<br>
    &gt;<br>
    &gt;     On 11/30/2015 8:29 AM, Lohitha Chiranjeewa wrote:<br>
    &gt;     &gt; Glad if you could look into the above and check if there are app level<br>
    &gt;     &gt; issues related to invalidations, thanks.<br>
    &gt;     &gt;<br>
    &gt;     &gt; One way to get around this through config changes is to make the<br>
    &gt;     &gt; relevant cache modes &#39;ASYNC&#39; and set up suitable values for &#39;queue-size&#39;<br>
    &gt;     &gt; and &#39;queue-flush-interval&#39; depending on your needs. Then the<br>
    &gt;     &gt; invalidations won&#39;t happen with each and every call.<br>
    &gt;     &gt;<br>
    &gt;     &gt; Obviously there would be problems if caches aren&#39;t invalidated in time<br>
    &gt;     &gt; with the above set up, it&#39;s up to the devs to come up with a suitable<br>
    &gt;     &gt; set of configs to cater to their needs.<br>
    &gt;     &gt;<br>
    &gt;     &gt;<br>
    &gt;     &gt; Regards,<br>
    &gt;     &gt; Lohitha.<br>
    &gt;     &gt;<br>
    &gt;     &gt;<br>
    &gt;     &gt;<br>
    &gt;     &gt; On Mon, Nov 30, 2015 at 1:08 PM, Stian Thorgersen &lt;<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> &lt;mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>&gt;<br>
    &lt;mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> &lt;mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>&gt;&gt;<br>
    &gt;     &gt; &lt;mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> &lt;mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>&gt;<br>
    &lt;mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> &lt;mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>&gt;&gt;&gt;&gt; wrote:<br>
    &gt;     &gt;<br>
    &gt;     &gt;     It is not expected behavior. Users should only be invalidated if<br>
    &gt;     &gt;     changes are made.<br>
    &gt;     &gt;<br>
    &gt;     &gt;     On 30 November 2015 at 07:01, Lohitha Chiranjeewa &lt;<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>&gt; &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a><br>
    &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>&gt;&gt;<br>
    &gt;     &gt;     &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>&gt; &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a><br>
    &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>&gt;&gt;&gt;&gt; wrote:<br>
    &gt;     &gt;<br>
    &gt;     &gt;         Just tested the same flow extensively with Keycloak 1.2.0 as<br>
    &gt;     &gt;         well, and it seems the behavior is the same. Lots of MySQL<br>
    &gt;     &gt;         select queries getting executed with each call. Seems there&#39;s a<br>
    &gt;     &gt;         number of unnecessary cache invalidations going on which causes<br>
    &gt;     &gt;         Keycloak to fetch data from the DB over and over.<br>
    &gt;     &gt;<br>
    &gt;     &gt;         Could you please confirm if this is the expected behavior? As<br>
    &gt;     &gt;         far as I can see there&#39;s considerable performance degradation<br>
    &gt;     &gt;         due to unnecessary cache invalidations.<br>
    &gt;     &gt;<br>
    &gt;     &gt;<br>
    &gt;     &gt;         On Fri, Nov 27, 2015 at 8:42 PM, Lohitha Chiranjeewa<br>
     &gt;     &gt;         &lt;<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>&gt;<br>
    &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>&gt;&gt;<br>
    &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>&gt;<br></div></div><div><div class="h5">
    &gt;     &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>&gt;&gt;&gt;&gt; wrote:<br>
    &gt;     &gt;<br>
    &gt;     &gt;             I&#39;m invoking the following API call:<br>
    &gt;     &gt;             https://{host}/auth/admin/realms/xxxxx/users/55ffe851-2d94-460e-88b9-bc7340531b56<br>
    &gt;     &gt;             (same realm and user ID over and over again). The two HA<br>
    &gt;     &gt;             server(s) are idle apart from serving those calls. And I&#39;m<br>
    &gt;     &gt;             seeing the following SQL getting logged in my MySQL log for<br>
    &gt;     &gt;             each and every call:<br>
    &gt;     &gt;<br>
    &gt;     &gt;             select userentity0_.ID as ID1_42_,<br>
    &gt;     &gt;             userentity0_.CREATED_TIMESTAMP as CREATED_2_42_,<br>
    &gt;     &gt;             userentity0_.EMAIL as EMAIL3_42_,<br>
    &gt;     &gt;             userentity0_.EMAIL_CONSTRAINT as EMAIL_CO4_42_,<br>
    &gt;     &gt;             userentity0_.EMAIL_VERIFIED as EMAIL_VE5_42_,<br>
    &gt;     &gt;             userentity0_.ENABLED as ENABLED6_42_,<br>
    &gt;     &gt;             userentity0_.federation_link as federati7_42_,<br>
    &gt;     &gt;             userentity0_.FIRST_NAME as FIRST_NA8_42_,<br>
    &gt;     &gt;             userentity0_.LAST_NAME as LAST_NAM9_42_,<br>
    &gt;     &gt;             userentity0_.REALM_ID as REALM_I10_42_,<br>
    &gt;     &gt;             userentity0_.SERVICE_ACCOUNT_CLIENT_LINK as SERVICE11_42_,<br>
    &gt;     &gt;             userentity0_.TOTP as TOTP12_42_, userentity0_.USERNAME as<br>
    &gt;     &gt;             USERNAM13_42_ from USER_ENTITY userentity0_ where<br>
    &gt;     &gt;             userentity0_.ID=&#39;55ffe851-2d94-460e-88b9-bc7340531b56&#39; and<br>
    &gt;     &gt;             userentity0_.REALM_ID=&#39;xxxxx&#39;<br>
    &gt;     &gt;<br>
    &gt;     &gt;<br>
    &gt;     &gt;<br>
    &gt;     &gt;             On Fri, Nov 27, 2015 at 8:03 PM, Stian Thorgersen<br>
    &gt;     &gt;             &lt;<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> &lt;mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>&gt;<br>
    &lt;mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> &lt;mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>&gt;&gt;<br></div></div>
    &gt;     &lt;mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> &lt;mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>&gt;<span class=""><br>
    &lt;mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> &lt;mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>&gt;&gt;&gt;&gt; wrote:<br>
    &gt;     &gt;<br>
    &gt;     &gt;                 Strange - what are you doing and what are the SQL queries?<br>
    &gt;     &gt;<br>
    &gt;     &gt;                 On 27 November 2015 at 15:23, Lohitha Chiranjeewa<br>
     &gt;     &gt;                 &lt;<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>&gt;<br>
    &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>&gt;&gt;<br></span>
    &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>&gt;<span class=""><br>
    &gt;     &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>&gt;&gt;&gt;&gt; wrote:<br>
    &gt;     &gt;<br>
    &gt;     &gt;                     Yes Stian, that I understand. But the problem here<br>
    &gt;     &gt;                     is even if I execute continuous user retrieval calls<br>
    &gt;     &gt;                     (same user - no other functionality in between),<br>
    &gt;     &gt;                     still MySQL select queries get executed for each<br>
    &gt;     &gt;                     call. So there lies an issue isn&#39;t it?<br>
    &gt;     &gt;<br>
    &gt;     &gt;<br>
    &gt;     &gt;                     On Fri, Nov 27, 2015 at 6:48 PM, Stian Thorgersen<br>
    &gt;      &gt;                     &lt;<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> &lt;mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>&gt;<br>
     &gt;     &lt;mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> &lt;mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>&gt;&gt;<br></span>
    &lt;mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> &lt;mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>&gt;<br>
    &gt;     &lt;mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> &lt;mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>&gt;&gt;&gt;&gt;<span class=""><br>
    &gt;     &gt;                     wrote:<br>
    &gt;     &gt;<br>
    &gt;     &gt;                         Things are still fetched from MySQL. Realms,<br>
    &gt;     &gt;                         clients, users, etc.. are then kept in the<br>
    &gt;     &gt;                         cache, but if it changes it&#39;s re-loaded from<br>
    &gt;     &gt;                         MySQL. We use an invalidation cache, not a<br>
    &gt;     &gt;                         distributed cache.<br>
    &gt;     &gt;<br>
    &gt;     &gt;                         On 27 November 2015 at 14:04, Lohitha<br>
    &gt;     &gt;                         Chiranjeewa &lt;<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>&gt; &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a><br>
    &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>&gt;&gt;<br></span><span class="">
    &gt;     &gt;                         &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>&gt; &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a><br></span><span class="">
    &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>&gt;&gt;&gt;&gt; wrote:<br>
    &gt;     &gt;<br>
    &gt;     &gt;                             What I mean is, if it were working, I<br>
    &gt;     &gt;                             shouldn&#39;t see mysql queries getting executed<br>
    &gt;     &gt;                             right? So my guess is data is still fetched<br>
    &gt;     &gt;                             from the db instead of the cache.<br>
    &gt;     &gt;<br>
    &gt;     &gt;                             On Nov 27, 2015 5:52 PM, &quot;Stian Thorgersen&quot;<br>
    &gt;     &gt;                             &lt;<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> &lt;mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>&gt;<br>
    &lt;mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> &lt;mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>&gt;&gt;<br></span>
    &gt;     &gt;                             &lt;mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> &lt;mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>&gt;<span class=""><br>
    &lt;mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> &lt;mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>&gt;&gt;&gt;&gt; wrote:<br>
    &gt;     &gt;<br>
    &gt;     &gt;                                 Yup, so it&#39;s working now?<br>
    &gt;     &gt;<br>
    &gt;     &gt;                                 On 27 November 2015 at 13:20, Lohitha<br>
    &gt;     &gt;                                 Chiranjeewa &lt;<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>&gt; &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a><br>
    &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>&gt;&gt;<br></span><span class="">
    &gt;     &gt;                                 &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>&gt; &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a><br></span><span class="">
    &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>&gt;&gt;&gt;&gt; wrote:<br>
    &gt;     &gt;<br>
    &gt;     &gt;                                     Apologies, keycloak-server.json<br>
    &gt;     &gt;                                     entries should change to:<br>
    &gt;     &gt;<br>
    &gt;     &gt;                                          &quot;realm&quot;: {<br>
    &gt;     &gt;                                              &quot;provider&quot;: &quot;jpa&quot;<br>
    &gt;     &gt;                                          },<br>
    &gt;     &gt;<br>
    &gt;     &gt;                                          &quot;user&quot;: {<br>
    &gt;     &gt;                                              &quot;provider&quot;: &quot;jpa&quot;<br>
    &gt;     &gt;                                          },<br>
    &gt;     &gt;<br>
    &gt;     &gt;                                          &quot;userSessionPersister&quot;: {<br>
    &gt;     &gt;                                              &quot;provider&quot;: &quot;jpa&quot;<br>
    &gt;     &gt;                                          },<br>
    &gt;     &gt;<br>
    &gt;     &gt;                                     On Fri, Nov 27, 2015 at 5:49 PM,<br>
    &gt;     &gt;                                     Lohitha Chiranjeewa<br>
    &gt;     &gt;                                     &lt;<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>&gt; &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a><br>
    &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>&gt;&gt;<br>
     &gt;      &gt;<br></span>
      &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>&gt;<div><div class="h5"><br>
     &gt;     &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>&gt;&gt;&gt;&gt; wrote:<br>
     &gt;      &gt;<br>
     &gt;      &gt;                                         Hi Stian,<br>
     &gt;      &gt;<br>
     &gt;      &gt;                                         As per the<br>
    migration guide, I<br>
     &gt;      &gt;                                         should have<br>
    Infinispan up and<br>
     &gt;      &gt;                                         running for<br>
    realms, users and<br>
     &gt;      &gt;                                         user sessions without<br>
     &gt;     doing any<br>
     &gt;      &gt;                                         specific changes.<br>
     &gt;      &gt;<br>
      keycloak-server.json was<br>
     &gt;      &gt;                                         reverted back to<br>
    have the<br>
     &gt;      &gt;                                         following entries:<br>
     &gt;      &gt;                                         ...<br>
     &gt;      &gt;                                              &quot;realm&quot;: {<br>
     &gt;      &gt;                                                  &quot;provider&quot;:<br>
     &gt;     &quot;infinispan&quot;<br>
     &gt;      &gt;                                              },<br>
     &gt;      &gt;<br>
     &gt;      &gt;                                              &quot;user&quot;: {<br>
     &gt;      &gt;                                                  &quot;provider&quot;:<br>
     &gt;     &quot;infinispan&quot;<br>
     &gt;      &gt;                                              },<br>
     &gt;      &gt;<br>
     &gt;      &gt;<br>
     &gt;     &quot;userSessionPersister&quot;: {<br>
     &gt;      &gt;                                                  &quot;provider&quot;:<br>
     &gt;     &quot;infinispan&quot;<br>
     &gt;      &gt;                                              },<br>
     &gt;      &gt;                                         ...<br>
     &gt;      &gt;<br>
     &gt;      &gt;                                         In the Admin Console I<br>
     &gt;     have both<br>
     &gt;      &gt;                                         Realm Cache and<br>
    User Cache<br>
     &gt;      &gt;                                         enables. I see certain<br>
     &gt;      &gt;                                         Infinispan related<br>
    logs<br>
     &gt;     getting<br>
     &gt;      &gt;                                         logged as well.<br>
     &gt;      &gt;<br>
     &gt;      &gt;                                         However, at the same<br>
     &gt;     time, I see<br>
     &gt;      &gt;                                         MySQL queries getting<br>
     &gt;     executed<br>
     &gt;      &gt;                                         for all user<br>
    retrieval API<br>
     &gt;      &gt;                                         invocations (even<br>
    if the same<br>
     &gt;      &gt;                                         user is retrieved<br>
     &gt;     continuously):<br>
     &gt;      &gt;                                         ...<br>
     &gt;      &gt;                                         select<br>
    userentity0_.ID as<br>
     &gt;      &gt;                                         ID1_42_,<br>
     &gt;      &gt;<br>
     &gt;       userentity0_.CREATED_TIMESTAMP<br>
     &gt;      &gt;                                         as CREATED_2_42_,<br>
     &gt;      &gt;                                         userentity0_.EMAIL as<br>
     &gt;      &gt;                                         EMAIL3_42_,<br>
     &gt;      &gt;<br>
     &gt;       userentity0_.EMAIL_CONSTRAINT as<br>
     &gt;      &gt;                                         EMAIL_CO4_42_,<br>
     &gt;      &gt;<br>
     &gt;       userentity0_.EMAIL_VERIFIED as<br>
     &gt;      &gt;                                         EMAIL_VE5_42_,<br>
     &gt;      &gt;<br>
      userentity0_.ENABLED as<br>
     &gt;      &gt;                                         ENABLED6_42_,<br>
     &gt;      &gt;<br>
     &gt;       userentity0_.federation_link as<br>
     &gt;      &gt;                                         federati7_42_,<br>
     &gt;      &gt;<br>
      userentity0_.FIRST_NAME as<br>
     &gt;      &gt;                                         FIRST_NA8_42_,<br>
     &gt;      &gt;<br>
      userentity0_.LAST_NAME as<br>
     &gt;      &gt;                                         LAST_NAM9_42_,<br>
     &gt;      &gt;<br>
      userentity0_.REALM_ID as<br>
     &gt;      &gt;                                         REALM_I10_42_,<br>
     &gt;      &gt;<br>
     &gt;       userentity0_.SERVICE_ACCOUNT_CLIENT_LINK<br>
     &gt;      &gt;                                         as SERVICE11_42_,<br>
     &gt;      &gt;                                         userentity0_.TOTP as<br>
     &gt;     TOTP12_42_,<br>
     &gt;      &gt;<br>
      userentity0_.USERNAME as<br>
     &gt;      &gt;                                         USERNAM13_42_ from<br>
     &gt;     USER_ENTITY<br>
     &gt;      &gt;                                         userentity0_ where<br>
     &gt;      &gt;<br>
     &gt;       userentity0_.ID=&#39;55ffe851-2d94-460e-88b9-bc7340531b56&#39;<br>
     &gt;      &gt;                                         and<br>
     &gt;     userentity0_.REALM_ID=&#39;xxxxx&#39;<br>
     &gt;      &gt;                                         ...<br>
     &gt;      &gt;<br>
     &gt;      &gt;                                         So it seems<br>
    something is<br>
     &gt;     wrong<br>
     &gt;      &gt;                                         here. Could you<br>
    point out any<br>
     &gt;      &gt;                                         areas that I could<br>
     &gt;     further look<br>
     &gt;      &gt;                                         into?<br>
     &gt;      &gt;<br>
     &gt;      &gt;<br>
     &gt;      &gt;                                         Regards,<br>
     &gt;      &gt;                                         Lohitha.<br>
     &gt;      &gt;<br>
     &gt;      &gt;                                         On Thu, Nov 26,<br>
    2015 at<br>
     &gt;     7:58 PM,<br>
     &gt;      &gt;                                         Stian Thorgersen<br>
     &gt;      &gt;<br>
      &lt;<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> &lt;mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>&gt;<br>
     &gt;     &lt;mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> &lt;mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>&gt;&gt;<br></div></div>
    &gt;     &gt;                                         &lt;mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> &lt;mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>&gt;<span class=""><br>
    &lt;mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> &lt;mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>&gt;&gt;&gt;&gt; wrote:<br>
    &gt;     &gt;<br>
    &gt;     &gt;                                             Please read the migration guide<br>
    &gt;     &gt;<br>
    &gt;     &gt;                                             On 26 November 2015 at<br>
    &gt;     &gt;                                             14:53, Lohitha Chiranjeewa<br>
    &gt;     &gt;                                             &lt;<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>&gt; &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a><br>
    &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>&gt;&gt;<br>
    &gt;      &gt;<br></span>
     &gt;       &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>&gt;<div><div class="h5"><br>
    &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> &lt;mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>&gt;&gt;&gt;&gt;<br>
     &gt;      &gt;                                             wrote:<br>
     &gt;      &gt;<br>
     &gt;      &gt;                                                 Hi,<br>
     &gt;      &gt;<br>
     &gt;      &gt;                                                 We&#39;re in the<br>
     &gt;     process of<br>
     &gt;      &gt;                                                 assessing the<br>
     &gt;     impact on<br>
     &gt;      &gt;                                                 upgrading from<br>
     &gt;     Keycloak<br>
     &gt;      &gt;                                                 1.2.0 to<br>
    1.6.1.<br>
     &gt;     We came<br>
     &gt;      &gt;                                                 across an<br>
    issue when<br>
     &gt;      &gt;                                                 trying to<br>
    enable<br>
     &gt;      &gt;                                                 Infinispan<br>
    cache<br>
     &gt;     through<br>
     &gt;      &gt;                                                 the<br>
     &gt;     keycloak-server.json<br>
     &gt;      &gt;                                                 file as we<br>
    used<br>
     &gt;     to do in<br>
     &gt;      &gt;                                                 1.2.0.<br>
     &gt;      &gt;<br>
     &gt;      &gt;                                                 We have<br>
    the following<br>
     &gt;      &gt;                                                 entries in<br>
    1.6.1:<br>
     &gt;      &gt;<br>
    &quot;realm&quot;: {<br>
     &gt;      &gt;<br>
    &quot;provider&quot;:<br>
     &gt;      &gt;                                                 &quot;infinispan&quot;<br>
     &gt;      &gt;                                                      },<br>
     &gt;      &gt;<br>
     &gt;      &gt;                                                      &quot;user&quot;: {<br>
     &gt;      &gt;<br>
    &quot;provider&quot;:<br>
     &gt;      &gt;                                                 &quot;infinispan&quot;<br>
     &gt;      &gt;                                                      },<br>
     &gt;      &gt;<br>
     &gt;      &gt;<br>
     &gt;      &gt;<br>
     &gt;       &quot;userSessionPersister&quot;: {<br>
     &gt;      &gt;<br>
    &quot;provider&quot;:<br>
     &gt;      &gt;                                                 &quot;infinispan&quot;<br>
     &gt;      &gt;                                                      },<br>
     &gt;      &gt;                                                 .........<br>
     &gt;      &gt;<br>
     &gt;      &gt;<br>
     &gt;       &quot;connectionsInfinispan&quot;: {<br>
     &gt;      &gt;<br>
     &gt;     &quot;default&quot; : {<br>
     &gt;      &gt;<br>
     &gt;      &gt;<br>
      &quot;cacheContainer&quot; :<br>
     &gt;      &gt;<br>
     &gt;       &quot;java:comp/env/infinispan/Keycloak&quot;<br>
     &gt;      &gt;                                                          }<br>
     &gt;      &gt;                                                      }<br>
     &gt;      &gt;<br>
     &gt;      &gt;                                                 All<br>
    configurations in<br>
     &gt;      &gt;                                                 1.6.1<br>
     &gt;     standalone-ha.xml<br>
     &gt;      &gt;                                                 file remains<br>
     &gt;     comparable<br>
     &gt;      &gt;                                                 (and<br>
    correct to<br>
     &gt;     the best<br>
     &gt;      &gt;                                                 of our<br>
    knowledge)<br>
     &gt;     with<br>
     &gt;      &gt;                                                 the ones<br>
    in 1.2.0.<br>
     &gt;      &gt;<br>
     &gt;      &gt;                                                 With the above<br>
     &gt;     configs,<br>
     &gt;      &gt;                                                 when we<br>
    start the<br>
     &gt;      &gt;                                                 Keycloak<br>
    service the<br>
     &gt;      &gt;                                                 following<br>
     &gt;     error(s) get<br>
     &gt;      &gt;                                                 logged:<br>
     &gt;      &gt;<br>
     &gt;      &gt;<br>
      18:03:31,610 ERROR<br>
     &gt;      &gt;<br>
     &gt;       [org.jboss.msc.service.fail]<br>
     &gt;      &gt;<br>
      (ServerService Thread<br>
     &gt;      &gt;                                                 Pool -- 64)<br>
     &gt;     MSC000001:<br>
     &gt;      &gt;                                                 Failed to<br>
    start<br>
     &gt;     service<br>
     &gt;      &gt;<br>
     &gt;       jboss.undertow.deployment.default-server.default-host./auth:<br>
     &gt;      &gt;<br>
     &gt;       org.jboss.msc.service.StartException<br>
     &gt;      &gt;                                                 in service<br>
     &gt;      &gt;<br>
     &gt;       jboss.undertow.deployment.default-server.default-host./auth:<br>
     &gt;      &gt;<br>
     &gt;       java.lang.RuntimeException:<br>
     &gt;      &gt;                                                 Failed to<br>
    construct<br>
     &gt;      &gt;                                                 public<br>
     &gt;      &gt;<br>
     &gt;<br>
      org.keycloak.services.resources.KeycloakApplication(javax.servlet.ServletContext,org.jboss.resteasy.core.Dispatcher)<br>
     &gt;      &gt;                                                      at<br>
     &gt;      &gt;<br>
     &gt;<br>
      org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:85)<br>
     &gt;      &gt;                                                      at<br>
     &gt;      &gt;<br>
     &gt;<br>
      java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)<br>
     &gt;      &gt;<br>
      [rt.jar:1.7.0_45]<br>
     &gt;      &gt;                                                      at<br>
     &gt;      &gt;<br>
     &gt;       java.util.concurrent.FutureTask.run(FutureTask.java:262)<br>
     &gt;      &gt;<br>
      [rt.jar:1.7.0_45]<br>
     &gt;      &gt;                                                      at<br>
     &gt;      &gt;<br>
     &gt;<br>
      java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)<br>
     &gt;      &gt;<br>
      [rt.jar:1.7.0_45]<br>
     &gt;      &gt;                                                      at<br>
     &gt;      &gt;<br>
     &gt;<br>
      java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)<br>
     &gt;      &gt;<br>
      [rt.jar:1.7.0_45]<br>
     &gt;      &gt;                                                      at<br>
     &gt;      &gt;<br>
     &gt;       java.lang.Thread.run(Thread.java:744)<br>
     &gt;      &gt;<br>
      [rt.jar:1.7.0_45]<br>
     &gt;      &gt;                                                      at<br>
     &gt;      &gt;<br>
     &gt;       org.jboss.threads.JBossThread.run(JBossThread.java:320)<br>
     &gt;      &gt;<br>
     &gt;       [jboss-threads-2.2.0.Final.jar:2.2.0.Final]<br>
     &gt;      &gt;                                                 Caused by:<br>
     &gt;      &gt;<br>
     &gt;       java.lang.RuntimeException:<br>
     &gt;      &gt;                                                 Failed to<br>
    construct<br>
     &gt;      &gt;                                                 public<br>
     &gt;      &gt;<br>
     &gt;<br>
      org.keycloak.services.resources.KeycloakApplication(javax.servlet.ServletContext,org.jboss.resteasy.core.Dispatcher)<br>
     &gt;      &gt;                                                      at<br>
     &gt;      &gt;<br>
     &gt;<br>
      org.jboss.resteasy.core.ConstructorInjectorImpl.construct(ConstructorInjectorImpl.java:160)<br>
     &gt;      &gt;                                                      at<br>
     &gt;      &gt;<br>
     &gt;<br>
      org.jboss.resteasy.spi.ResteasyProviderFactory.createProviderInstance(ResteasyProviderFactory.java:2211)<br>
     &gt;      &gt;                                                      at<br>
     &gt;      &gt;<br>
     &gt;<br>
      org.jboss.resteasy.spi.ResteasyDeployment.createApplication(ResteasyDeployment.java:295)<br>
     &gt;      &gt;                                                      at<br>
     &gt;      &gt;<br>
     &gt;<br>
      org.jboss.resteasy.spi.ResteasyDeployment.start(ResteasyDeployment.java:236)<br>
     &gt;      &gt;                                                      at<br>
     &gt;      &gt;<br>
     &gt;<br>
      org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.init(ServletContainerDispatcher.java:112)<br>
     &gt;      &gt;                                                      at<br>
     &gt;      &gt;<br>
     &gt;<br>
      org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.init(HttpServletDispatcher.java:36)<br>
     &gt;      &gt;                                                      at<br>
     &gt;      &gt;<br>
     &gt;<br>
      io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:117)<br>
     &gt;      &gt;                                                      at<br>
     &gt;      &gt;<br>
     &gt;<br>
      org.wildfly.extension.undertow.security.RunAsLifecycleInterceptor.init(RunAsLifecycleInterceptor.java:78)<br>
     &gt;      &gt;                                                      at<br>
     &gt;      &gt;<br>
     &gt;<br>
      io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:103)<br>
     &gt;      &gt;                                                      at<br>
     &gt;      &gt;<br>
     &gt;<br>
      io.undertow.servlet.core.ManagedServlet$DefaultInstanceStrategy.start(ManagedServlet.java:230)<br>
     &gt;      &gt;                                                      at<br>
     &gt;      &gt;<br>
     &gt;<br>
      io.undertow.servlet.core.ManagedServlet.createServlet(ManagedServlet.java:131)<br>
     &gt;      &gt;                                                      at<br>
     &gt;      &gt;<br>
     &gt;<br>
      io.undertow.servlet.core.DeploymentManagerImpl.start(DeploymentManagerImpl.java:511)<br>
     &gt;      &gt;                                                      at<br>
     &gt;      &gt;<br>
     &gt;<br>
      org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:101)<br>
     &gt;      &gt;                                                      at<br>
     &gt;      &gt;<br>
     &gt;<br>
      org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:82)<br>
     &gt;      &gt;                                                      ... 6<br>
    more<br>
     &gt;      &gt;                                                 Caused by:<br>
     &gt;      &gt;<br>
     &gt;       java.lang.RuntimeException:<br>
     &gt;      &gt;                                                 Failed to find<br>
     &gt;     provider<br>
     &gt;      &gt;                                                 infinispan<br>
    for realm<br>
     &gt;      &gt;                                                      at<br>
     &gt;      &gt;<br>
     &gt;<br>
      org.keycloak.services.DefaultKeycloakSessionFactory.init(DefaultKeycloakSessionFactory.java:66)<br>
     &gt;      &gt;                                                      at<br>
     &gt;      &gt;<br>
     &gt;<br>
      org.keycloak.services.resources.KeycloakApplication.createSessionFactory(KeycloakApplication.java:162)<br>
     &gt;      &gt;                                                      at<br>
     &gt;      &gt;<br>
     &gt;<br>
      org.keycloak.services.resources.KeycloakApplication.&lt;init&gt;(KeycloakApplication.java:62)<br>
     &gt;      &gt;                                                      at<br>
     &gt;      &gt;<br>
     &gt;       sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native<br>
     &gt;      &gt;                                                 Method)<br>
     &gt;     [rt.jar:1.7.0_45]<br>
     &gt;      &gt;                                                      at<br>
     &gt;      &gt;<br>
     &gt;<br>
      sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)<br>
     &gt;      &gt;<br>
      [rt.jar:1.7.0_45]<br>
     &gt;      &gt;                                                      at<br>
     &gt;      &gt;<br>
     &gt;<br>
      sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)<br>
     &gt;      &gt;<br>
      [rt.jar:1.7.0_45]<br>
     &gt;      &gt;                                                      at<br>
     &gt;      &gt;<br>
     &gt;       java.lang.reflect.Constructor.newInstance(Constructor.java:526)<br>
     &gt;      &gt;<br>
      [rt.jar:1.7.0_45]<br>
     &gt;      &gt;                                                      at<br>
     &gt;      &gt;<br>
     &gt;<br>
      org.jboss.resteasy.core.ConstructorInjectorImpl.construct(ConstructorInjectorImpl.java:148)<br>
     &gt;      &gt;                                                      ...<br>
    19 more<br>
     &gt;      &gt;<br>
     &gt;      &gt;<br>
     &gt;      &gt;                                                 Is the new<br>
    way to<br>
     &gt;     enable<br>
     &gt;      &gt;                                                 Infinispan<br>
     &gt;     different to<br>
     &gt;      &gt;                                                 what we had<br>
     &gt;     earlier? If<br>
     &gt;      &gt;                                                 so, can<br>
    someone<br>
     &gt;     please<br>
     &gt;      &gt;                                                 point out the<br>
     &gt;     correct way?<br>
     &gt;      &gt;<br>
     &gt;      &gt;<br>
     &gt;      &gt;                                                 Regards,<br>
     &gt;      &gt;                                                 Lohitha.<br>
     &gt;      &gt;<br>
     &gt;      &gt;<br>
     &gt;      &gt;<br>
     &gt;      &gt;<br>
     &gt;       _______________________________________________<br>
     &gt;      &gt;                                                 keycloak-user<br>
     &gt;     mailing list<br>
     &gt;      &gt; <a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a><br>
    &lt;mailto:<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a>&gt;<br>
    &lt;mailto:<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a><br>
    &lt;mailto:<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a>&gt;&gt;<br>
     &gt;      &gt;<br>
     &gt;       &lt;mailto:<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a><br>
    &lt;mailto:<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a>&gt;<br></div></div>
    &gt;     &lt;mailto:<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a> &lt;mailto:<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a>&gt;&gt;&gt;<span class=""><br>
    &gt;      &gt;<a href="https://lists.jboss.org/mailman/listinfo/keycloak-user" rel="noreferrer" target="_blank">https://lists.jboss.org/mailman/listinfo/keycloak-user</a><br>
    &gt;     &gt;<br>
    &gt;     &gt;<br>
    &gt;     &gt;<br>
    &gt;     &gt;<br>
    &gt;     &gt;<br>
    &gt;     &gt;<br>
    &gt;     &gt;<br>
    &gt;     &gt;<br>
    &gt;     &gt;<br>
    &gt;     &gt;<br>
    &gt;     &gt;<br>
    &gt;     &gt;<br>
    &gt;     &gt;<br>
    &gt;     &gt;<br>
    &gt;     &gt; _______________________________________________<br>
    &gt;     &gt; keycloak-user mailing list<br>
     &gt;     &gt;<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a><br>
    &lt;mailto:<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a>&gt;<br>
    &lt;mailto:<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a><br>
    &lt;mailto:<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a>&gt;&gt;<br>
    &gt;     &gt;<a href="https://lists.jboss.org/mailman/listinfo/keycloak-user" rel="noreferrer" target="_blank">https://lists.jboss.org/mailman/listinfo/keycloak-user</a><br>
    &gt;     &gt;<br>
    &gt;<br></span><span class="">
    &gt;     --<br>
    &gt;     Bill Burke<br>
    &gt;     JBoss, a division of Red Hat<br>
    &gt;<a href="http://bill.burkecentral.com" rel="noreferrer" target="_blank">http://bill.burkecentral.com</a><br>
    &gt;     _______________________________________________<br>
    &gt;     keycloak-user mailing list<br>
     &gt; <a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a><br>
    &lt;mailto:<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a>&gt;<br></span><span class="">
    &lt;mailto:<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a><br>
    &lt;mailto:<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a>&gt;&gt;<br>
     &gt; <a href="https://lists.jboss.org/mailman/listinfo/keycloak-user" rel="noreferrer" target="_blank">https://lists.jboss.org/mailman/listinfo/keycloak-user</a><br>
     &gt;<br>
     &gt;<br>
<br></span><span class="">
    --<br>
    Bill Burke<br>
    JBoss, a division of Red Hat<br>
    <a href="http://bill.burkecentral.com" rel="noreferrer" target="_blank">http://bill.burkecentral.com</a><br>
    _______________________________________________<br>
    keycloak-user mailing list<br>
    <a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a> &lt;mailto:<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a>&gt;<br>
    <a href="https://lists.jboss.org/mailman/listinfo/keycloak-user" rel="noreferrer" target="_blank">https://lists.jboss.org/mailman/listinfo/keycloak-user</a><br>
<br>
<br>
</span></blockquote><div class="HOEnZb"><div class="h5">
<br>
-- <br>
Bill Burke<br>
JBoss, a division of Red Hat<br>
<a href="http://bill.burkecentral.com" rel="noreferrer" target="_blank">http://bill.burkecentral.com</a><br>
</div></div></blockquote></div><br></div>