<div dir="ltr">If you can try building master and enable trace logging for "org.keycloak.models.cache.infinispan.InfinispanCacheUserProviderFactory". It will print log statements when entries are removed/invalidated in the cache.</div><div class="gmail_extra"><br><div class="gmail_quote">On 30 November 2015 at 15:19, Bill Burke <span dir="ltr"><<a href="mailto:bburke@redhat.com" target="_blank">bburke@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Can you describe your load tests please? What kind of authentication<br>
are you doing? The only thing that would cause invalidations is HOTP.<br>
<span class=""><br>
On 11/30/2015 9:13 AM, Lohitha Chiranjeewa wrote:<br>
> Bill, the problem here is that our load tests cannot go beyond a paltry<br>
> 15-20 concurrent threads with the default set up (invalidation caches<br>
> with SYNC mode - two HA AWS M3-Medium servers). Hence we're trying to<br>
> find ways to improve on the performance and the avg response time under<br>
> a load.<br>
><br>
> On Mon, Nov 30, 2015 at 7:16 PM, Bill Burke <<a href="mailto:bburke@redhat.com">bburke@redhat.com</a><br>
</span><span class="">> <mailto:<a href="mailto:bburke@redhat.com">bburke@redhat.com</a>>> wrote:<br>
><br>
> BTW, besides being more scalable, we invalidate rather than replicate to<br>
> avoid transmitting sensitive security data over the network.<br>
><br>
> On 11/30/2015 8:29 AM, Lohitha Chiranjeewa wrote:<br>
> > Glad if you could look into the above and check if there are app level<br>
> > issues related to invalidations, thanks.<br>
> ><br>
> > One way to get around this through config changes is to make the<br>
> > relevant cache modes 'ASYNC' and set up suitable values for 'queue-size'<br>
> > and 'queue-flush-interval' depending on your needs. Then the<br>
> > invalidations won't happen with each and every call.<br>
> ><br>
> > Obviously there would be problems if caches aren't invalidated in time<br>
> > with the above set up, it's up to the devs to come up with a suitable<br>
> > set of configs to cater to their needs.<br>
> ><br>
> ><br>
> > Regards,<br>
> > Lohitha.<br>
> ><br>
> ><br>
> ><br>
> > On Mon, Nov 30, 2015 at 1:08 PM, Stian Thorgersen <<a href="mailto:sthorger@redhat.com">sthorger@redhat.com</a> <mailto:<a href="mailto:sthorger@redhat.com">sthorger@redhat.com</a>><br>
</span><span class="">> > <mailto:<a href="mailto:sthorger@redhat.com">sthorger@redhat.com</a> <mailto:<a href="mailto:sthorger@redhat.com">sthorger@redhat.com</a>>>> wrote:<br>
> ><br>
> > It is not expected behavior. Users should only be invalidated if<br>
> > changes are made.<br>
> ><br>
> > On 30 November 2015 at 07:01, Lohitha Chiranjeewa <<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a>><br>
</span><span class="">> > <mailto:<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a>>>> wrote:<br>
> ><br>
> > Just tested the same flow extensively with Keycloak 1.2.0 as<br>
> > well, and it seems the behavior is the same. Lots of MySQL<br>
> > select queries getting executed with each call. Seems there's a<br>
> > number of unnecessary cache invalidations going on which causes<br>
> > Keycloak to fetch data from the DB over and over.<br>
> ><br>
> > Could you please confirm if this is the expected behavior? As<br>
> > far as I can see there's considerable performance degradation<br>
> > due to unnecessary cache invalidations.<br>
> ><br>
> ><br>
> > On Fri, Nov 27, 2015 at 8:42 PM, Lohitha Chiranjeewa<br>
</span>> > <<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a>> <mailto:<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a><br>
<span class="">> <mailto:<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a>>>> wrote:<br>
> ><br>
> > I'm invoking the following API call:<br>
> > https://{host}/auth/admin/realms/xxxxx/users/55ffe851-2d94-460e-88b9-bc7340531b56<br>
> > (same realm and user ID over and over again). The two HA<br>
> > server(s) are idle apart from serving those calls. And I'm<br>
> > seeing the following SQL getting logged in my MySQL log for<br>
> > each and every call:<br>
> ><br>
> > select userentity0_.ID as ID1_42_,<br>
> > userentity0_.CREATED_TIMESTAMP as CREATED_2_42_,<br>
> > userentity0_.EMAIL as EMAIL3_42_,<br>
> > userentity0_.EMAIL_CONSTRAINT as EMAIL_CO4_42_,<br>
> > userentity0_.EMAIL_VERIFIED as EMAIL_VE5_42_,<br>
> > userentity0_.ENABLED as ENABLED6_42_,<br>
> > userentity0_.federation_link as federati7_42_,<br>
> > userentity0_.FIRST_NAME as FIRST_NA8_42_,<br>
> > userentity0_.LAST_NAME as LAST_NAM9_42_,<br>
> > userentity0_.REALM_ID as REALM_I10_42_,<br>
> > userentity0_.SERVICE_ACCOUNT_CLIENT_LINK as SERVICE11_42_,<br>
> > userentity0_.TOTP as TOTP12_42_, userentity0_.USERNAME as<br>
> > USERNAM13_42_ from USER_ENTITY userentity0_ where<br>
> > userentity0_.ID='55ffe851-2d94-460e-88b9-bc7340531b56' and<br>
> > userentity0_.REALM_ID='xxxxx'<br>
> ><br>
> ><br>
> ><br>
> > On Fri, Nov 27, 2015 at 8:03 PM, Stian Thorgersen<br>
> > <<a href="mailto:sthorger@redhat.com">sthorger@redhat.com</a> <mailto:<a href="mailto:sthorger@redhat.com">sthorger@redhat.com</a>><br>
</span><span class="">> <mailto:<a href="mailto:sthorger@redhat.com">sthorger@redhat.com</a> <mailto:<a href="mailto:sthorger@redhat.com">sthorger@redhat.com</a>>>> wrote:<br>
> ><br>
> > Strange - what are you doing and what are the SQL queries?<br>
> ><br>
> > On 27 November 2015 at 15:23, Lohitha Chiranjeewa<br>
</span>> > <<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a>> <mailto:<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a><br>
<span class="">> <mailto:<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a>>>> wrote:<br>
> ><br>
> > Yes Stian, that I understand. But the problem here<br>
> > is even if I execute continuous user retrieval calls<br>
> > (same user - no other functionality in between),<br>
> > still MySQL select queries get executed for each<br>
> > call. So there lies an issue isn't it?<br>
> ><br>
> ><br>
> > On Fri, Nov 27, 2015 at 6:48 PM, Stian Thorgersen<br>
> > <<a href="mailto:sthorger@redhat.com">sthorger@redhat.com</a><br>
</span>> <mailto:<a href="mailto:sthorger@redhat.com">sthorger@redhat.com</a>> <mailto:<a href="mailto:sthorger@redhat.com">sthorger@redhat.com</a><br>
<span class="">> <mailto:<a href="mailto:sthorger@redhat.com">sthorger@redhat.com</a>>>><br>
> > wrote:<br>
> ><br>
> > Things are still fetched from MySQL. Realms,<br>
> > clients, users, etc.. are then kept in the<br>
> > cache, but if it changes it's re-loaded from<br>
> > MySQL. We use an invalidation cache, not a<br>
> > distributed cache.<br>
> ><br>
> > On 27 November 2015 at 14:04, Lohitha<br>
> > Chiranjeewa <<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a>><br>
</span><span class="">> > <mailto:<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a>>>> wrote:<br>
> ><br>
> > What I mean is, if it were working, I<br>
> > shouldn't see mysql queries getting executed<br>
> > right? So my guess is data is still fetched<br>
> > from the db instead of the cache.<br>
> ><br>
> > On Nov 27, 2015 5:52 PM, "Stian Thorgersen"<br>
> > <<a href="mailto:sthorger@redhat.com">sthorger@redhat.com</a> <mailto:<a href="mailto:sthorger@redhat.com">sthorger@redhat.com</a>><br>
</span><span class="">> > <mailto:<a href="mailto:sthorger@redhat.com">sthorger@redhat.com</a> <mailto:<a href="mailto:sthorger@redhat.com">sthorger@redhat.com</a>>>> wrote:<br>
> ><br>
> > Yup, so it's working now?<br>
> ><br>
> > On 27 November 2015 at 13:20, Lohitha<br>
> > Chiranjeewa <<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a>><br>
</span><span class="">> > <mailto:<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a>>>> wrote:<br>
> ><br>
> > Apologies, keycloak-server.json<br>
> > entries should change to:<br>
> ><br>
> > "realm": {<br>
> > "provider": "jpa"<br>
> > },<br>
> ><br>
> > "user": {<br>
> > "provider": "jpa"<br>
> > },<br>
> ><br>
> > "userSessionPersister": {<br>
> > "provider": "jpa"<br>
> > },<br>
> ><br>
> > On Fri, Nov 27, 2015 at 5:49 PM,<br>
> > Lohitha Chiranjeewa<br>
> > <<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a>><br>
</span>> > <mailto:<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a><br>
<div><div class="h5">> <mailto:<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a>>>> wrote:<br>
> ><br>
> > Hi Stian,<br>
> ><br>
> > As per the migration guide, I<br>
> > should have Infinispan up and<br>
> > running for realms, users and<br>
> > user sessions without<br>
> doing any<br>
> > specific changes.<br>
> > keycloak-server.json was<br>
> > reverted back to have the<br>
> > following entries:<br>
> > ...<br>
> > "realm": {<br>
> > "provider":<br>
> "infinispan"<br>
> > },<br>
> ><br>
> > "user": {<br>
> > "provider":<br>
> "infinispan"<br>
> > },<br>
> ><br>
> ><br>
> "userSessionPersister": {<br>
> > "provider":<br>
> "infinispan"<br>
> > },<br>
> > ...<br>
> ><br>
> > In the Admin Console I<br>
> have both<br>
> > Realm Cache and User Cache<br>
> > enables. I see certain<br>
> > Infinispan related logs<br>
> getting<br>
> > logged as well.<br>
> ><br>
> > However, at the same<br>
> time, I see<br>
> > MySQL queries getting<br>
> executed<br>
> > for all user retrieval API<br>
> > invocations (even if the same<br>
> > user is retrieved<br>
> continuously):<br>
> > ...<br>
> > select userentity0_.ID as<br>
> > ID1_42_,<br>
> ><br>
> userentity0_.CREATED_TIMESTAMP<br>
> > as CREATED_2_42_,<br>
> > userentity0_.EMAIL as<br>
> > EMAIL3_42_,<br>
> ><br>
> userentity0_.EMAIL_CONSTRAINT as<br>
> > EMAIL_CO4_42_,<br>
> ><br>
> userentity0_.EMAIL_VERIFIED as<br>
> > EMAIL_VE5_42_,<br>
> > userentity0_.ENABLED as<br>
> > ENABLED6_42_,<br>
> ><br>
> userentity0_.federation_link as<br>
> > federati7_42_,<br>
> > userentity0_.FIRST_NAME as<br>
> > FIRST_NA8_42_,<br>
> > userentity0_.LAST_NAME as<br>
> > LAST_NAM9_42_,<br>
> > userentity0_.REALM_ID as<br>
> > REALM_I10_42_,<br>
> ><br>
> userentity0_.SERVICE_ACCOUNT_CLIENT_LINK<br>
> > as SERVICE11_42_,<br>
> > userentity0_.TOTP as<br>
> TOTP12_42_,<br>
> > userentity0_.USERNAME as<br>
> > USERNAM13_42_ from<br>
> USER_ENTITY<br>
> > userentity0_ where<br>
> ><br>
> userentity0_.ID='55ffe851-2d94-460e-88b9-bc7340531b56'<br>
> > and<br>
> userentity0_.REALM_ID='xxxxx'<br>
> > ...<br>
> ><br>
> > So it seems something is<br>
> wrong<br>
> > here. Could you point out any<br>
> > areas that I could<br>
> further look<br>
> > into?<br>
> ><br>
> ><br>
> > Regards,<br>
> > Lohitha.<br>
> ><br>
> > On Thu, Nov 26, 2015 at<br>
> 7:58 PM,<br>
> > Stian Thorgersen<br>
> > <<a href="mailto:sthorger@redhat.com">sthorger@redhat.com</a><br>
> <mailto:<a href="mailto:sthorger@redhat.com">sthorger@redhat.com</a>><br>
</div></div><span class="">> > <mailto:<a href="mailto:sthorger@redhat.com">sthorger@redhat.com</a> <mailto:<a href="mailto:sthorger@redhat.com">sthorger@redhat.com</a>>>> wrote:<br>
> ><br>
> > Please read the migration guide<br>
> ><br>
> > On 26 November 2015 at<br>
> > 14:53, Lohitha Chiranjeewa<br>
> > <<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a>><br>
> ><br>
</span>> <mailto:<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a>>>><br>
<div><div class="h5">> > wrote:<br>
> ><br>
> > Hi,<br>
> ><br>
> > We're in the<br>
> process of<br>
> > assessing the<br>
> impact on<br>
> > upgrading from<br>
> Keycloak<br>
> > 1.2.0 to 1.6.1.<br>
> We came<br>
> > across an issue when<br>
> > trying to enable<br>
> > Infinispan cache<br>
> through<br>
> > the<br>
> keycloak-server.json<br>
> > file as we used<br>
> to do in<br>
> > 1.2.0.<br>
> ><br>
> > We have the following<br>
> > entries in 1.6.1:<br>
> > "realm": {<br>
> > "provider":<br>
> > "infinispan"<br>
> > },<br>
> ><br>
> > "user": {<br>
> > "provider":<br>
> > "infinispan"<br>
> > },<br>
> ><br>
> ><br>
> ><br>
> "userSessionPersister": {<br>
> > "provider":<br>
> > "infinispan"<br>
> > },<br>
> > .........<br>
> ><br>
> ><br>
> "connectionsInfinispan": {<br>
> ><br>
> "default" : {<br>
> ><br>
> > "cacheContainer" :<br>
> ><br>
> "java:comp/env/infinispan/Keycloak"<br>
> > }<br>
> > }<br>
> ><br>
> > All configurations in<br>
> > 1.6.1<br>
> standalone-ha.xml<br>
> > file remains<br>
> comparable<br>
> > (and correct to<br>
> the best<br>
> > of our knowledge)<br>
> with<br>
> > the ones in 1.2.0.<br>
> ><br>
> > With the above<br>
> configs,<br>
> > when we start the<br>
> > Keycloak service the<br>
> > following<br>
> error(s) get<br>
> > logged:<br>
> ><br>
> > 18:03:31,610 ERROR<br>
> ><br>
> [org.jboss.msc.service.fail]<br>
> > (ServerService Thread<br>
> > Pool -- 64)<br>
> MSC000001:<br>
> > Failed to start<br>
> service<br>
> ><br>
> jboss.undertow.deployment.default-server.default-host./auth:<br>
> ><br>
> org.jboss.msc.service.StartException<br>
> > in service<br>
> ><br>
> jboss.undertow.deployment.default-server.default-host./auth:<br>
> ><br>
> java.lang.RuntimeException:<br>
> > Failed to construct<br>
> > public<br>
> ><br>
> org.keycloak.services.resources.KeycloakApplication(javax.servlet.ServletContext,org.jboss.resteasy.core.Dispatcher)<br>
> > at<br>
> ><br>
> org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:85)<br>
> > at<br>
> ><br>
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)<br>
> > [rt.jar:1.7.0_45]<br>
> > at<br>
> ><br>
> java.util.concurrent.FutureTask.run(FutureTask.java:262)<br>
> > [rt.jar:1.7.0_45]<br>
> > at<br>
> ><br>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)<br>
> > [rt.jar:1.7.0_45]<br>
> > at<br>
> ><br>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)<br>
> > [rt.jar:1.7.0_45]<br>
> > at<br>
> ><br>
> java.lang.Thread.run(Thread.java:744)<br>
> > [rt.jar:1.7.0_45]<br>
> > at<br>
> ><br>
> org.jboss.threads.JBossThread.run(JBossThread.java:320)<br>
> ><br>
> [jboss-threads-2.2.0.Final.jar:2.2.0.Final]<br>
> > Caused by:<br>
> ><br>
> java.lang.RuntimeException:<br>
> > Failed to construct<br>
> > public<br>
> ><br>
> org.keycloak.services.resources.KeycloakApplication(javax.servlet.ServletContext,org.jboss.resteasy.core.Dispatcher)<br>
> > at<br>
> ><br>
> org.jboss.resteasy.core.ConstructorInjectorImpl.construct(ConstructorInjectorImpl.java:160)<br>
> > at<br>
> ><br>
> org.jboss.resteasy.spi.ResteasyProviderFactory.createProviderInstance(ResteasyProviderFactory.java:2211)<br>
> > at<br>
> ><br>
> org.jboss.resteasy.spi.ResteasyDeployment.createApplication(ResteasyDeployment.java:295)<br>
> > at<br>
> ><br>
> org.jboss.resteasy.spi.ResteasyDeployment.start(ResteasyDeployment.java:236)<br>
> > at<br>
> ><br>
> org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.init(ServletContainerDispatcher.java:112)<br>
> > at<br>
> ><br>
> org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.init(HttpServletDispatcher.java:36)<br>
> > at<br>
> ><br>
> io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:117)<br>
> > at<br>
> ><br>
> org.wildfly.extension.undertow.security.RunAsLifecycleInterceptor.init(RunAsLifecycleInterceptor.java:78)<br>
> > at<br>
> ><br>
> io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:103)<br>
> > at<br>
> ><br>
> io.undertow.servlet.core.ManagedServlet$DefaultInstanceStrategy.start(ManagedServlet.java:230)<br>
> > at<br>
> ><br>
> io.undertow.servlet.core.ManagedServlet.createServlet(ManagedServlet.java:131)<br>
> > at<br>
> ><br>
> io.undertow.servlet.core.DeploymentManagerImpl.start(DeploymentManagerImpl.java:511)<br>
> > at<br>
> ><br>
> org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:101)<br>
> > at<br>
> ><br>
> org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:82)<br>
> > ... 6 more<br>
> > Caused by:<br>
> ><br>
> java.lang.RuntimeException:<br>
> > Failed to find<br>
> provider<br>
> > infinispan for realm<br>
> > at<br>
> ><br>
> org.keycloak.services.DefaultKeycloakSessionFactory.init(DefaultKeycloakSessionFactory.java:66)<br>
> > at<br>
> ><br>
> org.keycloak.services.resources.KeycloakApplication.createSessionFactory(KeycloakApplication.java:162)<br>
> > at<br>
> ><br>
> org.keycloak.services.resources.KeycloakApplication.<init>(KeycloakApplication.java:62)<br>
> > at<br>
> ><br>
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native<br>
> > Method)<br>
> [rt.jar:1.7.0_45]<br>
> > at<br>
> ><br>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)<br>
> > [rt.jar:1.7.0_45]<br>
> > at<br>
> ><br>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)<br>
> > [rt.jar:1.7.0_45]<br>
> > at<br>
> ><br>
> java.lang.reflect.Constructor.newInstance(Constructor.java:526)<br>
> > [rt.jar:1.7.0_45]<br>
> > at<br>
> ><br>
> org.jboss.resteasy.core.ConstructorInjectorImpl.construct(ConstructorInjectorImpl.java:148)<br>
> > ... 19 more<br>
> ><br>
> ><br>
> > Is the new way to<br>
> enable<br>
> > Infinispan<br>
> different to<br>
> > what we had<br>
> earlier? If<br>
> > so, can someone<br>
> please<br>
> > point out the<br>
> correct way?<br>
> ><br>
> ><br>
> > Regards,<br>
> > Lohitha.<br>
> ><br>
> ><br>
> ><br>
> ><br>
> _______________________________________________<br>
> > keycloak-user<br>
> mailing list<br>
> > <a href="mailto:keycloak-user@lists.jboss.org">keycloak-user@lists.jboss.org</a> <mailto:<a href="mailto:keycloak-user@lists.jboss.org">keycloak-user@lists.jboss.org</a>><br>
> ><br>
</div></div>> <mailto:<a href="mailto:keycloak-user@lists.jboss.org">keycloak-user@lists.jboss.org</a><br>
<span class="">> <mailto:<a href="mailto:keycloak-user@lists.jboss.org">keycloak-user@lists.jboss.org</a>>><br>
> > <a href="https://lists.jboss.org/mailman/listinfo/keycloak-user" rel="noreferrer" target="_blank">https://lists.jboss.org/mailman/listinfo/keycloak-user</a><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > keycloak-user mailing list<br>
</span>> ><a href="mailto:keycloak-user@lists.jboss.org">keycloak-user@lists.jboss.org</a> <mailto:<a href="mailto:keycloak-user@lists.jboss.org">keycloak-user@lists.jboss.org</a>><br>
<span class="">> ><a href="https://lists.jboss.org/mailman/listinfo/keycloak-user" rel="noreferrer" target="_blank">https://lists.jboss.org/mailman/listinfo/keycloak-user</a><br>
> ><br>
><br>
> --<br>
> Bill Burke<br>
> JBoss, a division of Red Hat<br>
> <a href="http://bill.burkecentral.com" rel="noreferrer" target="_blank">http://bill.burkecentral.com</a><br>
> _______________________________________________<br>
> keycloak-user mailing list<br>
</span>> <a href="mailto:keycloak-user@lists.jboss.org">keycloak-user@lists.jboss.org</a> <mailto:<a href="mailto:keycloak-user@lists.jboss.org">keycloak-user@lists.jboss.org</a>><br>
<div class="HOEnZb"><div class="h5">> <a href="https://lists.jboss.org/mailman/listinfo/keycloak-user" rel="noreferrer" target="_blank">https://lists.jboss.org/mailman/listinfo/keycloak-user</a><br>
><br>
><br>
<br>
--<br>
Bill Burke<br>
JBoss, a division of Red Hat<br>
<a href="http://bill.burkecentral.com" rel="noreferrer" target="_blank">http://bill.burkecentral.com</a><br>
_______________________________________________<br>
keycloak-user mailing list<br>
<a href="mailto:keycloak-user@lists.jboss.org">keycloak-user@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/keycloak-user" rel="noreferrer" target="_blank">https://lists.jboss.org/mailman/listinfo/keycloak-user</a><br>
</div></div></blockquote></div><br></div>