<div dir="ltr">Bill, the problem here is that our load tests cannot go beyond a paltry 15-20 concurrent threads with the default set up (invalidation caches with SYNC mode - two HA AWS M3-Medium servers). Hence we're trying to find ways to improve on the performance and the avg response time under a load.<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Nov 30, 2015 at 7:16 PM, Bill Burke <span dir="ltr"><<a href="mailto:bburke@redhat.com" target="_blank">bburke@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">BTW, besides being more scalable, we invalidate rather than replicate to<br>
avoid transmitting sensitive security data over the network.<br>
<span class=""><br>
On 11/30/2015 8:29 AM, Lohitha Chiranjeewa wrote:<br>
> Glad if you could look into the above and check if there are app level<br>
> issues related to invalidations, thanks.<br>
><br>
> One way to get around this through config changes is to make the<br>
> relevant cache modes 'ASYNC' and set up suitable values for 'queue-size'<br>
> and 'queue-flush-interval' depending on your needs. Then the<br>
> invalidations won't happen with each and every call.<br>
><br>
> Obviously there would be problems if caches aren't invalidated in time<br>
> with the above set up, it's up to the devs to come up with a suitable<br>
> set of configs to cater to their needs.<br>
><br>
><br>
> Regards,<br>
> Lohitha.<br>
><br>
><br>
><br>
> On Mon, Nov 30, 2015 at 1:08 PM, Stian Thorgersen <<a href="mailto:sthorger@redhat.com">sthorger@redhat.com</a><br>
</span><span class="">> <mailto:<a href="mailto:sthorger@redhat.com">sthorger@redhat.com</a>>> wrote:<br>
><br>
> It is not expected behavior. Users should only be invalidated if<br>
> changes are made.<br>
><br>
> On 30 November 2015 at 07:01, Lohitha Chiranjeewa <<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a><br>
</span><span class="">> <mailto:<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a>>> wrote:<br>
><br>
> Just tested the same flow extensively with Keycloak 1.2.0 as<br>
> well, and it seems the behavior is the same. Lots of MySQL<br>
> select queries getting executed with each call. Seems there's a<br>
> number of unnecessary cache invalidations going on which causes<br>
> Keycloak to fetch data from the DB over and over.<br>
><br>
> Could you please confirm if this is the expected behavior? As<br>
> far as I can see there's considerable performance degradation<br>
> due to unnecessary cache invalidations.<br>
><br>
><br>
> On Fri, Nov 27, 2015 at 8:42 PM, Lohitha Chiranjeewa<br>
</span><span class="">> <<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a>>> wrote:<br>
><br>
> I'm invoking the following API call:<br>
> https://{host}/auth/admin/realms/xxxxx/users/55ffe851-2d94-460e-88b9-bc7340531b56<br>
> (same realm and user ID over and over again). The two HA<br>
> server(s) are idle apart from serving those calls. And I'm<br>
> seeing the following SQL getting logged in my MySQL log for<br>
> each and every call:<br>
><br>
> select userentity0_.ID as ID1_42_,<br>
> userentity0_.CREATED_TIMESTAMP as CREATED_2_42_,<br>
> userentity0_.EMAIL as EMAIL3_42_,<br>
> userentity0_.EMAIL_CONSTRAINT as EMAIL_CO4_42_,<br>
> userentity0_.EMAIL_VERIFIED as EMAIL_VE5_42_,<br>
> userentity0_.ENABLED as ENABLED6_42_,<br>
> userentity0_.federation_link as federati7_42_,<br>
> userentity0_.FIRST_NAME as FIRST_NA8_42_,<br>
> userentity0_.LAST_NAME as LAST_NAM9_42_,<br>
> userentity0_.REALM_ID as REALM_I10_42_,<br>
> userentity0_.SERVICE_ACCOUNT_CLIENT_LINK as SERVICE11_42_,<br>
> userentity0_.TOTP as TOTP12_42_, userentity0_.USERNAME as<br>
> USERNAM13_42_ from USER_ENTITY userentity0_ where<br>
> userentity0_.ID='55ffe851-2d94-460e-88b9-bc7340531b56' and<br>
> userentity0_.REALM_ID='xxxxx'<br>
><br>
><br>
><br>
> On Fri, Nov 27, 2015 at 8:03 PM, Stian Thorgersen<br>
</span><span class="">> <<a href="mailto:sthorger@redhat.com">sthorger@redhat.com</a> <mailto:<a href="mailto:sthorger@redhat.com">sthorger@redhat.com</a>>> wrote:<br>
><br>
> Strange - what are you doing and what are the SQL queries?<br>
><br>
> On 27 November 2015 at 15:23, Lohitha Chiranjeewa<br>
</span><span class="">> <<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a>>> wrote:<br>
><br>
> Yes Stian, that I understand. But the problem here<br>
> is even if I execute continuous user retrieval calls<br>
> (same user - no other functionality in between),<br>
> still MySQL select queries get executed for each<br>
> call. So there lies an issue isn't it?<br>
><br>
><br>
> On Fri, Nov 27, 2015 at 6:48 PM, Stian Thorgersen<br>
</span>> <<a href="mailto:sthorger@redhat.com">sthorger@redhat.com</a> <mailto:<a href="mailto:sthorger@redhat.com">sthorger@redhat.com</a>>><br>
<span class="">> wrote:<br>
><br>
> Things are still fetched from MySQL. Realms,<br>
> clients, users, etc.. are then kept in the<br>
> cache, but if it changes it's re-loaded from<br>
> MySQL. We use an invalidation cache, not a<br>
> distributed cache.<br>
><br>
> On 27 November 2015 at 14:04, Lohitha<br>
> Chiranjeewa <<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a><br>
</span><span class="">> <mailto:<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a>>> wrote:<br>
><br>
> What I mean is, if it were working, I<br>
> shouldn't see mysql queries getting executed<br>
> right? So my guess is data is still fetched<br>
> from the db instead of the cache.<br>
><br>
> On Nov 27, 2015 5:52 PM, "Stian Thorgersen"<br>
> <<a href="mailto:sthorger@redhat.com">sthorger@redhat.com</a><br>
</span><span class="">> <mailto:<a href="mailto:sthorger@redhat.com">sthorger@redhat.com</a>>> wrote:<br>
><br>
> Yup, so it's working now?<br>
><br>
> On 27 November 2015 at 13:20, Lohitha<br>
> Chiranjeewa <<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a><br>
</span><span class="">> <mailto:<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a>>> wrote:<br>
><br>
> Apologies, keycloak-server.json<br>
> entries should change to:<br>
><br>
> "realm": {<br>
> "provider": "jpa"<br>
> },<br>
><br>
> "user": {<br>
> "provider": "jpa"<br>
> },<br>
><br>
> "userSessionPersister": {<br>
> "provider": "jpa"<br>
> },<br>
><br>
> On Fri, Nov 27, 2015 at 5:49 PM,<br>
> Lohitha Chiranjeewa<br>
> <<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a><br>
</span><div><div class="h5">> <mailto:<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a>>> wrote:<br>
><br>
> Hi Stian,<br>
><br>
> As per the migration guide, I<br>
> should have Infinispan up and<br>
> running for realms, users and<br>
> user sessions without doing any<br>
> specific changes.<br>
> keycloak-server.json was<br>
> reverted back to have the<br>
> following entries:<br>
> ...<br>
> "realm": {<br>
> "provider": "infinispan"<br>
> },<br>
><br>
> "user": {<br>
> "provider": "infinispan"<br>
> },<br>
><br>
> "userSessionPersister": {<br>
> "provider": "infinispan"<br>
> },<br>
> ...<br>
><br>
> In the Admin Console I have both<br>
> Realm Cache and User Cache<br>
> enables. I see certain<br>
> Infinispan related logs getting<br>
> logged as well.<br>
><br>
> However, at the same time, I see<br>
> MySQL queries getting executed<br>
> for all user retrieval API<br>
> invocations (even if the same<br>
> user is retrieved continuously):<br>
> ...<br>
> select userentity0_.ID as<br>
> ID1_42_,<br>
> userentity0_.CREATED_TIMESTAMP<br>
> as CREATED_2_42_,<br>
> userentity0_.EMAIL as<br>
> EMAIL3_42_,<br>
> userentity0_.EMAIL_CONSTRAINT as<br>
> EMAIL_CO4_42_,<br>
> userentity0_.EMAIL_VERIFIED as<br>
> EMAIL_VE5_42_,<br>
> userentity0_.ENABLED as<br>
> ENABLED6_42_,<br>
> userentity0_.federation_link as<br>
> federati7_42_,<br>
> userentity0_.FIRST_NAME as<br>
> FIRST_NA8_42_,<br>
> userentity0_.LAST_NAME as<br>
> LAST_NAM9_42_,<br>
> userentity0_.REALM_ID as<br>
> REALM_I10_42_,<br>
> userentity0_.SERVICE_ACCOUNT_CLIENT_LINK<br>
> as SERVICE11_42_,<br>
> userentity0_.TOTP as TOTP12_42_,<br>
> userentity0_.USERNAME as<br>
> USERNAM13_42_ from USER_ENTITY<br>
> userentity0_ where<br>
> userentity0_.ID='55ffe851-2d94-460e-88b9-bc7340531b56'<br>
> and userentity0_.REALM_ID='xxxxx'<br>
> ...<br>
><br>
> So it seems something is wrong<br>
> here. Could you point out any<br>
> areas that I could further look<br>
> into?<br>
><br>
><br>
> Regards,<br>
> Lohitha.<br>
><br>
> On Thu, Nov 26, 2015 at 7:58 PM,<br>
> Stian Thorgersen<br>
> <<a href="mailto:sthorger@redhat.com">sthorger@redhat.com</a><br>
</div></div><span class="">> <mailto:<a href="mailto:sthorger@redhat.com">sthorger@redhat.com</a>>> wrote:<br>
><br>
> Please read the migration guide<br>
><br>
> On 26 November 2015 at<br>
> 14:53, Lohitha Chiranjeewa<br>
> <<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a><br>
</span>> <mailto:<a href="mailto:kalc04@gmail.com">kalc04@gmail.com</a>>><br>
<div><div class="h5">> wrote:<br>
><br>
> Hi,<br>
><br>
> We're in the process of<br>
> assessing the impact on<br>
> upgrading from Keycloak<br>
> 1.2.0 to 1.6.1. We came<br>
> across an issue when<br>
> trying to enable<br>
> Infinispan cache through<br>
> the keycloak-server.json<br>
> file as we used to do in<br>
> 1.2.0.<br>
><br>
> We have the following<br>
> entries in 1.6.1:<br>
> "realm": {<br>
> "provider":<br>
> "infinispan"<br>
> },<br>
><br>
> "user": {<br>
> "provider":<br>
> "infinispan"<br>
> },<br>
><br>
><br>
> "userSessionPersister": {<br>
> "provider":<br>
> "infinispan"<br>
> },<br>
> .........<br>
><br>
> "connectionsInfinispan": {<br>
> "default" : {<br>
><br>
> "cacheContainer" :<br>
> "java:comp/env/infinispan/Keycloak"<br>
> }<br>
> }<br>
><br>
> All configurations in<br>
> 1.6.1 standalone-ha.xml<br>
> file remains comparable<br>
> (and correct to the best<br>
> of our knowledge) with<br>
> the ones in 1.2.0.<br>
><br>
> With the above configs,<br>
> when we start the<br>
> Keycloak service the<br>
> following error(s) get<br>
> logged:<br>
><br>
> 18:03:31,610 ERROR<br>
> [org.jboss.msc.service.fail]<br>
> (ServerService Thread<br>
> Pool -- 64) MSC000001:<br>
> Failed to start service<br>
> jboss.undertow.deployment.default-server.default-host./auth:<br>
> org.jboss.msc.service.StartException<br>
> in service<br>
> jboss.undertow.deployment.default-server.default-host./auth:<br>
> java.lang.RuntimeException:<br>
> Failed to construct<br>
> public<br>
> org.keycloak.services.resources.KeycloakApplication(javax.servlet.ServletContext,org.jboss.resteasy.core.Dispatcher)<br>
> at<br>
> org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:85)<br>
> at<br>
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)<br>
> [rt.jar:1.7.0_45]<br>
> at<br>
> java.util.concurrent.FutureTask.run(FutureTask.java:262)<br>
> [rt.jar:1.7.0_45]<br>
> at<br>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)<br>
> [rt.jar:1.7.0_45]<br>
> at<br>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)<br>
> [rt.jar:1.7.0_45]<br>
> at<br>
> java.lang.Thread.run(Thread.java:744)<br>
> [rt.jar:1.7.0_45]<br>
> at<br>
> org.jboss.threads.JBossThread.run(JBossThread.java:320)<br>
> [jboss-threads-2.2.0.Final.jar:2.2.0.Final]<br>
> Caused by:<br>
> java.lang.RuntimeException:<br>
> Failed to construct<br>
> public<br>
> org.keycloak.services.resources.KeycloakApplication(javax.servlet.ServletContext,org.jboss.resteasy.core.Dispatcher)<br>
> at<br>
> org.jboss.resteasy.core.ConstructorInjectorImpl.construct(ConstructorInjectorImpl.java:160)<br>
> at<br>
> org.jboss.resteasy.spi.ResteasyProviderFactory.createProviderInstance(ResteasyProviderFactory.java:2211)<br>
> at<br>
> org.jboss.resteasy.spi.ResteasyDeployment.createApplication(ResteasyDeployment.java:295)<br>
> at<br>
> org.jboss.resteasy.spi.ResteasyDeployment.start(ResteasyDeployment.java:236)<br>
> at<br>
> org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.init(ServletContainerDispatcher.java:112)<br>
> at<br>
> org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.init(HttpServletDispatcher.java:36)<br>
> at<br>
> io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:117)<br>
> at<br>
> org.wildfly.extension.undertow.security.RunAsLifecycleInterceptor.init(RunAsLifecycleInterceptor.java:78)<br>
> at<br>
> io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:103)<br>
> at<br>
> io.undertow.servlet.core.ManagedServlet$DefaultInstanceStrategy.start(ManagedServlet.java:230)<br>
> at<br>
> io.undertow.servlet.core.ManagedServlet.createServlet(ManagedServlet.java:131)<br>
> at<br>
> io.undertow.servlet.core.DeploymentManagerImpl.start(DeploymentManagerImpl.java:511)<br>
> at<br>
> org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:101)<br>
> at<br>
> org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:82)<br>
> ... 6 more<br>
> Caused by:<br>
> java.lang.RuntimeException:<br>
> Failed to find provider<br>
> infinispan for realm<br>
> at<br>
> org.keycloak.services.DefaultKeycloakSessionFactory.init(DefaultKeycloakSessionFactory.java:66)<br>
> at<br>
> org.keycloak.services.resources.KeycloakApplication.createSessionFactory(KeycloakApplication.java:162)<br>
> at<br>
> org.keycloak.services.resources.KeycloakApplication.<init>(KeycloakApplication.java:62)<br>
> at<br>
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native<br>
> Method) [rt.jar:1.7.0_45]<br>
> at<br>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)<br>
> [rt.jar:1.7.0_45]<br>
> at<br>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)<br>
> [rt.jar:1.7.0_45]<br>
> at<br>
> java.lang.reflect.Constructor.newInstance(Constructor.java:526)<br>
> [rt.jar:1.7.0_45]<br>
> at<br>
> org.jboss.resteasy.core.ConstructorInjectorImpl.construct(ConstructorInjectorImpl.java:148)<br>
> ... 19 more<br>
><br>
><br>
> Is the new way to enable<br>
> Infinispan different to<br>
> what we had earlier? If<br>
> so, can someone please<br>
> point out the correct way?<br>
><br>
><br>
> Regards,<br>
> Lohitha.<br>
><br>
><br>
><br>
> _______________________________________________<br>
> keycloak-user mailing list<br>
> <a href="mailto:keycloak-user@lists.jboss.org">keycloak-user@lists.jboss.org</a><br>
</div></div>> <mailto:<a href="mailto:keycloak-user@lists.jboss.org">keycloak-user@lists.jboss.org</a>><br>
> <a href="https://lists.jboss.org/mailman/listinfo/keycloak-user" rel="noreferrer" target="_blank">https://lists.jboss.org/mailman/listinfo/keycloak-user</a><br>
<span class="im HOEnZb">><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
> _______________________________________________<br>
> keycloak-user mailing list<br>
> <a href="mailto:keycloak-user@lists.jboss.org">keycloak-user@lists.jboss.org</a><br>
> <a href="https://lists.jboss.org/mailman/listinfo/keycloak-user" rel="noreferrer" target="_blank">https://lists.jboss.org/mailman/listinfo/keycloak-user</a><br>
><br>
<br>
</span><span class="HOEnZb"><font color="#888888">--<br>
Bill Burke<br>
JBoss, a division of Red Hat<br>
<a href="http://bill.burkecentral.com" rel="noreferrer" target="_blank">http://bill.burkecentral.com</a><br>
</font></span><div class="HOEnZb"><div class="h5">_______________________________________________<br>
keycloak-user mailing list<br>
<a href="mailto:keycloak-user@lists.jboss.org">keycloak-user@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/keycloak-user" rel="noreferrer" target="_blank">https://lists.jboss.org/mailman/listinfo/keycloak-user</a><br>
</div></div></blockquote></div><br></div>