[keycloak-user] Clustering on localhost with shared DB

Stian Thorgersen stian at redhat.com
Mon Apr 27 09:02:27 EDT 2015 


----- Original Message -----
> From: "Libor Krzyžanek" <lkrzyzan at redhat.com>
> To: "Marek Posolda" <mposolda at redhat.com>
> Cc: "keycloak-user" <keycloak-user at lists.jboss.org>
> Sent: Monday, 27 April, 2015 2:55:43 PM
> Subject: Re: [keycloak-user] Clustering on localhost with shared DB
> 
> Hi,
> yeah this helps little bit:
> <invalidation-cache name="realms" mode="SYNC"/>
> <invalidation-cache name="users" mode="SYNC"/>
> <distributed-cache name="sessions" mode="SYNC" owners="2" segments="60" >
> <state-transfer enabled="true" />
> </distributed-cache>
> <distributed-cache name="loginFailures" mode="SYNC" owners="2" segments="60"
> >
> <state-transfer enabled="true" />
> </distributed-cache>
> 
> When both caches on both nodes are up then syncing works fine.
> Also /sessions works OK.
> 
> But I’m still facing issue no 1.
> 
> When node is up I see in logs this:
> 
> 14:51:19,088 INFO [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss
> EAP 6.4.0.GA (AS 7.5.0.Final-redhat-21) started in 18527ms - Started 242 of
> 347 services (141 services are lazy, passive or on-demand)
> 
> Caches are initialised after first hit not after KC start

Have you tried putting start="EAGER" on both the cache-container and all caches in standalone.xml?

> 
> I’m talking about this in log:
> 14:51:52,597 INFO [org.infinispan.jmx.CacheJmxRegistration]
> (http-/127.0.0.1:8080-1) ISPN000031: MBeans were successfully registered to
> the platform MBean server.
> 14:51:52,605 INFO [org.jboss.as.clustering.infinispan]
> (http-/127.0.0.1:8080-1) JBAS010281: Started users cache from keycloak
> container
> 14:51:52,710 INFO [org.infinispan.jmx.CacheJmxRegistration]
> (http-/127.0.0.1:8080-2) ISPN000031: MBeans were successfully registered to
> the platform MBean server.
> 14:51:52,815 INFO [org.jboss.as.clustering.infinispan]
> (http-/127.0.0.1:8080-2) JBAS010281: Started sessions cache from keycloak
> container
> 14:51:52,822 INFO [org.infinispan.jmx.CacheJmxRegistration]
> (http-/127.0.0.1:8080-2) ISPN000031: MBeans were successfully registered to
> the platform MBean server.
> 14:51:52,847 INFO [org.jboss.as.clustering.infinispan]
> (http-/127.0.0.1:8080-2) JBAS010281: Started loginFailures cache from
> keycloak container
> 
> 
> 
> Thanks,
> 
> Libor Krzyžanek
> jboss.org Development Team
> 
> 
> 
> 
> On 27 Apr 2015, at 14:24, Marek Posolda < mposolda at redhat.com > wrote:
> 
> On 27.4.2015 13:50, Libor Krzyžanek wrote:
> 
> 
> Hi,
> I have now apache webproxy with this configuration:
> <Proxy *>
> Order allow,deny
> Allow from all
> </Proxy>
> <Proxy balancer://app/ >
> BalancerMember http://localhost:8080 route=app02
> BalancerMember http://localhost:8180 route=app03
> ProxySet lbmethod=byrequests
> </Proxy>
> ProxyPass /balancer-manager !
> ProxyPass /server-status !
> ProxyPass /server-info !
> ProxyPass / balancer://app/
> ProxyPassReverse / balancer://app/
> 
> It looks it helped.
> When I have started both nodes and I see that caches on both nodes are
> started then everything is fine.
> Scenario: When I login to node1, then stop node1, then I’m redirected to
> node2 and I’m still logged in. Great!
> 
> But I see two issues right now:
> 1. Caches are replicated to newly started node too late.
> Scenario is:
> 1. start node1, log in.
> 2. start node2, wait till you see that node1 knows new node and node2 is
> fully started
> 3. killl node1.
> 
> Then I’m redirected to login page.
> 
> This happens really only when no request hits newly started node2. If I do
> few reloads in browser before I kill node1 then I see in logs that those
> infinispan caches are created on node2 and fully replicated.
> 
> Is it related to “start = EAGER” ?
> Will it help if you use in standalone-ha.xml the config like this? :
> 
> <distributed-cache name="sessions" mode="SYNC" owners="2" segments="60" >
> <state-transfer enabled="true" />
> </distributed-cache>
> 
> 
> 
> 
> 
> 
> 2. Weird thing is on /account/session page (
> http://localhost/auth/realms/cluster-test/account/sessions ).
> 
> I got:
> 
> 13:30:50,291 ERROR
> [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/auth].[Keycloak
> REST Interface]] (http-/127.0.0.1:8080-2) JBWEB000236: Servlet.service() for
> servlet Keycloak REST Interface threw exception: java.lang.RuntimeException:
> request path: /auth/realms/cluster-test/account/sessions
> at
> org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:54)
> [keycloak-services-1.2.0.Beta1.jar:1.2.0.Beta1]
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
> [jbossweb-7.5.7.Final-redhat-1.jar:7.5.7.Final-redhat-1]
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
> [jbossweb-7.5.7.Final-redhat-1.jar:7.5.7.Final-redhat-1]
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:231)
> [jbossweb-7.5.7.Final-redhat-1.jar:7.5.7.Final-redhat-1]
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149)
> [jbossweb-7.5.7.Final-redhat-1.jar:7.5.7.Final-redhat-1]
> at
> org.jboss.modcluster.container.jbossweb.JBossWebContext$RequestListenerValve.event(JBossWebContext.java:91)
> at
> org.jboss.modcluster.container.jbossweb.JBossWebContext$RequestListenerValve.invoke(JBossWebContext.java:72)
> at
> org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50)
> [jboss-as-jpa-7.5.0.Final-redhat-21.jar:7.5.0.Final-redhat-21]
> at
> org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50)
> [jboss-as-jpa-7.5.0.Final-redhat-21.jar:7.5.0.Final-redhat-21]
> at
> org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169)
> [jboss-as-web-7.5.0.Final-redhat-21.jar:7.5.0.Final-redhat-21]
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:150)
> [jbossweb-7.5.7.Final-redhat-1.jar:7.5.7.Final-redhat-1]
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97)
> [jbossweb-7.5.7.Final-redhat-1.jar:7.5.7.Final-redhat-1]
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102)
> [jbossweb-7.5.7.Final-redhat-1.jar:7.5.7.Final-redhat-1]
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344)
> [jbossweb-7.5.7.Final-redhat-1.jar:7.5.7.Final-redhat-1]
> at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:854)
> [jbossweb-7.5.7.Final-redhat-1.jar:7.5.7.Final-redhat-1]
> at
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653)
> [jbossweb-7.5.7.Final-redhat-1.jar:7.5.7.Final-redhat-1]
> at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926)
> [jbossweb-7.5.7.Final-redhat-1.jar:7.5.7.Final-redhat-1]
> at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_40]
> Caused by: org.jboss.resteasy.spi.UnhandledException:
> java.lang.IllegalStateException: Cache mode should be DIST, rather than
> REPL_SYNC
> at
> org.jboss.resteasy.core.ExceptionHandler.handleApplicationException(ExceptionHandler.java:76)
> [resteasy-jaxrs-3.0.9.Final.jar:]
> at
> org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:212)
> [resteasy-jaxrs-3.0.9.Final.jar:]
> at
> org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:149)
> [resteasy-jaxrs-3.0.9.Final.jar:]
> at
> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:372)
> [resteasy-jaxrs-3.0.9.Final.jar:]
> at
> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179)
> [resteasy-jaxrs-3.0.9.Final.jar:]
> at
> org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220)
> [resteasy-jaxrs-3.0.9.Final.jar:]
> at
> org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
> [resteasy-jaxrs-3.0.9.Final.jar:]
> at
> org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
> [resteasy-jaxrs-3.0.9.Final.jar:]
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
> [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-2.jar:1.0.2.Final-redhat-2]
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295)
> [jbossweb-7.5.7.Final-redhat-1.jar:7.5.7.Final-redhat-1]
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
> [jbossweb-7.5.7.Final-redhat-1.jar:7.5.7.Final-redhat-1]
> at
> org.keycloak.services.filters.ClientConnectionFilter.doFilter(ClientConnectionFilter.java:41)
> [keycloak-services-1.2.0.Beta1.jar:1.2.0.Beta1]
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
> [jbossweb-7.5.7.Final-redhat-1.jar:7.5.7.Final-redhat-1]
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
> [jbossweb-7.5.7.Final-redhat-1.jar:7.5.7.Final-redhat-1]
> at
> org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:40)
> [keycloak-services-1.2.0.Beta1.jar:1.2.0.Beta1]
> ... 17 more
> Caused by: java.lang.IllegalStateException: Cache mode should be DIST, rather
> than REPL_SYNC
> at
> org.infinispan.distexec.mapreduce.MapReduceTask.ensureProperCacheState(MapReduceTask.java:685)
> [infinispan-core-5.2.11.Final-redhat-2.jar:5.2.11.Final-redhat-2]
> at
> org.infinispan.distexec.mapreduce.MapReduceTask.<init>(MapReduceTask.java:226)
> [infinispan-core-5.2.11.Final-redhat-2.jar:5.2.11.Final-redhat-2]
> at
> org.infinispan.distexec.mapreduce.MapReduceTask.<init>(MapReduceTask.java:190)
> [infinispan-core-5.2.11.Final-redhat-2.jar:5.2.11.Final-redhat-2]
> at
> org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider.getUserSessions(InfinispanUserSessionProvider.java:121)
> [keycloak-model-sessions-infinispan-1.2.0.Beta1.jar:1.2.0.Beta1]
> at
> org.keycloak.services.resources.AccountService.sessionsPage(AccountService.java:344)
> [keycloak-services-1.2.0.Beta1.jar:1.2.0.Beta1]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [rt.jar:1.8.0_40]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [rt.jar:1.8.0_40]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [rt.jar:1.8.0_40]
> at java.lang.reflect.Method.invoke(Method.java:497) [rt.jar:1.8.0_40]
> at
> org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:137)
> [resteasy-jaxrs-3.0.9.Final.jar:]
> at
> org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:296)
> [resteasy-jaxrs-3.0.9.Final.jar:]
> at
> org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:250)
> [resteasy-jaxrs-3.0.9.Final.jar:]
> at
> org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:140)
> [resteasy-jaxrs-3.0.9.Final.jar:]
> at
> org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:103)
> [resteasy-jaxrs-3.0.9.Final.jar:]
> at
> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:356)
> [resteasy-jaxrs-3.0.9.Final.jar:]
> ... 28 more
> 
> 
> Same error I get in admin console (
> http://localhost/auth/admin/master/console/#/realms/cluster-test/sessions/realm
> )
> Strange... Are you using "distributed-cache" with mode "SYNC" on both cluster
> nodes?
> 
> Marek
> 
> 
> 
> 
> Thanks,
> 
> Libor Krzyžanek
> jboss.org Development Team
> 
> 
> 
> 
> On 27 Apr 2015, at 09:05, Libor Krzyžanek < lkrzyzan at redhat.com > wrote:
> 
> Hi Marek,
> your’re right that i’m hitting directly localhsot on different ports.
> 
> I was thinking about cookies resp. load balancer so I checked cookies and
> they were sent on both ports.
> 
> I’ll set up load balancer and I’ll will see.
> 
> Thanks,
> 
> Libor Krzyžanek
> jboss.org Development Team
> 
> 
> 
> 
> On 24 Apr 2015, at 19:06, Marek Posolda < mposolda at redhat.com > wrote:
> 
> Hi Libor,
> 
> the config files looks good (at least for the first look), but question is if
> you're using loadbalancer?
> 
> If you're not using loadbalancer and you access keycloak servers directly on
> localhost:8080 and localhost:8180, the problem might be just in the fact
> that browser cookie KEYCLOAK_IDENTITY is not shared between them and hence
> going to localhost:8180 will not find KEYCLOAK_IDENTITY cookie from
> localhost:8080 and will try to create new session.
> 
> You can check admin console or account management and list available user
> sessions on both nodes. If both cluster nodes have same sessions, then
> replication of userSessions works fine, but only issue is really the cookie.
> 
> I suspect that in production, you will use loadbalancer, so this issue won't
> happen.
> 
> Marek
> 
> On 24.4.2015 15:50, Libor Krzyžanek wrote:
> 
> 
> Attaching keycloak-server.json and standalone-ha.xml
> 
> Thanks,
> 
> Libor Krzyžanek
> jboss.org Development Team
> 
> 
> 
> 
> 
> 
> 
> 
> On 24 Apr 2015, at 15:36, Stian Thorgersen < stian at redhat.com > wrote:
> 
> Can you attach your keycloak-server.json and standalone.xml?
> 
> ----- Original Message -----
> 
> 
> From: "Libor Krzyžanek" < lkrzyzan at redhat.com >
> To: "keycloak-user" < keycloak-user at lists.jboss.org >
> Sent: Friday, 24 April, 2015 3:12:29 PM
> Subject: [keycloak-user] Clustering on localhost with shared DB
> 
> Hi,
> I’m trying to achieve full user session replication which means when I’m
> logged in on node 1 and then hit node 2 then I expect to be logged in but
> I’m forced to log in again.
> 
> I have:
> 1. two localhost nodes with JBoss EAP 6.4 + War installation
> 2. Postgres
> 3. EAP cofigured based on
> http://docs.jboss.org/keycloak/docs/1.2.0.Beta1/userguide/html/clustering.html
> 
> I triedeither
> <distributed-cache name="sessions" mode="SYNC" owners=“ 2 " />
> <distributed-cache name="loginFailures" mode="SYNC" owners=“ 2 " />
> or
> <replicated-cache name="sessions" mode="SYNC"/>
> <replicated-cache name="loginFailures" mode="SYNC”/>
> but with same result.
> 
> I’m starting nodes by
> ./jb1/bin/standalone.sh --server-config=standalone-ha.xml
> -Djboss.node.name=node1
> ./jb2/bin/standalone.sh --server-config=standalone-ha.xml
> -Djboss.socket.binding.port-offset=100 -Djboss.node.name=node2
> 
> 
> both jb1 and jb2 are identical and they know each other (Received new cluster
> view: [node1/keycloak|1] [node1/keycloak, node2/keycloak])
> 
> How do you test clustering of KC please?
> 
> Thanks,
> 
> Libor Krzyžanek
> jboss.org Development Team
> 
> 
> _______________________________________________
> keycloak-user mailing list
> keycloak-user at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user
> 
> 
> 
> _______________________________________________
> keycloak-user mailing list keycloak-user at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user
> 
> 
> _______________________________________________
> keycloak-user mailing list
> keycloak-user at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user
> 
> 
> 
> 
> _______________________________________________
> keycloak-user mailing list
> keycloak-user at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user

More information about the keycloak-user mailing list