[keycloak-user] Clustering on localhost with shared DB

Libor Krzyžanek lkrzyzan at redhat.com
Mon Apr 27 07:50:29 EDT 2015 

Hi,
I have now apache webproxy with this configuration:
<Proxy *>
  Order allow,deny
  Allow from all
</Proxy>
<Proxy balancer://app/>
  BalancerMember http://localhost:8080 route=app02
  BalancerMember http://localhost:8180 route=app03
  ProxySet lbmethod=byrequests
</Proxy>
ProxyPass /balancer-manager !
ProxyPass /server-status !
ProxyPass /server-info !
ProxyPass / balancer://app/ <balancer://app/>
ProxyPassReverse / balancer://app/

It looks it helped.
When I have started both nodes and I see that caches on both nodes are started then everything is fine.
Scenario: When I login to node1, then stop node1, then I’m redirected to node2 and I’m still logged in. Great!

But I see two issues right now:
1. Caches are replicated to newly started node too late.
Scenario is:
1. start node1, log in.
2. start node2, wait till you see that node1 knows new node and node2 is fully started
3. killl node1.

Then I’m redirected to login page.

This happens really only when no request hits newly started node2. If I do few reloads in browser before I kill node1 then I see in logs that those infinispan caches are created on node2 and fully replicated.

Is it related to “start = EAGER” ?

2. Weird thing is on /account/session page (http://localhost/auth/realms/cluster-test/account/sessions <http://localhost/auth/realms/cluster-test/account/sessions>).

I got:

13:30:50,291 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/auth].[Keycloak REST Interface]] (http-/127.0.0.1:8080-2) JBWEB000236: Servlet.service() for servlet Keycloak REST Interface threw exception: java.lang.RuntimeException: request path: /auth/realms/cluster-test/account/sessions
	at org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:54) [keycloak-services-1.2.0.Beta1.jar:1.2.0.Beta1]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) [jbossweb-7.5.7.Final-redhat-1.jar:7.5.7.Final-redhat-1]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.5.7.Final-redhat-1.jar:7.5.7.Final-redhat-1]
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:231) [jbossweb-7.5.7.Final-redhat-1.jar:7.5.7.Final-redhat-1]
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) [jbossweb-7.5.7.Final-redhat-1.jar:7.5.7.Final-redhat-1]
	at org.jboss.modcluster.container.jbossweb.JBossWebContext$RequestListenerValve.event(JBossWebContext.java:91)
	at org.jboss.modcluster.container.jbossweb.JBossWebContext$RequestListenerValve.invoke(JBossWebContext.java:72)
	at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50) [jboss-as-jpa-7.5.0.Final-redhat-21.jar:7.5.0.Final-redhat-21]
	at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50) [jboss-as-jpa-7.5.0.Final-redhat-21.jar:7.5.0.Final-redhat-21]
	at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) [jboss-as-web-7.5.0.Final-redhat-21.jar:7.5.0.Final-redhat-21]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:150) [jbossweb-7.5.7.Final-redhat-1.jar:7.5.7.Final-redhat-1]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.5.7.Final-redhat-1.jar:7.5.7.Final-redhat-1]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.5.7.Final-redhat-1.jar:7.5.7.Final-redhat-1]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) [jbossweb-7.5.7.Final-redhat-1.jar:7.5.7.Final-redhat-1]
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:854) [jbossweb-7.5.7.Final-redhat-1.jar:7.5.7.Final-redhat-1]
	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653) [jbossweb-7.5.7.Final-redhat-1.jar:7.5.7.Final-redhat-1]
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926) [jbossweb-7.5.7.Final-redhat-1.jar:7.5.7.Final-redhat-1]
	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_40]
Caused by: org.jboss.resteasy.spi.UnhandledException: java.lang.IllegalStateException: Cache mode should be DIST, rather than REPL_SYNC
	at org.jboss.resteasy.core.ExceptionHandler.handleApplicationException(ExceptionHandler.java:76) [resteasy-jaxrs-3.0.9.Final.jar:]
	at org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:212) [resteasy-jaxrs-3.0.9.Final.jar:]
	at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:149) [resteasy-jaxrs-3.0.9.Final.jar:]
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:372) [resteasy-jaxrs-3.0.9.Final.jar:]
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179) [resteasy-jaxrs-3.0.9.Final.jar:]
	at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220) [resteasy-jaxrs-3.0.9.Final.jar:]
	at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) [resteasy-jaxrs-3.0.9.Final.jar:]
	at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) [resteasy-jaxrs-3.0.9.Final.jar:]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-2.jar:1.0.2.Final-redhat-2]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295) [jbossweb-7.5.7.Final-redhat-1.jar:7.5.7.Final-redhat-1]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.5.7.Final-redhat-1.jar:7.5.7.Final-redhat-1]
	at org.keycloak.services.filters.ClientConnectionFilter.doFilter(ClientConnectionFilter.java:41) [keycloak-services-1.2.0.Beta1.jar:1.2.0.Beta1]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) [jbossweb-7.5.7.Final-redhat-1.jar:7.5.7.Final-redhat-1]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.5.7.Final-redhat-1.jar:7.5.7.Final-redhat-1]
	at org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:40) [keycloak-services-1.2.0.Beta1.jar:1.2.0.Beta1]
	... 17 more
Caused by: java.lang.IllegalStateException: Cache mode should be DIST, rather than REPL_SYNC
	at org.infinispan.distexec.mapreduce.MapReduceTask.ensureProperCacheState(MapReduceTask.java:685) [infinispan-core-5.2.11.Final-redhat-2.jar:5.2.11.Final-redhat-2]
	at org.infinispan.distexec.mapreduce.MapReduceTask.<init>(MapReduceTask.java:226) [infinispan-core-5.2.11.Final-redhat-2.jar:5.2.11.Final-redhat-2]
	at org.infinispan.distexec.mapreduce.MapReduceTask.<init>(MapReduceTask.java:190) [infinispan-core-5.2.11.Final-redhat-2.jar:5.2.11.Final-redhat-2]
	at org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider.getUserSessions(InfinispanUserSessionProvider.java:121) [keycloak-model-sessions-infinispan-1.2.0.Beta1.jar:1.2.0.Beta1]
	at org.keycloak.services.resources.AccountService.sessionsPage(AccountService.java:344) [keycloak-services-1.2.0.Beta1.jar:1.2.0.Beta1]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_40]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_40]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_40]
	at java.lang.reflect.Method.invoke(Method.java:497) [rt.jar:1.8.0_40]
	at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:137) [resteasy-jaxrs-3.0.9.Final.jar:]
	at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:296) [resteasy-jaxrs-3.0.9.Final.jar:]
	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:250) [resteasy-jaxrs-3.0.9.Final.jar:]
	at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:140) [resteasy-jaxrs-3.0.9.Final.jar:]
	at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:103) [resteasy-jaxrs-3.0.9.Final.jar:]
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:356) [resteasy-jaxrs-3.0.9.Final.jar:]
	... 28 more


Same error I get in admin console (http://localhost/auth/admin/master/console/#/realms/cluster-test/sessions/realm <http://localhost/auth/admin/master/console/#/realms/cluster-test/sessions/realm>)

Thanks,

Libor Krzyžanek
jboss.org <http://jboss.org/> Development Team

> On 27 Apr 2015, at 09:05, Libor Krzyžanek <lkrzyzan at redhat.com> wrote:
> 
> Hi Marek,
> your’re right that i’m hitting directly localhsot on different ports.
> 
> I was thinking about cookies resp. load balancer so I checked cookies and they were sent on both ports.
> 
> I’ll set up load balancer and I’ll will see. 
> 
> Thanks,
> 
> Libor Krzyžanek
> jboss.org <http://jboss.org/> Development Team
> 
>> On 24 Apr 2015, at 19:06, Marek Posolda <mposolda at redhat.com <mailto:mposolda at redhat.com>> wrote:
>> 
>> Hi Libor,
>> 
>> the config files looks good (at least for the first look), but question is if you're using loadbalancer? 
>> 
>> If you're not using loadbalancer and you access keycloak servers directly on localhost:8080 and localhost:8180, the problem might be just in the fact that browser cookie KEYCLOAK_IDENTITY is not shared between them and hence going to localhost:8180 will not find KEYCLOAK_IDENTITY cookie from localhost:8080 and will try to create new session. 
>> 
>> You can check admin console or account management and list available user sessions on both nodes. If both cluster nodes have same sessions, then replication of userSessions works fine, but only issue is really the cookie. 
>> 
>> I suspect that in production, you will use loadbalancer, so this issue won't happen.
>> 
>> Marek
>> 
>> On 24.4.2015 15:50, Libor Krzyžanek wrote:
>>> Attaching keycloak-server.json and standalone-ha.xml
>>> 
>>> Thanks,
>>> 
>>> Libor Krzyžanek
>>> jboss.org <http://jboss.org/> Development Team
>>> 
>>> 
>>> 
>>> 
>>> 
>>>> On 24 Apr 2015, at 15:36, Stian Thorgersen <stian at redhat.com <mailto:stian at redhat.com>> wrote:
>>>> 
>>>> Can you attach your keycloak-server.json and standalone.xml?
>>>> 
>>>> ----- Original Message -----
>>>>> From: "Libor Krzyžanek" <lkrzyzan at redhat.com <mailto:lkrzyzan at redhat.com>>
>>>>> To: "keycloak-user" <keycloak-user at lists.jboss.org <mailto:keycloak-user at lists.jboss.org>>
>>>>> Sent: Friday, 24 April, 2015 3:12:29 PM
>>>>> Subject: [keycloak-user] Clustering on localhost with shared DB
>>>>> 
>>>>> Hi,
>>>>> I’m trying to achieve full user session replication which means when I’m
>>>>> logged in on node 1 and then hit node 2 then I expect to be logged in but
>>>>> I’m forced to log in again.
>>>>> 
>>>>> I have:
>>>>> 1. two localhost nodes with JBoss EAP 6.4 + War installation
>>>>> 2. Postgres
>>>>> 3. EAP cofigured based on
>>>>> http://docs.jboss.org/keycloak/docs/1.2.0.Beta1/userguide/html/clustering.html <http://docs.jboss.org/keycloak/docs/1.2.0.Beta1/userguide/html/clustering.html>
>>>>> 
>>>>> I triedeither
>>>>> <distributed-cache name="sessions" mode="SYNC" owners=“ 2 " />
>>>>> <distributed-cache name="loginFailures" mode="SYNC" owners=“ 2 " />
>>>>> or
>>>>> <replicated-cache name="sessions" mode="SYNC"/>
>>>>> <replicated-cache name="loginFailures" mode="SYNC”/>
>>>>> but with same result.
>>>>> 
>>>>> I’m starting nodes by
>>>>> ./jb1/bin/standalone.sh --server-config=standalone-ha.xml
>>>>> -Djboss.node.name=node1
>>>>> ./jb2/bin/standalone.sh --server-config=standalone-ha.xml
>>>>> -Djboss.socket.binding.port-offset=100 -Djboss.node.name=node2
>>>>> 
>>>>> 
>>>>> both jb1 and jb2 are identical and they know each other (Received new cluster
>>>>> view: [node1/keycloak|1] [node1/keycloak, node2/keycloak])
>>>>> 
>>>>> How do you test clustering of KC please?
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> Libor Krzyžanek
>>>>> jboss.org <http://jboss.org/> Development Team
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> keycloak-user mailing list
>>>>> keycloak-user at lists.jboss.org <mailto:keycloak-user at lists.jboss.org>
>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-user <https://lists.jboss.org/mailman/listinfo/keycloak-user>
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> keycloak-user mailing list
>>> keycloak-user at lists.jboss.org <mailto:keycloak-user at lists.jboss.org>
>>> https://lists.jboss.org/mailman/listinfo/keycloak-user <https://lists.jboss.org/mailman/listinfo/keycloak-user>
> 
> _______________________________________________
> keycloak-user mailing list
> keycloak-user at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/keycloak-user/attachments/20150427/eef0f54e/attachment-0001.html

More information about the keycloak-user mailing list