Re: [keycloak-user] Distributed Keycloak user sessions using Infinispan

From: "Rajat Nair" <rajat.nair(a)hp.com&gt; To: "Stian Thorgersen" <stian(a)redhat.com&gt; Cc: keycloak-user(a)lists.jboss.org Sent: Monday, 27 July, 2015 7:33:25 PM Subject: RE: [keycloak-user] Distributed Keycloak user sessions using Infinispan > Can you send me your standalone-ha.xml and keycloak-server.json? Files attached. The service is started like - /opt/jboss/keycloak/bin/standalone.sh -c standalone-ha.xml -b=test-server-110 -bmanagement=test-server-110 -u 230.0.0.4 -Djboss.node.name=test-server-110 > Also, any chance you can try it out with master? I've been testing with > that as we're about to do 1.4 release soon Glad to give back to the community. Will build and deploy the master on my nodes. Will send findings tomorrow. Regarding a scenario I described earlier - Case 2 1. Start with 1 Node down. We bring it back up. We wait for some time so that Infinispan can sync. 2. Bring down other node. 3. Try to get user info using existing token. Is this a valid use-case?

-- Rajat -----Original Message----- From: Stian Thorgersen [mailto:stian@redhat.com] Sent: 27 July 2015 19:16 To: Nair, Rajat Cc: keycloak-user(a)lists.jboss.org Subject: Re: [keycloak-user] Distributed Keycloak user sessions using Infinispan Also, any chance you can try it out with master? I've been testing with that as we're about to do 1.4 release soon ----- Original Message ----- > From: "Stian Thorgersen" <stian(a)redhat.com&gt; > To: "Rajat Nair" <rajat.nair(a)hp.com&gt; > Cc: keycloak-user(a)lists.jboss.org > Sent: Monday, 27 July, 2015 3:45:46 PM > Subject: Re: [keycloak-user] Distributed Keycloak user sessions using > Infinispan > > Can you send me your standalone-ha.xml and keycloak-server.json? > > ----- Original Message ----- > > From: "Rajat Nair" <rajat.nair(a)hp.com&gt; > > To: "Stian Thorgersen" <stian(a)redhat.com&gt; > > Cc: keycloak-user(a)lists.jboss.org > > Sent: Monday, 27 July, 2015 3:41:36 PM > > Subject: RE: [keycloak-user] Distributed Keycloak user sessions > > using Infinispan > > > > > Do you have both nodes fully up and running before you kill one node? > > Yes. > > This is what we tried - > > Case 1 > > 1. Two node cluster (both running Keycloak engines) - both up and > > running. > > Configured load balancing using mod_cluster. > > 2. Login and get token. > > 3. Bring down one node. > > 4. Get user info using existing token. This is when we get NPE. > > > > Case 2 > > 1. Start with 1 Node down. We bring it back up. We wait for some > > time so that Infinispan can sync. > > 2. Bring down other node. > > 3. Try to get user info using existing token. Again we see NPE. > > > > >It's a bug - if session is expired it should return an error > > >message, not a NPE (see > > >https://issues.jboss.org/browse/KEYCLOAK-1710) > > Thanks for tracking this. > > > > -- Rajat > > > > ----- Original Message ----- > > > From: "Rajat Nair" <rajat.nair(a)hp.com&gt; > > > To: "Stian Thorgersen" <stian(a)redhat.com&gt; > > > Cc: keycloak-user(a)lists.jboss.org > > > Sent: Monday, 27 July, 2015 3:20:27 PM > > > Subject: RE: [keycloak-user] Distributed Keycloak user sessions > > > using Infinispan > > > > > > Thanks for quick reply Stian. > > > > > > > What version? > > > We are using Keycloak 1.3.1 Final. > > > > > > > Did you remember to change userSessions provider to infinispan > > > > in keycloak-server.json? > > > Yes. We got following in keycloak-server.json - > > > "userSessions": { > > > "provider": "infinispan" > > > } > > > > > > > Firstly owners="2" should work fine as long as only one node > > > > dies and the other remains active. Secondly it should return a > > > > NPE, but an error if user session is not found. > > > Could you elaborate on your 2nd point? > > > > Do you have both nodes fully up and running before you kill one node? > > > > It's a bug - if session is expired it should return an error > > message, not a NPE (see > > https://issues.jboss.org/browse/KEYCLOAK-1710) > > > > > > > > -- Rajat > > > > > > -----Original Message----- > > > From: Stian Thorgersen [mailto:stian@redhat.com] > > > Sent: 27 July 2015 18:07 > > > To: Nair, Rajat > > > Cc: keycloak-user(a)lists.jboss.org > > > Subject: Re: [keycloak-user] Distributed Keycloak user sessions > > > using Infinispan > > > > > > Did you remember to change userSessions provider to infinispan in > > > keycloak-server.json? > > > > > > ----- Original Message ----- > > > > From: "Stian Thorgersen" <stian(a)redhat.com&gt; > > > > To: "Rajat Nair" <rajat.nair(a)hp.com&gt; > > > > Cc: keycloak-user(a)lists.jboss.org > > > > Sent: Monday, 27 July, 2015 2:24:17 PM > > > > Subject: Re: [keycloak-user] Distributed Keycloak user sessions > > > > using Infinispan > > > > > > > > What version? > > > > > > > > Firstly owners="2" should work fine as long as only one node > > > > dies and the other remains active. Secondly it should return a > > > > NPE, but an error if user session is not found. > > > > > > > > ----- Original Message ----- > > > > > From: "Rajat Nair" <rajat.nair(a)hp.com&gt; > > > > > To: keycloak-user(a)lists.jboss.org > > > > > Sent: Monday, 27 July, 2015 2:03:47 PM > > > > > Subject: [keycloak-user] Distributed Keycloak user sessions > > > > > using Infinispan > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > I’m in the process of setting up distributed user sessions > > > > > using Infinispan on my Keycloak cluster. This is the > > > > > configuration I use – > > > > > > > > > > <cache-container name="keycloak" > > > > > jndi-name="java:jboss/infinispan/Keycloak"> > > > > > lock-timeout="60000"/> > > > > > > > > > > <invalidation-cache name="realms" mode="SYNC"/> > > > > > > > > > > <invalidation-cache name="users" mode="SYNC"/> > > > > > > > > > > <distributed-cache name="sessions" mode="SYNC" owners="2"/> > > > > > > > > > > <distributed-cache name="loginFailures" mode="SYNC" > > > > > owners="1"/> > > > > > > > > > > </cache-container> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > And in server.logs, I can see my servers communicate – > > > > > > > > > > 2015-07-27 10:27:24,662 INFO [org.infinispan.CLUSTER] > > > > > (remote-thread--p3-t7) > > > > > ISPN000310: Starting cluster-wide rebalance for cache users, > > > > > topology CacheTopology{id=57, rebalanceId=17, > > > > > currentCH=ReplicatedConsistentHash{ns > > > > > = > > > > > 60, owners = (1)[test-server-110: 60]}, > > > > > pendingCH=ReplicatedConsistentHash{ns = 60, owners = > > > > > (2)[test-server-110: > > > > > 30, test-server-111: 30]}, unionCH=null, > > > > > actualMembers=[test-server-110, test-server-111]} > > > > > > > > > > 2015-07-27 10:27:24,665 INFO [org.infinispan.CLUSTER] > > > > > (remote-thread--p3-t10) > > > > > ISPN000310: Starting cluster-wide rebalance for cache realms, > > > > > topology CacheTopology{id=57, rebalanceId=17, > > > > > currentCH=ReplicatedConsistentHash{ns > > > > > = > > > > > 60, owners = (1)[test-server-110: 60]}, > > > > > pendingCH=ReplicatedConsistentHash{ns = 60, owners = > > > > > (2)[test-server-110: > > > > > 30, test-server-111: 30]}, unionCH=null, > > > > > actualMembers=[test-server-110, test-server-111]} > > > > > > > > > > 2015-07-27 10:27:24,665 INFO [org.infinispan.CLUSTER] > > > > > (remote-thread--p3-t8) > > > > > ISPN000310: Starting cluster-wide rebalance for cache > > > > > loginFailures, topology CacheTopology{id=57, rebalanceId=17, > > > > > currentCH=DefaultConsistentHash{ns=80, owners = > > > > > (1)[test-server-110: > > > > > 80+0]}, > > > > > pendingCH=DefaultConsistentHash{ns=80, owners = > > > > > (2)[test-server-110: > > > > > 40+0, > > > > > test-server-111: 40+0]}, unionCH=null, > > > > > actualMembers=[test-server-110, test-server-111]} > > > > > > > > > > 2015-07-27 10:27:24,669 INFO [org.infinispan.CLUSTER] > > > > > (remote-thread--p3-t9) > > > > > ISPN000310: Starting cluster-wide rebalance for cache > > > > > sessions, topology CacheTopology{id=56, rebalanceId=17, > > > > > currentCH=DefaultConsistentHash{ns=80, > > > > > owners = (1)[test-server-110: 80+0]}, > > > > > pendingCH=DefaultConsistentHash{ns=80, > > > > > owners = (2)[test-server-110: 40+0, test-server-111: 40+0]}, > > > > > unionCH=null, actualMembers=[test-server-110, > > > > > test-server-111]} > > > > > > > > > > 2015-07-27 10:27:24,808 INFO [org.infinispan.CLUSTER] > > > > > (remote-thread--p3-t9) > > > > > ISPN000336: Finished cluster-wide rebalance for cache > > > > > loginFailures, topology id = 57 > > > > > > > > > > 2015-07-27 10:27:24,810 INFO [org.infinispan.CLUSTER] > > > > > (remote-thread--p3-t12) > > > > > ISPN000336: Finished cluster-wide rebalance for cache > > > > > sessions, topology id = 56 > > > > > > > > > > 2015-07-27 10:27:24,988 INFO [org.infinispan.CLUSTER] > > > > > (remote-thread--p3-t12) > > > > > ISPN000336: Finished cluster-wide rebalance for cache realms, > > > > > topology id = > > > > > 57 > > > > > > > > > > 2015-07-27 10:27:25,530 INFO [org.infinispan.CLUSTER] > > > > > (remote-thread--p3-t8) > > > > > ISPN000336: Finished cluster-wide rebalance for cache users, > > > > > topology id = > > > > > 57 > > > > > > > > > > > > > > > > > > > > I can successfully login, get a token and fetch user details > > > > > with this token. > > > > > > > > > > > > > > > > > > > > Problem is, if one of the nodes on the cluster goes down and > > > > > if we try to reuse a token which was already issued (so > > > > > workflow is – user logins in, get token, (a node in the > > > > > cluster goes down) and then fetch user details using > > > > > token) – we see an internal server exception. From the logs – > > > > > > > > > > > > > > > > > > > > 2015-07-27 10:24:25,714 ERROR [io.undertow.request] (default > > > > > task-1) > > > > > UT005023: Exception handling request to > > > > > /auth/realms/scaletest/protocol/openid-connect/userinfo: > > > > > java.lang.RuntimeException: request path: > > > > > /auth/realms/scaletest/protocol/openid-connect/userinfo > > > > > > > > > > at > > > > > org.keycloak.services.filters.KeycloakSessionServletFilter.doF > > > > > ilte > > > > > r( > > > > > KeycloakSessionServletFilter.java:54) > > > > > > > > > > at > > > > > io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter. > > > > > java > > > > > :6 > > > > > 0) > > > > > > > > > > at > > > > > io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doF > > > > > ilte > > > > > r( > > > > > FilterHandler.java:132) > > > > > > > > > > at > > > > > io.undertow.servlet.handlers.FilterHandler.handleRequest(Filte > > > > > rHan > > > > > dl > > > > > er.java:85) > > > > > > > > > > at > > > > > io.undertow.servlet.handlers.security.ServletSecurityRoleHandl > > > > > er.h > > > > > an > > > > > dleRequest(ServletSecurityRoleHandler.java:62) > > > > > > > > > > at > > > > > io.undertow.servlet.handlers.ServletDispatchingHandler.handleR > > > > > eque > > > > > st > > > > > (ServletDispatchingHandler.java:36) > > > > > > > > > > at > > > > > org.wildfly.extension.undertow.security.SecurityContextAssocia > > > > > tion > > > > > Ha > > > > > ndler.handleRequest(SecurityContextAssociationHandler.java:78) > > > > > > > > > > at > > > > > io.undertow.server.handlers.PredicateHandler.handleRequest(Pre > > > > > dica > > > > > te > > > > > Handler.java:43) > > > > > > > > > > at > > > > > io.undertow.servlet.handlers.security.SSLInformationAssociatio > > > > > nHan > > > > > dl > > > > > er.handleRequest(SSLInformationAssociationHandler.java:131) > > > > > > > > > > at > > > > > io.undertow.servlet.handlers.security.ServletAuthenticationCal > > > > > lHan > > > > > dl > > > > > er.handleRequest(ServletAuthenticationCallHandler.java:57) > > > > > > > > > > at > > > > > io.undertow.server.handlers.PredicateHandler.handleRequest(Pre > > > > > dica > > > > > te > > > > > Handler.java:43) > > > > > > > > > > at > > > > > io.undertow.security.handlers.AbstractConfidentialityHandler.h > > > > > andl > > > > > eR > > > > > equest(AbstractConfidentialityHandler.java:46) > > > > > > > > > > at > > > > > io.undertow.servlet.handlers.security.ServletConfidentialityCo > > > > > nstr > > > > > ai > > > > > ntHandler.handleRequest(ServletConfidentialityConstraintHandle > > > > > r.ja > > > > > va > > > > > :64) > > > > > > > > > > at > > > > > io.undertow.security.handlers.AuthenticationMechanismsHandler. > > > > > hand > > > > > le > > > > > Request(AuthenticationMechanismsHandler.java:58) > > > > > > > > > > at > > > > > io.undertow.servlet.handlers.security.CachedAuthenticatedSessi > > > > > onHa > > > > > nd > > > > > ler.handleRequest(CachedAuthenticatedSessionHandler.java:72) > > > > > > > > > > at > > > > > io.undertow.security.handlers.NotificationReceiverHandler.hand > > > > > leRe > > > > > qu > > > > > est(NotificationReceiverHandler.java:50) > > > > > > > > > > at > > > > > io.undertow.security.handlers.SecurityInitialHandler.handleReq > > > > > uest > > > > > (S > > > > > ecurityInitialHandler.java:76) > > > > > > > > > > at > > > > > io.undertow.server.handlers.PredicateHandler.handleRequest(Pre > > > > > dica > > > > > te > > > > > Handler.java:43) > > > > > > > > > > at > > > > > org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler. > > > > > ha > > > > > ndleRequest(JACCContextIdHandler.java:61) > > > > > > > > > > at > > > > > io.undertow.server.handlers.PredicateHandler.handleRequest(Pre > > > > > dica > > > > > te > > > > > Handler.java:43) > > > > > > > > > > at > > > > > io.undertow.server.handlers.PredicateHandler.handleRequest(Pre > > > > > dica > > > > > te > > > > > Handler.java:43) > > > > > > > > > > at > > > > > io.undertow.server.handlers.MetricsHandler.handleRequest(Metri > > > > > csHa > > > > > nd > > > > > ler.java:62) > > > > > > > > > > at > > > > > io.undertow.servlet.core.MetricsChainHandler.handleRequest(Met > > > > > rics > > > > > Ch > > > > > ainHandler.java:59) > > > > > > > > > > at > > > > > io.undertow.servlet.handlers.ServletInitialHandler.handleFirst > > > > > Requ > > > > > es > > > > > t(ServletInitialHandler.java:274) > > > > > > > > > > at > > > > > io.undertow.servlet.handlers.ServletInitialHandler.dispatchReq > > > > > uest > > > > > (S > > > > > ervletInitialHandler.java:253) > > > > > > > > > > at > > > > > io.undertow.servlet.handlers.ServletInitialHandler.access$000( > > > > > Serv > > > > > le > > > > > tInitialHandler.java:80) > > > > > > > > > > at > > > > > io.undertow.servlet.handlers.ServletInitialHandler$1.handleReq > > > > > uest > > > > > (S > > > > > ervletInitialHandler.java:172) > > > > > > > > > > at > > > > > io.undertow.server.Connectors.executeRootHandler(Connectors.ja > > > > > va:1 > > > > > 99 > > > > > ) > > > > > > > > > > at > > > > > io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java: > > > > > 774) > > > > > > > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown > > > > > Source) > > > > > > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > > > > > Source) > > > > > > > > > > at java.lang.Thread.run(Unknown Source) > > > > > > > > > > Caused by: org.jboss.resteasy.spi.UnhandledException: > > > > > java.lang.NullPointerException > > > > > > > > > > at > > > > > org.jboss.resteasy.core.ExceptionHandler.handleApplicationExce > > > > > ptio > > > > > n( > > > > > ExceptionHandler.java:76) > > > > > > > > > > at > > > > > org.jboss.resteasy.core.ExceptionHandler.handleException(Excep > > > > > tion > > > > > Ha > > > > > ndler.java:212) > > > > > > > > > > at > > > > > org.jboss.resteasy.core.SynchronousDispatcher.writeException(S > > > > > ynch > > > > > ro > > > > > nousDispatcher.java:149) > > > > > > > > > > at > > > > > org.jboss.resteasy.core.SynchronousDispatcher.invoke(Synchrono > > > > > usDi > > > > > sp > > > > > atcher.java:372) > > > > > > > > > > at > > > > > org.jboss.resteasy.core.SynchronousDispatcher.invoke(Synchrono > > > > > usDi > > > > > sp > > > > > atcher.java:179) > > > > > > > > > > at > > > > > org.jboss.resteasy.plugins.server.servlet.ServletContainerDisp > > > > > atch > > > > > er > > > > > .service(ServletContainerDispatcher.java:220) > > > > > > > > > > at > > > > > org.jboss.resteasy.plugins.server.servlet.HttpServletDispatche > > > > > r.se > > > > > rv > > > > > ice(HttpServletDispatcher.java:56) > > > > > > > > > > at > > > > > org.jboss.resteasy.plugins.server.servlet.HttpServletDispatche > > > > > r.se > > > > > rv > > > > > ice(HttpServletDispatcher.java:51) > > > > > > > > > > at > > > > > javax.servlet.http.HttpServlet.service(HttpServlet.java:790) > > > > > > > > > > at > > > > > io.undertow.servlet.handlers.ServletHandler.handleRequest(Serv > > > > > letH > > > > > an > > > > > dler.java:86) > > > > > > > > > > at > > > > > io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doF > > > > > ilte > > > > > r( > > > > > FilterHandler.java:130) > > > > > > > > > > at > > > > > org.keycloak.services.filters.ClientConnectionFilter.doFilter( > > > > > Clie > > > > > nt > > > > > ConnectionFilter.java:41) > > > > > > > > > > at > > > > > io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter. > > > > > java > > > > > :6 > > > > > 0) > > > > > > > > > > at > > > > > io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doF > > > > > ilte > > > > > r( > > > > > FilterHandler.java:132) > > > > > > > > > > at > > > > > org.keycloak.services.filters.KeycloakSessionServletFilter.doF > > > > > ilte > > > > > r( > > > > > KeycloakSessionServletFilter.java:40) > > > > > > > > > > ... 31 more > > > > > > > > > > Caused by: java.lang.NullPointerException > > > > > > > > > > at > > > > > org.keycloak.protocol.oidc.endpoints.UserInfoEndpoint.issueUse > > > > > rInf > > > > > o( > > > > > UserInfoEndpoint.java:128) > > > > > > > > > > at > > > > > org.keycloak.protocol.oidc.endpoints.UserInfoEndpoint.issueUse > > > > > rInf > > > > > oG > > > > > et(UserInfoEndpoint.java:101) > > > > > > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > > > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) > > > > > > > > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > > > > > Source) > > > > > > > > > > at java.lang.reflect.Method.invoke(Unknown Source) > > > > > > > > > > at > > > > > org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInject > > > > > orIm > > > > > pl > > > > > .java:137) > > > > > > > > > > at > > > > > org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(R > > > > > esou > > > > > rc > > > > > eMethodInvoker.java:296) > > > > > > > > > > at > > > > > org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceM > > > > > etho > > > > > dI > > > > > nvoker.java:250) > > > > > > > > > > at > > > > > org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetO > > > > > bjec > > > > > t( > > > > > ResourceLocatorInvoker.java:140) > > > > > > > > > > at > > > > > org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(Resource > > > > > Loca > > > > > to > > > > > rInvoker.java:109) > > > > > > > > > > at > > > > > org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetO > > > > > bjec > > > > > t( > > > > > ResourceLocatorInvoker.java:135) > > > > > > > > > > at > > > > > org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(Resource > > > > > Loca > > > > > to > > > > > rInvoker.java:103) > > > > > > > > > > at > > > > > org.jboss.resteasy.core.SynchronousDispatcher.invoke(Synchrono > > > > > usDi > > > > > sp > > > > > atcher.java:356) > > > > > > > > > > ... 42 more > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The user guide says – > > > > > > > > > > If you need to prevent node failures from requiring users to > > > > > log in again, set the owners attribute to 2 or more for the > > > > > sessions cache > > > > > > > > > > > > > > > > > > > > Questions - > > > > > > > > > > 1. Have we configured Infinispan incorrectly? We don’t want > > > > > the users to login again if any of the nodes in the cluster go > > > > > down. > > > > > > > > > > 2. Will changing distributed-cache to replicated-cache help in > > > > > this scenario? > > > > > > > > > > 3. Any way we can see the contents of the cache? > > > > > > > > > > > > > > > > > > > > -- Rajat > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > keycloak-user mailing list > > > > > keycloak-user(a)lists.jboss.org > > > > > https://lists.jboss.org/mailman/listinfo/keycloak-user > > > > > > > > _______________________________________________ > > > > keycloak-user mailing list > > > > keycloak-user(a)lists.jboss.org > > > > https://lists.jboss.org/mailman/listinfo/keycloak-user > > > > > >