Re: [keycloak-user] Sync Issues

Hi Aaron, This all sounds very weird. Off the top of my head: - try latest Keycloak (4.1.0), is the issue reproducible? - Infinispan exposes quite a lot of stuff via JMX. Run JMC or JConsole, connect to the Keycloak process, go to MBeans -> org.wildfly.clustering.infinispan -> Cache -> "keycloak" -> Cache. How many caches are there? (should be 15 as of KC 4.1.0) Are they all running? Are there any abnormalities? Entries under CacheManager might be useful, too. Cheers, Dmitry Telegin CTO, Acutus s.r.o. Keycloak Consulting and Training Pod lipami street 339/52, 130 00 Prague 3, Czech Republic +42 (022) 888-30-71 E-mail: info(a)acutus.pro On Tue, 2018-07-17 at 13:28 -0700, Aaron Echols wrote: > Hello All, > > I've successfully setup a cluster with 2 nodes. Everything is working > great, except for one issue I can't figure out. I'm starting to pull my > hair out and wanted to see if anyone else has seen the issue and how to > correct it. > > I've setup a user federation using Active Directory (Server 2016) using > Keycloak 3.4.3. They are load balanced behind Netscaler 12.0.x. Infinispan > seems to be working correctly. It's backed by a MariaDB 10.1.x, 3 node > cluster. Things I've noted: > > - I can create a local user and it syncs instantly between the KC 3.4.3 > nodes > - Password syncs work, all changes to attributes sync, etc > - I change settings for the user federation I created and they DON'T > sync, so creating a mapper, changing a sync setting, etc, they have to be > changed by hand manually on each node. > - Same with Role and realm-management. I can apply a permission to a > group or user and it doesn't sync. > - If I restart the wildfly server, the changes to propagate to the > opposite node everytime. > > > > I deleted a custom role in the realm-management client, and it deleted it > from the database. On the secondary node, I saw the file was still listed, > even with hard refreshes of the browser. I clicked to delete the custom > role and got the following in the server.log: > > > > ERROR [org.keycloak.services.error.KeycloakErrorHandler] (default task-26) > Uncaught server error: java.lang.IllegalStateException: Not found in > database > at > org.keycloak.models.cache.infinispan.RoleAdapter.isUpdated(RoleAdapter.java:66) > at > org.keycloak.models.cache.infinispan.RoleAdapter.getId(RoleAdapter.java:105) > at > org.keycloak.models.cache.infinispan.RealmCacheSession.removeRole(RealmCacheSession.java:736) > at > org.keycloak.models.cache.infinispan.ClientAdapter.removeRole(ClientAdapter.java:587) > at > org.keycloak.services.resources.admin.RoleResource.deleteRole(RoleResource.java:53) > at > org.keycloak.services.resources.admin.RoleByIdResource.deleteRole(RoleByIdResource.java:115) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:140) > at > org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:295) > at > org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:249) > at > org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:138) > at > org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:107) > at > org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:133) > at > org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:107) > at > org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:133) > at > org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:101) > at > org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:406) > at > org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:213) > at > org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:228) > at > org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) > at > org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) > at > io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85) > at > io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129) > at > org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:90) > at > io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) > at > io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) > at > io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) > at > io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) > at > io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) > at > org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) > at > io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) > at > io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) > at > io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) > at > io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) > at > io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) > at > io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) > at > io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) > at > io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) > at > io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) > at > io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) > at > io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) > at > org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) > at > io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) > at > org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68) > at > io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) > at > io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292) > at > io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) > at > io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) > at > io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) > at > io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) > at > io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) > at > org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) > at > org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508) > at > org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508) > at > org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508) > at > org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508) > at > io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) > at > io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) > at > io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) > at > io.undertow.server.Connectors.executeRootHandler(Connectors.java:326) > at > io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:812) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > > > I'm not sure if there is an issue with Infinispan or a sql connection > issue. I've included my SQL connection string as well: > > > > <datasource jndi-name="java:jboss/datasources/KeycloakDS" > pool-name="KeycloakDS" enabled="true" use-java-context="true"> > <connection-url>jdbc:mariadb:// > 10.5.30.202:3306/keycloak?useUnicode=yes;characterEncoding=UTF-8;sessionVariables=wait_timeout=180;autoRe > connect=true</connection-url> > <driver>mariadb</driver> > <pool> > <max-pool-size>20</max-pool-size> > </pool> > <security> > <user-name>keycloak_user</user-name> > <password><some-passphrase></password> > </security> > <validation> > <check-valid-connection-sql>select > 1</check-valid-connection-sql> > <validate-on-match>true</validate-on-match> > <background-validation>true</background-validation> > > <background-validation-millis>10000</background-validation-millis> > </validation> > </datasource> > <drivers> > <!-- driver declaration --> > <driver name="mariadb" module="org.mariadb"> > > <xa-datasource-class>org.mariadb.jdbc.Driver</xa-datasource-class> > </driver> > <driver name="h2" module="com.h2database.h2"> > > <xa-datasource-class>org.h2.jdbcx.JdbcDataSource</xa-datasource-class> > </driver> > </drivers> > </datasources> > > > > I'm using the mariadb-java-client-2.2.3 driver. > > > > <?xml version="1.0" ?> > <module xmlns="urn:jboss:module:1.3" name="org.mariadb"> > > <resources> > <resource-root path="mariadb-java-client-2.2.3.jar"/> > </resources> > > <dependencies> > <module name="javax.api"/> > <module name="javax.transaction.api"/> > </dependencies> > </module> > > > Any assistance would be appreciated. I'll grab whatever information is > needed. Thank you in advance. :) > -- > *Aaron Echols* > _______________________________________________ > keycloak-user mailing list > keycloak-user(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-user