Re: [keycloak-user] Sync Issues

Hi Dmitry, Thanks for the reply! I just finished upgrading to 4.1.0 and the issue persists... Let me try running the console and take a look there and see what it shows. I'll post back shortly. Thanks for the help! -- *Aaron Echols* On Tue, Jul 17, 2018 at 3:58 PM Dmitry Telegin <dt(a)acutus.pro&gt; wrote: > Hi Aaron, > > This all sounds very weird. Off the top of my head: > - try latest Keycloak (4.1.0), is the issue reproducible? > - Infinispan exposes quite a lot of stuff via JMX. Run JMC or JConsole, > connect to the Keycloak process, go to MBeans -> > org.wildfly.clustering.infinispan -> Cache -> "keycloak" -> Cache. How > many caches are there? (should be 15 as of KC 4.1.0) Are they all > running? Are there any abnormalities? Entries under CacheManager might > be useful, too. > > Cheers, > Dmitry Telegin > CTO, Acutus s.r.o. > Keycloak Consulting and Training > > Pod lipami street 339/52, 130 00 Prague 3, Czech Republic > +42 (022) 888-30-71 > E-mail: info(a)acutus.pro > > On Tue, 2018-07-17 at 13:28 -0700, Aaron Echols wrote: > > Hello All, > > > > I've successfully setup a cluster with 2 nodes. Everything is working > > great, except for one issue I can't figure out. I'm starting to pull my > > hair out and wanted to see if anyone else has seen the issue and how to > > correct it. > > > > I've setup a user federation using Active Directory (Server 2016) using > > Keycloak 3.4.3. They are load balanced behind Netscaler 12.0.x. > Infinispan > > seems to be working correctly. It's backed by a MariaDB 10.1.x, 3 node > > cluster. Things I've noted: > > > > - I can create a local user and it syncs instantly between the KC > 3.4.3 > > nodes > > - Password syncs work, all changes to attributes sync, etc > > - I change settings for the user federation I created and they DON'T > > sync, so creating a mapper, changing a sync setting, etc, they have > to be > > changed by hand manually on each node. > > - Same with Role and realm-management. I can apply a permission to a > > group or user and it doesn't sync. > > - If I restart the wildfly server, the changes to propagate to the > > opposite node everytime. > > > > > > > > I deleted a custom role in the realm-management client, and it deleted > it > > from the database. On the secondary node, I saw the file was still > listed, > > even with hard refreshes of the browser. I clicked to delete the custom > > role and got the following in the server.log: > > > > > > > > ERROR [org.keycloak.services.error.KeycloakErrorHandler] (default > task-26) > > Uncaught server error: java.lang.IllegalStateException: Not found in > > database > > at > > > org.keycloak.models.cache.infinispan.RoleAdapter.isUpdated(RoleAdapter.java:66) > > at > > > org.keycloak.models.cache.infinispan.RoleAdapter.getId(RoleAdapter.java:105) > > at > > > org.keycloak.models.cache.infinispan.RealmCacheSession.removeRole(RealmCacheSession.java:736) > > at > > > org.keycloak.models.cache.infinispan.ClientAdapter.removeRole(ClientAdapter.java:587) > > at > > > org.keycloak.services.resources.admin.RoleResource.deleteRole(RoleResource.java:53) > > at > > > org.keycloak.services.resources.admin.RoleByIdResource.deleteRole(RoleByIdResource.java:115) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > > at > > > org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:140) > > at > > > org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:295) > > at > > > org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:249) > > at > > > org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:138) > > at > > > org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:107) > > at > > > org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:133) > > at > > > org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:107) > > at > > > org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:133) > > at > > > org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:101) > > at > > > org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:406) > > at > > > org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:213) > > at > > > org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:228) > > at > > > org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) > > at > > > org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) > > at > > > io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85) > > at > > > io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129) > > at > > > org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:90) > > at > > io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) > > at > > > io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) > > at > > > io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) > > at > > > io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) > > at > > > io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) > > at > > > org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) > > at > > > io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) > > at > > > io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) > > at > > > io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) > > at > > > io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) > > at > > > io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) > > at > > > io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) > > at > > > io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) > > at > > > io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) > > at > > > io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) > > at > > > io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) > > at > > > io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) > > at > > > org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) > > at > > > io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) > > at > > > org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68) > > at > > > io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) > > at > > > io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292) > > at > > > io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) > > at > > > io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) > > at > > > io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) > > at > > > io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) > > at > > > io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) > > at > > > org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) > > at > > > org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508) > > at > > > org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508) > > at > > > org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508) > > at > > > org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508) > > at > > > io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) > > at > > > io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) > > at > > > io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) > > at > > io.undertow.server.Connectors.executeRootHandler(Connectors.java:326) > > at > > io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:812) > > at > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > > > > > > > I'm not sure if there is an issue with Infinispan or a sql connection > > issue. I've included my SQL connection string as well: > > > > > > > > <datasource > jndi-name="java:jboss/datasources/KeycloakDS" > > pool-name="KeycloakDS" enabled="true" use-java-context="true"> > > <connection-url>jdbc:mariadb:// > > > 10.5.30.202:3306/keycloak?useUnicode=yes;characterEncoding=UTF-8;sessionVariables=wait_timeout=180;autoRe > > connect=true</connection-url> > > <driver>mariadb</driver> > > <pool> > > <max-pool-size>20</max-pool-size> > > </pool> > > <security> > > <user-name>keycloak_user</user-name> > > <password><some-passphrase></password> > > </security> > > <validation> > > <check-valid-connection-sql>select > > 1</check-valid-connection-sql> > > <validate-on-match>true</validate-on-match> > > > <background-validation>true</background-validation> > > > > <background-validation-millis>10000</background-validation-millis> > > </validation> > > </datasource> > > <drivers> > > <!-- driver declaration --> > > <driver name="mariadb" module="org.mariadb"> > > > > <xa-datasource-class>org.mariadb.jdbc.Driver</xa-datasource-class> > > </driver> > > <driver name="h2" module="com.h2database.h2"> > > > > <xa-datasource-class>org.h2.jdbcx.JdbcDataSource</xa-datasource-class> > > </driver> > > </drivers> > > </datasources> > > > > > > > > I'm using the mariadb-java-client-2.2.3 driver. > > > > > > > > <?xml version="1.0" ?> > > <module xmlns="urn:jboss:module:1.3" name="org.mariadb"> > > > > <resources> > > <resource-root path="mariadb-java-client-2.2.3.jar"/> > > </resources> > > > > <dependencies> > > <module name="javax.api"/> > > <module name="javax.transaction.api"/> > > </dependencies> > > </module> > > > > > > Any assistance would be appreciated. I'll grab whatever information is > > needed. Thank you in advance. :) > > -- > > *Aaron Echols* > > _______________________________________________ > > keycloak-user mailing list > > keycloak-user(a)lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/keycloak-user >