[keycloak-user] Sync Issues

Sun Jul 22 20:41:25 EDT 2018

You're welcome, glad it helped :) Good luck with Keycloak!

Dmitry

On Wed, 2018-07-18 at 12:32 -0700, Aaron Echols wrote:
> Ok, I fixed a variable in my
> 
> /etc/default/wildfly.conf
> 
> Forgot to change the hostname in there:
> 
> # Hostname:
> WILDFLY_HOST=srv-iam-02
> 
> Once I fixed that, the server started syncing immediately. Thanks for
> helping point me in the right direction. :)
> --
> Aaron Echols
> 
> 
> On Wed, Jul 18, 2018 at 12:25 PM Aaron Echols <aechols at bfcsaz.com>
> wrote:
> > Hi Dmitry,
> > 
> > I did as you suggested, but something seems amiss. When looking
> > under:
> > 
> > MBeans > org.wildfly.clustering.infinispan > CacheManager >
> > "keycloak" > CacheManager > Attributes > clusterMembers
> > 
> > shows the same hosts 2x: [srv-iam-01, srv-iam-01], the later should
> > be 02. The other option you said to look it didn't seem to actually
> > exist:
> > 
> > MBeans -> org.wildfly.clustering.infinispan -> Cache -> "keycloak"
> > -> Cache
> > 
> > I'm still confused and looking through the configs to see if I can
> > figure out what is going on. Thanks :)
> > --
> > Aaron Echols
> > Lead Administrator (IT)
> > Benjamin Franklin Charter School | IT
> > Email: aechols at bfcsaz.com
> > Phone: (480) 677-8400
> > Website: http://www.bfcsaz.com
> > Support Email: techsupport at bfcsaz.com
> > Support Portal: https://bfcs.freshservice.com/support/home
> > Common Questions: https://bfcs.freshservice.com/support/solutions
> > Forgot your password: https://accounts.bfcsaz.com
> > 
> >   
> > 
> > 
> > 
> > *CONFIDENTIALITY NOTICE: This e-mail message, including any
> > attachments, is for the sole use of the intended recipient(s) and
> > may contain confidential and privileged information. Any
> > unauthorized review, copy, use, disclosure, or distribution is
> > prohibited. If you are not the intended recipient, please contact
> > the sender by reply e-mail and destroy all copies of the original
> > message.
> > 
> > 
> > On Tue, Jul 17, 2018 at 4:01 PM Aaron Echols <aechols at bfcsaz.com>
> > wrote:
> > > Hi  Dmitry,
> > > 
> > > Thanks for the reply!
> > > 
> > > I just finished upgrading to 4.1.0 and the issue persists...
> > > 
> > > Let me try running the console and take a look there and see what
> > > it shows. I'll post back shortly. Thanks for the help!
> > > --
> > > Aaron Echols
> > > 
> > > On Tue, Jul 17, 2018 at 3:58 PM Dmitry Telegin <dt at acutus.pro>
> > > wrote:
> > > > Hi Aaron,
> > > > 
> > > > This all sounds very weird. Off the top of my head:
> > > > - try latest Keycloak (4.1.0), is the issue reproducible?
> > > > - Infinispan exposes quite a lot of stuff via JMX. Run JMC or
> > > > JConsole,
> > > > connect to the Keycloak process, go to MBeans ->
> > > > org.wildfly.clustering.infinispan -> Cache -> "keycloak" ->
> > > > Cache. How
> > > > many caches are there? (should be 15 as of KC 4.1.0) Are they
> > > > all
> > > > running? Are there any abnormalities? Entries under
> > > > CacheManager might
> > > > be useful, too.
> > > > 
> > > > Cheers,
> > > > Dmitry Telegin
> > > > CTO, Acutus s.r.o.
> > > > Keycloak Consulting and Training
> > > > 
> > > > Pod lipami street 339/52, 130 00 Prague 3, Czech Republic
> > > > +42 (022) 888-30-71
> > > > E-mail: info at acutus.pro
> > > > 
> > > > On Tue, 2018-07-17 at 13:28 -0700, Aaron Echols wrote:
> > > > > Hello All,
> > > > > 
> > > > > I've successfully setup a cluster with 2 nodes. Everything is
> > > > working
> > > > > great, except for one issue I can't figure out. I'm starting
> > > > to pull my
> > > > > hair out and wanted to see if anyone else has seen the issue
> > > > and how to
> > > > > correct it.
> > > > > 
> > > > > I've setup a user federation using Active Directory (Server
> > > > 2016) using
> > > > > Keycloak 3.4.3. They are load balanced behind Netscaler
> > > > 12.0.x. Infinispan
> > > > > seems to be working correctly. It's backed by a MariaDB
> > > > 10.1.x, 3 node
> > > > > cluster. Things I've noted:
> > > > > 
> > > > >    - I can create a local user and it syncs instantly between
> > > > the KC 3.4.3
> > > > >    nodes
> > > > >    - Password syncs work, all changes to attributes sync, etc
> > > > >    - I change settings for the user federation I created and
> > > > they DON'T
> > > > >    sync, so creating a mapper, changing a sync setting, etc,
> > > > they have to be
> > > > >    changed by hand manually on each node.
> > > > >    - Same with Role and realm-management. I can apply a
> > > > permission to a
> > > > >    group or user and it doesn't sync.
> > > > >    - If I restart the wildfly server, the changes to
> > > > propagate to the
> > > > >    opposite node everytime.
> > > > > 
> > > > > 
> > > > > 
> > > > > I deleted a custom role in the realm-management client, and
> > > > it deleted it
> > > > > from the database. On the secondary node, I saw the file was
> > > > still listed,
> > > > > even with hard refreshes of the browser. I clicked to delete
> > > > the custom
> > > > > role and got the following in the server.log:
> > > > > 
> > > > > 
> > > > > 
> > > > > ERROR [org.keycloak.services.error.KeycloakErrorHandler]
> > > > (default task-26)
> > > > > Uncaught server error: java.lang.IllegalStateException: Not
> > > > found in
> > > > > database
> > > > >         at
> > > > >
> > > > org.keycloak.models.cache.infinispan.RoleAdapter.isUpdated(Role
> > > > Adapter.java:66)
> > > > >         at
> > > > >
> > > > org.keycloak.models.cache.infinispan.RoleAdapter.getId(RoleAdap
> > > > ter.java:105)
> > > > >         at
> > > > >
> > > > org.keycloak.models.cache.infinispan.RealmCacheSession.removeRo
> > > > le(RealmCacheSession.java:736)
> > > > >         at
> > > > >
> > > > org.keycloak.models.cache.infinispan.ClientAdapter.removeRole(C
> > > > lientAdapter.java:587)
> > > > >         at
> > > > >
> > > > org.keycloak.services.resources.admin.RoleResource.deleteRole(R
> > > > oleResource.java:53)
> > > > >         at
> > > > >
> > > > org.keycloak.services.resources.admin.RoleByIdResource.deleteRo
> > > > le(RoleByIdResource.java:115)
> > > > >         at
> > > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > > >         at
> > > > >
> > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccesso
> > > > rImpl.java:62)
> > > > >         at
> > > > >
> > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMetho
> > > > dAccessorImpl.java:43)
> > > > >         at java.lang.reflect.Method.invoke(Method.java:498)
> > > > >         at
> > > > >
> > > > org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjecto
> > > > rImpl.java:140)
> > > > >         at
> > > > >
> > > > org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(Re
> > > > sourceMethodInvoker.java:295)
> > > > >         at
> > > > >
> > > > org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMe
> > > > thodInvoker.java:249)
> > > > >         at
> > > > >
> > > > org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetOb
> > > > ject(ResourceLocatorInvoker.java:138)
> > > > >         at
> > > > >
> > > > org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceL
> > > > ocatorInvoker.java:107)
> > > > >         at
> > > > >
> > > > org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetOb
> > > > ject(ResourceLocatorInvoker.java:133)
> > > > >         at
> > > > >
> > > > org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceL
> > > > ocatorInvoker.java:107)
> > > > >         at
> > > > >
> > > > org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetOb
> > > > ject(ResourceLocatorInvoker.java:133)
> > > > >         at
> > > > >
> > > > org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceL
> > > > ocatorInvoker.java:101)
> > > > >         at
> > > > >
> > > > org.jboss.resteasy.core.SynchronousDispatcher.invoke(Synchronou
> > > > sDispatcher.java:406)
> > > > >         at
> > > > >
> > > > org.jboss.resteasy.core.SynchronousDispatcher.invoke(Synchronou
> > > > sDispatcher.java:213)
> > > > >         at
> > > > >
> > > > org.jboss.resteasy.plugins.server.servlet.ServletContainerDispa
> > > > tcher.service(ServletContainerDispatcher.java:228)
> > > > >         at
> > > > >
> > > > org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher
> > > > .service(HttpServletDispatcher.java:56)
> > > > >         at
> > > > >
> > > > org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher
> > > > .service(HttpServletDispatcher.java:51)
> > > > >         at
> > > > javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> > > > >         at
> > > > >
> > > > io.undertow.servlet.handlers.ServletHandler.handleRequest(Servl
> > > > etHandler.java:85)
> > > > >         at
> > > > >
> > > > io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFi
> > > > lter(FilterHandler.java:129)
> > > > >         at
> > > > >
> > > > org.keycloak.services.filters.KeycloakSessionServletFilter.doFi
> > > > lter(KeycloakSessionServletFilter.java:90)
> > > > >         at
> > > > >
> > > > io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.j
> > > > ava:61)
> > > > >         at
> > > > >
> > > > io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFi
> > > > lter(FilterHandler.java:131)
> > > > >         at
> > > > >
> > > > io.undertow.servlet.handlers.FilterHandler.handleRequest(Filter
> > > > Handler.java:84)
> > > > >         at
> > > > >
> > > > io.undertow.servlet.handlers.security.ServletSecurityRoleHandle
> > > > r.handleRequest(ServletSecurityRoleHandler.java:62)
> > > > >         at
> > > > >
> > > > io.undertow.servlet.handlers.ServletDispatchingHandler.handleRe
> > > > quest(ServletDispatchingHandler.java:36)
> > > > >         at
> > > > >
> > > > org.wildfly.extension.undertow.security.SecurityContextAssociat
> > > > ionHandler.handleRequest(SecurityContextAssociationHandler.java
> > > > :78)
> > > > >         at
> > > > >
> > > > io.undertow.server.handlers.PredicateHandler.handleRequest(Pred
> > > > icateHandler.java:43)
> > > > >         at
> > > > >
> > > > io.undertow.servlet.handlers.security.SSLInformationAssociation
> > > > Handler.handleRequest(SSLInformationAssociationHandler.java:131
> > > > )
> > > > >         at
> > > > >
> > > > io.undertow.servlet.handlers.security.ServletAuthenticationCall
> > > > Handler.handleRequest(ServletAuthenticationCallHandler.java:57)
> > > > >         at
> > > > >
> > > > io.undertow.server.handlers.PredicateHandler.handleRequest(Pred
> > > > icateHandler.java:43)
> > > > >         at
> > > > >
> > > > io.undertow.security.handlers.AbstractConfidentialityHandler.ha
> > > > ndleRequest(AbstractConfidentialityHandler.java:46)
> > > > >         at
> > > > >
> > > > io.undertow.servlet.handlers.security.ServletConfidentialityCon
> > > > straintHandler.handleRequest(ServletConfidentialityConstraintHa
> > > > ndler.java:64)
> > > > >         at
> > > > >
> > > > io.undertow.security.handlers.AuthenticationMechanismsHandler.h
> > > > andleRequest(AuthenticationMechanismsHandler.java:60)
> > > > >         at
> > > > >
> > > > io.undertow.servlet.handlers.security.CachedAuthenticatedSessio
> > > > nHandler.handleRequest(CachedAuthenticatedSessionHandler.java:7
> > > > 7)
> > > > >         at
> > > > >
> > > > io.undertow.security.handlers.NotificationReceiverHandler.handl
> > > > eRequest(NotificationReceiverHandler.java:50)
> > > > >         at
> > > > >
> > > > io.undertow.security.handlers.AbstractSecurityContextAssociatio
> > > > nHandler.handleRequest(AbstractSecurityContextAssociationHandle
> > > > r.java:43)
> > > > >         at
> > > > >
> > > > io.undertow.server.handlers.PredicateHandler.handleRequest(Pred
> > > > icateHandler.java:43)
> > > > >         at
> > > > >
> > > > org.wildfly.extension.undertow.security.jacc.JACCContextIdHandl
> > > > er.handleRequest(JACCContextIdHandler.java:61)
> > > > >         at
> > > > >
> > > > io.undertow.server.handlers.PredicateHandler.handleRequest(Pred
> > > > icateHandler.java:43)
> > > > >         at
> > > > >
> > > > org.wildfly.extension.undertow.deployment.GlobalRequestControll
> > > > erHandler.handleRequest(GlobalRequestControllerHandler.java:68)
> > > > >         at
> > > > >
> > > > io.undertow.server.handlers.PredicateHandler.handleRequest(Pred
> > > > icateHandler.java:43)
> > > > >         at
> > > > >
> > > > io.undertow.servlet.handlers.ServletInitialHandler.handleFirstR
> > > > equest(ServletInitialHandler.java:292)
> > > > >         at
> > > > >
> > > > io.undertow.servlet.handlers.ServletInitialHandler.access$100(S
> > > > ervletInitialHandler.java:81)
> > > > >         at
> > > > >
> > > > io.undertow.servlet.handlers.ServletInitialHandler$2.call(Servl
> > > > etInitialHandler.java:138)
> > > > >         at
> > > > >
> > > > io.undertow.servlet.handlers.ServletInitialHandler$2.call(Servl
> > > > etInitialHandler.java:135)
> > > > >         at
> > > > >
> > > > io.undertow.servlet.core.ServletRequestContextThreadSetupAction
> > > > $1.call(ServletRequestContextThreadSetupAction.java:48)
> > > > >         at
> > > > >
> > > > io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(C
> > > > ontextClassLoaderSetupAction.java:43)
> > > > >         at
> > > > >
> > > > org.wildfly.extension.undertow.security.SecurityContextThreadSe
> > > > tupAction.lambda$create$0(SecurityContextThreadSetupAction.java
> > > > :105)
> > > > >         at
> > > > >
> > > > org.wildfly.extension.undertow.deployment.UndertowDeploymentInf
> > > > oService$UndertowThreadSetupAction.lambda$create$0(UndertowDepl
> > > > oymentInfoService.java:1508)
> > > > >         at
> > > > >
> > > > org.wildfly.extension.undertow.deployment.UndertowDeploymentInf
> > > > oService$UndertowThreadSetupAction.lambda$create$0(UndertowDepl
> > > > oymentInfoService.java:1508)
> > > > >         at
> > > > >
> > > > org.wildfly.extension.undertow.deployment.UndertowDeploymentInf
> > > > oService$UndertowThreadSetupAction.lambda$create$0(UndertowDepl
> > > > oymentInfoService.java:1508)
> > > > >         at
> > > > >
> > > > org.wildfly.extension.undertow.deployment.UndertowDeploymentInf
> > > > oService$UndertowThreadSetupAction.lambda$create$0(UndertowDepl
> > > > oymentInfoService.java:1508)
> > > > >         at
> > > > >
> > > > io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequ
> > > > est(ServletInitialHandler.java:272)
> > > > >         at
> > > > >
> > > > io.undertow.servlet.handlers.ServletInitialHandler.access$000(S
> > > > ervletInitialHandler.java:81)
> > > > >         at
> > > > >
> > > > io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequ
> > > > est(ServletInitialHandler.java:104)
> > > > >         at
> > > > >
> > > > io.undertow.server.Connectors.executeRootHandler(Connectors.jav
> > > > a:326)
> > > > >         at
> > > > >
> > > > io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.
> > > > java:812)
> > > > >         at
> > > > >
> > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExe
> > > > cutor.java:1149)
> > > > >         at
> > > > >
> > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolEx
> > > > ecutor.java:624)
> > > > >         at java.lang.Thread.run(Thread.java:748)
> > > > > 
> > > > > 
> > > > > 
> > > > > I'm not sure if there is an issue with Infinispan or a sql
> > > > connection
> > > > > issue. I've included my SQL connection string as well:
> > > > > 
> > > > > 
> > > > > 
> > > > >                 <datasource jndi-
> > > > name="java:jboss/datasources/KeycloakDS"
> > > > > pool-name="KeycloakDS" enabled="true" use-java-
> > > > context="true">
> > > > >                     <connection-url>jdbc:mariadb://
> > > > >
> > > > 10.5.30.202:3306/keycloak?useUnicode=yes;characterEncoding=UTF-
> > > > 8;sessionVariables=wait_timeout=180;autoRe
> > > > > connect=true</connection-url>
> > > > >                     <driver>mariadb</driver>
> > > > >                     <pool>
> > > > >                         <max-pool-size>20</max-pool-size>
> > > > >                     </pool>
> > > > >                     <security>
> > > > >                         <user-name>keycloak_user</user-name>
> > > > >                         <password><some-
> > > > passphrase></password>
> > > > >                     </security>
> > > > >                     <validation>
> > > > >                         <check-valid-connection-sql>select
> > > > > 1</check-valid-connection-sql>
> > > > >                         <validate-on-match>true</validate-on-
> > > > match>
> > > > >                         <background-
> > > > validation>true</background-validation>
> > > > > 
> > > > > <background-validation-millis>10000</background-validation-
> > > > millis>
> > > > >                     </validation>
> > > > >                 </datasource>
> > > > >                 <drivers>
> > > > >                     <!-- driver declaration -->
> > > > >                     <driver name="mariadb"
> > > > module="org.mariadb">
> > > > > 
> > > > > <xa-datasource-class>org.mariadb.jdbc.Driver</xa-datasource-
> > > > class>
> > > > >                     </driver>
> > > > >                     <driver name="h2"
> > > > module="com.h2database.h2">
> > > > > 
> > > > > <xa-datasource-class>org.h2.jdbcx.JdbcDataSource</xa-
> > > > datasource-class>
> > > > >                     </driver>
> > > > >                 </drivers>
> > > > >             </datasources>
> > > > > 
> > > > > 
> > > > > 
> > > > > I'm using the mariadb-java-client-2.2.3 driver.
> > > > > 
> > > > > 
> > > > > 
> > > > > <?xml version="1.0" ?>
> > > > > <module xmlns="urn:jboss:module:1.3" name="org.mariadb">
> > > > > 
> > > > >     <resources>
> > > > >         <resource-root path="mariadb-java-client-2.2.3.jar"/>
> > > > >     </resources>
> > > > > 
> > > > >     <dependencies>
> > > > >         <module name="javax.api"/>
> > > > >         <module name="javax.transaction.api"/>
> > > > >     </dependencies>
> > > > > </module>
> > > > > 
> > > > > 
> > > > > Any assistance would be appreciated. I'll grab whatever
> > > > information is
> > > > > needed. Thank you in advance. :)
> > > > > --
> > > > > *Aaron Echols*
> > > > > _______________________________________________
> > > > > keycloak-user mailing list
> > > > > keycloak-user at lists.jboss.org
> > > > > https://lists.jboss.org/mailman/listinfo/keycloak-user
> > > > 