<html><body><div style="font-family: times new roman, new york, times, serif; font-size: 12pt; color: #000000"><div>Hey Charles,<br></div><div><br></div><div>I modified the standalone-ha.xml similar to your file to add the S3_PING protocol, and I was able to get two KeyCloak nodes to cluster. I did have to set the jboss.bind.address and jboss.bind.address.management. From your logs I can see that these addresses are both set in your environment as well. How are you setting these? My log is included below.</div><div><br></div><div>Thanks,</div><div>Alan</div><div><br></div><div><p style="margin: 0px;" data-mce-style="margin: 0px;">[ec2-user@ip-172-31-4-165 keycloak-1.7.0.Final]$ bin/standalone.sh -c standalone-ha.xml -Djboss.socket.binding.port-offset=0 -Djboss.node.name=node0 -Djboss.bind.address=$IP -Djboss.bind.address.management=$IP</p><p style="margin: 0px;" data-mce-style="margin: 0px;">=========================================================================</p><p style="margin: 0px;" data-mce-style="margin: 0px;">JBoss Bootstrap Environment</p><p style="margin: 0px;" data-mce-style="margin: 0px;">JBOSS_HOME: /radargun/keycloak-1.7.0.Final</p><p style="margin: 0px;" data-mce-style="margin: 0px;">JAVA: java</p><p style="margin: 0px;" data-mce-style="margin: 0px;">JAVA_OPTS: -server -XX:+UseCompressedOops -server -XX:+UseCompressedOops -Xms64m -Xmx512m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true</p><p style="margin: 0px;" data-mce-style="margin: 0px;">=========================================================================</p><p style="margin: 0px;" data-mce-style="margin: 0px;">Picked up _JAVA_OPTIONS: -Djgroups.s3.secret_access_key=ndWrqybZHIaFsEWEuMlWf5fJVffEapbu7DJgcYTd -Djgroups.s3.access_key=AKIAIVP76RKVWTPN6S6Q -Djgroups.s3.bucket=jdg-jgroups<br>10:49:41,196 INFO [org.jboss.modules] (main) JBoss Modules version 1.4.3.Final<br>10:49:41,418 INFO [org.jboss.msc] (main) JBoss MSC version 1.2.6.Final<br>10:49:41,497 INFO [org.jboss.as] (MSC service thread 1-7) WFLYSRV0049: Keycloak 1.7.0.Final (WildFly Core 1.0.2.Final) starting<br>10:49:42,524 INFO [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 19) WFLYCTL0028: Attribute 'default-stack' in the resource at address '/subsystem=jgroups' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.<br>10:49:42,525 INFO [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 21) WFLYCTL0028: Attribute 'job-repository-type' in the resource at address '/subsystem=batch' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.<br>10:49:42,531 INFO [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 25) WFLYCTL0028: Attribute 'enabled' in the resource at address '/subsystem=datasources/data-source=ExampleDS' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.<br>10:49:42,534 INFO [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 25) WFLYCTL0028: Attribute 'enabled' in the resource at address '/subsystem=datasources/data-source=KeycloakDS' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.<br>10:49:42,707 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http)<br>10:49:42,741 INFO [org.xnio] (MSC service thread 1-2) XNIO version 3.3.1.Final<br>10:49:42,750 INFO [org.xnio.nio] (MSC service thread 1-2) XNIO NIO Implementation Version 3.3.1.Final<br>10:49:42,782 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 39) WFLYCLINF0001: Activating Infinispan subsystem.<br>10:49:42,782 INFO [org.wildfly.extension.io] (ServerService Thread Pool -- 38) WFLYIO001: Worker 'default' has auto-configured to 16 core threads with 128 task threads based on your 8 available processors<br>10:49:42,784 INFO [org.jboss.remoting] (MSC service thread 1-2) JBoss Remoting version 4.0.9.Final<br>10:49:42,800 INFO [org.jboss.as.clustering.jgroups] (ServerService Thread Pool -- 43) WFLYCLJG0001: Activating JGroups subsystem.<br>10:49:42,819 INFO [org.jboss.as.security] (ServerService Thread Pool -- 56) WFLYSEC0002: Activating Security Subsystem<br>10:49:42,824 WARN [org.jboss.as.txn] (ServerService Thread Pool -- 57) WFLYTX0013: Node identifier property is set to the default value. Please make sure it is unique.<br>10:49:42,843 INFO [org.jboss.as.naming] (ServerService Thread Pool -- 49) WFLYNAM0001: Activating Naming Subsystem<br>10:49:42,841 INFO [org.jboss.as.jsf] (ServerService Thread Pool -- 46) WFLYJSF0007: Activated the following JSF Implementations: [main]<br>10:49:42,859 INFO [org.jboss.as.security] (MSC service thread 1-3) WFLYSEC0001: Current PicketBox version=4.9.2.Final<br>10:49:42,922 INFO [org.jboss.as.connector] (MSC service thread 1-8) WFLYJCA0009: Starting JCA Subsystem (IronJacamar 1.2.5.Final)<br>10:49:42,976 INFO [org.jboss.as.webservices] (ServerService Thread Pool -- 59) WFLYWS0002: Activating WebServices Extension<br>10:49:42,986 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 58) WFLYUT0003: Undertow 1.2.9.Final starting<br>10:49:42,986 INFO [org.wildfly.extension.undertow] (MSC service thread 1-5) WFLYUT0003: Undertow 1.2.9.Final starting<br>10:49:42,991 INFO [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 35) WFLYJCA0004: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3)<br>10:49:42,994 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-2) WFLYJCA0018: Started Driver service with driver-name = h2<br>10:49:43,118 INFO [org.jboss.as.naming] (MSC service thread 1-7) WFLYNAM0003: Starting Naming Service<br>10:49:43,119 INFO [org.jboss.as.mail.extension] (MSC service thread 1-6) WFLYMAIL0001: Bound mail session [java:jboss/mail/Default]<br>10:49:43,168 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 58) WFLYUT0014: Creating file handler for path /radargun/keycloak-1.7.0.Final/welcome-content<br>10:49:43,181 INFO [org.wildfly.extension.undertow] (MSC service thread 1-7) WFLYUT0012: Started server default-server.<br>10:49:43,213 INFO [org.wildfly.extension.undertow] (MSC service thread 1-7) WFLYUT0018: Host default-host starting<br>10:49:43,294 INFO [org.wildfly.extension.undertow] (MSC service thread 1-8) WFLYUT0006: Undertow AJP listener ajp listening on /172.31.4.165:8009<br>10:49:43,303 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 62) MODCLUSTER000001: Initializing mod_cluster version 1.3.1.Final<br>10:49:43,307 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0006: Undertow HTTP listener default listening on /172.31.4.165:8080<br>10:49:43,333 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 62) MODCLUSTER000032: Listening to proxy advertisements on /224.0.1.105:23364<br>10:49:43,480 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-3) WFLYJCA0001: Bound data source [java:jboss/datasources/ExampleDS]<br>10:49:43,480 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-4) WFLYJCA0001: Bound data source [java:jboss/datasources/KeycloakDS]<br>10:49:43,739 INFO [org.jboss.as.server.deployment] (MSC service thread 1-3) WFLYSRV0027: Starting deployment of "keycloak-server.war" (runtime-name: "keycloak-server.war")<br>10:49:43,784 INFO [org.jboss.ws.common.management] (MSC service thread 1-8) JBWS022052: Starting JBoss Web Services - Stack CXF Server 5.0.0.Final<br>10:49:44,786 INFO [stdout] (MSC service thread 1-1) <br>10:49:44,786 INFO [stdout] (MSC service thread 1-1) -------------------------------------------------------------------<br>10:49:44,786 INFO [stdout] (MSC service thread 1-1) GMS: address=node0, cluster=ee, physical address=172.31.4.165:7600<br>10:49:44,786 INFO [stdout] (MSC service thread 1-1) -------------------------------------------------------------------<br>10:49:45,049 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 64) ISPN000078: Starting JGroups channel keycloak<br><strong>10:49:45,055 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 64) ISPN000094: Received new cluster view for channel keycloak: [node0|0] (1) [node0]</strong><br>10:49:45,064 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 64) ISPN000079: Channel keycloak local address is node0, physical addresses are [172.31.4.165:7600]<br>10:49:45,067 INFO [org.infinispan.factories.GlobalComponentRegistry] (ServerService Thread Pool -- 64) ISPN000128: Infinispan version: Infinispan 'Insanely Bad Elf' 7.2.3.Final<br>10:49:45,242 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 64) WFLYCLINF0002: Started users cache from keycloak container<br>10:49:45,242 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 65) WFLYCLINF0002: Started realms cache from keycloak container<br>10:49:45,243 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 66) WFLYCLINF0002: Started sessions cache from keycloak container<br>10:49:45,243 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 62) WFLYCLINF0002: Started loginFailures cache from keycloak container<br>10:49:45,846 INFO [org.keycloak.services.resources.KeycloakApplication] (ServerService Thread Pool -- 63) Load config from /radargun/keycloak-1.7.0.Final/standalone/configuration/keycloak-server.json<br>10:49:46,642 INFO [org.hibernate.jpa.internal.util.LogHelper] (ServerService Thread Pool -- 63) HHH000204: Processing PersistenceUnitInfo [<br> name: keycloak-default<br> ...]<br>10:49:46,695 INFO [org.hibernate.Version] (ServerService Thread Pool -- 63) HHH000412: Hibernate Core {4.3.10.Final}<br>10:49:46,696 INFO [org.hibernate.cfg.Environment] (ServerService Thread Pool -- 63) HHH000206: hibernate.properties not found<br>10:49:46,697 INFO [org.hibernate.cfg.Environment] (ServerService Thread Pool -- 63) HHH000021: Bytecode provider name : javassist<br>10:49:46,856 INFO [org.hibernate.annotations.common.Version] (ServerService Thread Pool -- 63) HCANN000001: Hibernate Commons Annotations {4.0.5.Final}<br>10:49:46,906 INFO [org.hibernate.dialect.Dialect] (ServerService Thread Pool -- 63) HHH000400: Using dialect: org.hibernate.dialect.H2Dialect<br>10:49:46,912 WARN [org.hibernate.dialect.H2Dialect] (ServerService Thread Pool -- 63) HHH000431: Unable to determine H2 database version, certain features may not work<br>10:49:47,191 INFO [org.hibernate.hql.internal.ast.ASTQueryTranslatorFactory] (ServerService Thread Pool -- 63) HHH000397: Using ASTQueryTranslatorFactory<br>10:49:47,223 INFO [org.hibernate.validator.internal.util.Version] (ServerService Thread Pool -- 63) HV000001: Hibernate Validator 5.1.3.Final<br>10:49:49,112 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 63) WFLYCLINF0002: Started offlineSessions cache from keycloak container<br>10:49:49,147 INFO [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 63) Deploying javax.ws.rs.core.Application: class org.keycloak.services.resources.KeycloakApplication<br>10:49:49,149 INFO [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 63) Adding class resource org.keycloak.services.resources.ThemeResource from Application class org.keycloak.services.resources.KeycloakApplication<br>10:49:49,149 INFO [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 63) Adding class resource org.keycloak.services.resources.JsResource from Application class org.keycloak.services.resources.KeycloakApplication<br>10:49:49,150 INFO [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 63) Adding class resource org.keycloak.services.resources.WelcomeResource from Application class org.keycloak.services.resources.KeycloakApplication<br>10:49:49,150 INFO [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 63) Adding class resource org.keycloak.services.resources.QRCodeResource from Application class org.keycloak.services.resources.KeycloakApplication<br>10:49:49,150 INFO [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 63) Adding singleton resource org.keycloak.services.resources.ServerVersionResource from Application class org.keycloak.services.resources.KeycloakApplication<br>10:49:49,151 INFO [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 63) Adding provider singleton org.keycloak.services.util.ObjectMapperResolver from Application class org.keycloak.services.resources.KeycloakApplication<br>10:49:49,151 INFO [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 63) Adding provider singleton org.keycloak.services.resources.ModelExceptionMapper from Application class org.keycloak.services.resources.KeycloakApplication<br>10:49:49,151 INFO [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 63) Adding singleton resource org.keycloak.services.resources.admin.AdminRoot from Application class org.keycloak.services.resources.KeycloakApplication<br>10:49:49,152 INFO [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 63) Adding singleton resource org.keycloak.services.resources.RealmsResource from Application class org.keycloak.services.resources.KeycloakApplication<br>10:49:49,229 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 63) WFLYUT0021: Registered web context: /auth<br>10:49:49,277 INFO [org.jboss.as.server] (ServerService Thread Pool -- 61) WFLYSRV0010: Deployed "keycloak-server.war" (runtime-name : "keycloak-server.war")<br>10:49:49,572 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://172.31.4.165:9990/management<br>10:49:49,573 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://172.31.4.165:9990<br>10:49:49,573 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Keycloak 1.7.0.Final (WildFly Core 1.0.2.Final) started in 8688ms - Started 344 of 593 services (339 services are lazy, passive or on-demand)<br><strong>10:50:16,178 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,ee,node0) ISPN000094: Received new cluster view for channel keycloak: [node0|1] (2) [node0, node1]</strong><br>10:50:16,650 INFO [org.infinispan.CLUSTER] (remote-thread--p3-t2) ISPN000310: Starting cluster-wide rebalance for cache users, topology CacheTopology{id=1, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 60, owners = (1)[node0: 60]}, pendingCH=ReplicatedConsistentHash{ns = 60, owners = (2)[node0: 30, node1: 30]}, unionCH=null, actualMembers=[node0, node1]}<br>10:50:16,650 INFO [org.infinispan.CLUSTER] (remote-thread--p3-t4) ISPN000310: Starting cluster-wide rebalance for cache realms, topology CacheTopology{id=1, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 60, owners = (1)[node0: 60]}, pendingCH=ReplicatedConsistentHash{ns = 60, owners = (2)[node0: 30, node1: 30]}, unionCH=null, actualMembers=[node0, node1]}<br>10:50:16,652 INFO [org.infinispan.CLUSTER] (remote-thread--p3-t1) ISPN000310: Starting cluster-wide rebalance for cache sessions, topology CacheTopology{id=1, rebalanceId=1, currentCH=DefaultConsistentHash{ns=80, owners = (1)[node0: 80+0]}, pendingCH=DefaultConsistentHash{ns=80, owners = (2)[node0: 40+0, node1: 40+0]}, unionCH=null, actualMembers=[node0, node1]}<br>10:50:16,652 INFO [org.infinispan.CLUSTER] (remote-thread--p3-t3) ISPN000310: Starting cluster-wide rebalance for cache loginFailures, topology CacheTopology{id=1, rebalanceId=1, currentCH=DefaultConsistentHash{ns=80, owners = (1)[node0: 80+0]}, pendingCH=DefaultConsistentHash{ns=80, owners = (2)[node0: 40+0, node1: 40+0]}, unionCH=null, actualMembers=[node0, node1]}<br>10:50:16,712 INFO [org.infinispan.CLUSTER] (remote-thread--p3-t4) ISPN000336: Finished cluster-wide rebalance for cache users, topology id = 1<br>10:50:16,712 INFO [org.infinispan.CLUSTER] (remote-thread--p3-t1) ISPN000336: Finished cluster-wide rebalance for cache realms, topology id = 1<br>10:50:16,758 INFO [org.infinispan.CLUSTER] (remote-thread--p3-t1) ISPN000336: Finished cluster-wide rebalance for cache sessions, topology id = 1<br>10:50:16,761 INFO [org.infinispan.CLUSTER] (remote-thread--p3-t4) ISPN000336: Finished cluster-wide rebalance for cache loginFailures, topology id = 1<br>10:50:22,061 INFO [org.infinispan.CLUSTER] (remote-thread--p3-t4) ISPN000310: Starting cluster-wide rebalance for cache offlineSessions, topology CacheTopology{id=1, rebalanceId=1, currentCH=DefaultConsistentHash{ns=80, owners = (1)[node0: 80+0]}, pendingCH=DefaultConsistentHash{ns=80, owners = (2)[node0: 40+0, node1: 40+0]}, unionCH=null, actualMembers=[node0, node1]}<br>10:50:22,085 INFO [org.infinispan.CLUSTER] (remote-thread--p3-t4) ISPN000336: Finished cluster-wide rebalance for cache offlineSessions, topology id = 1</p></div><div><br></div><hr id="zwchr"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From: </b>"Alan Field" <afield@redhat.com><br><b>To: </b>"charles-edouard gagnaire" <c.gagnaire@kreactive.com><br><b>Cc: </b>keycloak-user@lists.jboss.org<br><b>Sent: </b>Friday, December 18, 2015 10:21:54 AM<br><b>Subject: </b>Re: [keycloak-user] Problem running keycloak cluster on EC2 with S3_ping<br><div><br></div><div style="font-family: times new roman, new york, times, serif; font-size: 12pt; color: #000000"><div>Hey Charles,<br></div><div><br></div><div>Thanks for the logs. I'm not sure what is wrong, but it looks like each server is creating a cluster of 1. I'll try it with my AWS account to see if I can figure out what is wrong.</div><div><br></div><div>Alan</div><div><br></div><hr id="zwchr"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From: </b>"charles-edouard gagnaire" <c.gagnaire@kreactive.com><br><b>To: </b>"Alan Field" <afield@redhat.com><br><b>Cc: </b>keycloak-user@lists.jboss.org<br><b>Sent: </b>Friday, December 18, 2015 10:04:53 AM<br><b>Subject: </b>Re: [keycloak-user] Problem running keycloak cluster on EC2 with S3_ping<br><div><br></div><div dir="ltr"><div>First i want to thank you guys for the quick answers, i was still reading the "Replace use of Infinispan with User Sessions SPI ?" discussion. </div><div><br></div><div>Yes of course i can send all the logs. You'll find them below. </div><div><br></div><div>The JGroups version is the one shipping with keycloak 1.7, but the problem was the same with Keycloak 1.6.</div><div>Looking at the config file, it looks like i'm using : <subsystem xmlns="urn:jboss:domain:jgroups:3.0"></div><div><br></div><div>I didn't mention it but i use the archive i found on Keycloak website. The archive is "keycloak-1.7.0.Final.tar.gz". </div><div>I just untar and modify the config files, then i launched it using : /opt/keycloak-1.7.0.Final/bin/standalone.sh -c standalone-ha.xml</div><div><br></div><div>Thank you again for your help </div><div><br></div><div>The logs for server 1 are : </div><div>=========================================================================</div><div><br></div><div> JBoss Bootstrap Environment</div><div><br></div><div> JBOSS_HOME: /opt/keycloak-1.7.0.Final</div><div><br></div><div> JAVA: /usr/lib/jvm/jre/bin/java</div><div><br></div><div> JAVA_OPTS: -server -XX:+UseCompressedOops -server -XX:+UseCompressedOops -Xms64m -Xmx512m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true</div><div><br></div><div>=========================================================================</div><div><br></div><div>OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0</div><div>14:38:44,910 INFO [org.jboss.modules] (main) JBoss Modules version 1.4.3.Final</div><div>14:38:45,091 INFO [org.jboss.msc] (main) JBoss MSC version 1.2.6.Final</div><div>14:38:45,163 INFO [<a href="http://org.jboss.as" target="_blank">org.jboss.as</a>] (MSC service thread 1-2) WFLYSRV0049: Keycloak 1.7.0.Final (WildFly Core 1.0.2.Final) starting</div><div>14:38:46,358 INFO [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 24) WFLYCTL0028: Attribute 'job-repository-type' in the resource at address '/subsystem=batch' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.</div><div>14:38:46,360 INFO [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 14) WFLYCTL0028: Attribute 'enabled' in the resource at address '/subsystem=datasources/data-source=PgDskeycloak' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.</div><div>14:38:46,362 INFO [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 14) WFLYCTL0028: Attribute 'enabled' in the resource at address '/subsystem=datasources/data-source=ExampleDS' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.</div><div>14:38:46,362 INFO [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 14) WFLYCTL0028: Attribute 'enabled' in the resource at address '/subsystem=datasources/data-source=KeycloakDS' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.</div><div>14:38:46,370 INFO [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 21) WFLYCTL0028: Attribute 'default-stack' in the resource at address '/subsystem=jgroups' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.</div><div>14:38:46,572 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http)</div><div>14:38:46,589 INFO [org.xnio] (MSC service thread 1-4) XNIO version 3.3.1.Final</div><div>14:38:46,607 INFO [org.xnio.nio] (MSC service thread 1-4) XNIO NIO Implementation Version 3.3.1.Final</div><div>14:38:46,655 INFO [org.jboss.remoting] (MSC service thread 1-4) JBoss Remoting version 4.0.9.Final</div><div>14:38:46,687 INFO [<a href="http://org.wildfly.extension.io" target="_blank">org.wildfly.extension.io</a>] (ServerService Thread Pool -- 38) WFLYIO001: Worker 'default' has auto-configured to 4 core threads with 32 task threads based on your 2 available processors</div><div>14:38:46,685 INFO [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 35) WFLYJCA0005: Deploying non-JDBC-compliant driver class org.postgresql.Driver (version 9.4)</div><div>14:38:46,715 INFO [org.jboss.as.clustering.jgroups] (ServerService Thread Pool -- 43) WFLYCLJG0001: Activating JGroups subsystem.</div><div>14:38:46,724 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 39) WFLYCLINF0001: Activating Infinispan subsystem.</div><div>14:38:46,744 INFO [org.jboss.as.connector] (MSC service thread 1-4) WFLYJCA0009: Starting JCA Subsystem (IronJacamar 1.2.5.Final)</div><div>14:38:46,746 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-2) WFLYJCA0018: Started Driver service with driver-name = postgresql</div><div>14:38:46,767 INFO [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 35) WFLYJCA0004: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3)</div><div>14:38:46,769 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-2) WFLYJCA0018: Started Driver service with driver-name = h2</div><div>14:38:46,781 INFO [org.jboss.as.jsf] (ServerService Thread Pool -- 46) WFLYJSF0007: Activated the following JSF Implementations: [main]</div><div>14:38:46,772 INFO [org.jboss.as.naming] (ServerService Thread Pool -- 49) WFLYNAM0001: Activating Naming Subsystem</div><div>14:38:46,914 INFO [org.jboss.as.security] (ServerService Thread Pool -- 56) WFLYSEC0002: Activating Security Subsystem</div><div>14:38:46,916 INFO [org.jboss.as.security] (MSC service thread 1-1) WFLYSEC0001: Current PicketBox version=4.9.2.Final</div><div>14:38:46,932 WARN [org.jboss.as.txn] (ServerService Thread Pool -- 57) WFLYTX0013: Node identifier property is set to the default value. Please make sure it is unique.</div><div>14:38:46,957 INFO [org.jboss.as.webservices] (ServerService Thread Pool -- 59) WFLYWS0002: Activating WebServices Extension</div><div>14:38:46,985 INFO [org.jboss.as.naming] (MSC service thread 1-2) WFLYNAM0003: Starting Naming Service</div><div>14:38:46,992 INFO [org.jboss.as.mail.extension] (MSC service thread 1-4) WFLYMAIL0001: Bound mail session [java:jboss/mail/Default]</div><div>14:38:47,115 INFO [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0003: Undertow 1.2.9.Final starting</div><div>14:38:47,119 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 58) WFLYUT0003: Undertow 1.2.9.Final starting</div><div>14:38:47,206 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 58) WFLYUT0014: Creating file handler for path /opt/keycloak-1.7.0.Final/welcome-content</div><div>14:38:47,229 INFO [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0012: Started server default-server.</div><div>14:38:47,263 INFO [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0018: Host default-host starting</div><div>14:38:47,320 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0006: Undertow AJP listener ajp listening on /<a href="http://10.1.7.103:8009" target="_blank">10.1.7.103:8009</a></div><div>14:38:47,324 INFO [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0006: Undertow HTTP listener default listening on /<a href="http://10.1.7.103:8080" target="_blank">10.1.7.103:8080</a></div><div>14:38:47,339 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 62) MODCLUSTER000001: Initializing mod_cluster version 1.3.1.Final</div><div>14:38:47,372 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 62) MODCLUSTER000032: Listening to proxy advertisements on /<a href="http://224.0.1.105:23364" target="_blank">224.0.1.105:23364</a></div><div>14:38:47,478 INFO [org.jboss.as.connector.subsystems.datasources.AbstractDataSourceService$AS7DataSourceDeployer] (MSC service thread 1-1) IJ020018: Enabling <validate-on-match> for java:jboss/datasources/PgDskeycloak</div><div>14:38:47,513 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0001: Bound data source [java:jboss/datasources/KeycloakDS]</div><div>14:38:47,513 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0001: Bound data source [java:jboss/datasources/ExampleDS]</div><div>14:38:47,530 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0001: Bound data source [java:jboss/datasources/PgDskeycloak]</div><div>14:38:47,673 INFO [org.jboss.as.server.deployment] (MSC service thread 1-4) WFLYSRV0027: Starting deployment of "keycloak-server.war" (runtime-name: "keycloak-server.war")</div><div>14:38:47,820 INFO [org.jboss.ws.common.management] (MSC service thread 1-3) JBWS022052: Starting JBoss Web Services - Stack CXF Server 5.0.0.Final</div><div>14:38:48,898 INFO [stdout] (MSC service thread 1-2)</div><div>14:38:48,898 INFO [stdout] (MSC service thread 1-2) -------------------------------------------------------------------</div><div>14:38:48,898 INFO [stdout] (MSC service thread 1-2) GMS: address=ip-10-1-7-103, cluster=ee, physical address=<a href="http://10.1.7.103:7600" target="_blank">10.1.7.103:7600</a></div><div>14:38:48,899 INFO [stdout] (MSC service thread 1-2) -------------------------------------------------------------------</div><div>14:38:49,250 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 62) ISPN000078: Starting JGroups channel keycloak</div><div>14:38:49,265 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 62) ISPN000094: Received new cluster view for channel keycloak: [ip-10-1-7-103|0] (1) [ip-10-1-7-103]</div><div>14:38:49,273 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 62) ISPN000079: Channel keycloak local address is ip-10-1-7-103, physical addresses are [<a href="http://10.1.7.103:7600" target="_blank">10.1.7.103:7600</a>]</div><div>14:38:49,277 INFO [org.infinispan.factories.GlobalComponentRegistry] (ServerService Thread Pool -- 62) ISPN000128: Infinispan version: Infinispan 'Insanely Bad Elf' 7.2.3.Final</div><div>14:38:49,521 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 65) WFLYCLINF0002: Started users cache from keycloak container</div><div>14:38:49,529 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 62) WFLYCLINF0002: Started loginFailures cache from keycloak container</div><div>14:38:49,530 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 66) WFLYCLINF0002: Started sessions cache from keycloak container</div><div>14:38:49,536 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 63) WFLYCLINF0002: Started realms cache from keycloak container</div><div>14:38:50,116 INFO [org.keycloak.services.resources.KeycloakApplication] (ServerService Thread Pool -- 66) Load config from /opt/keycloak-1.7.0.Final/standalone/configuration/keycloak-server.json</div><div>14:38:50,638 INFO [org.hibernate.jpa.internal.util.LogHelper] (ServerService Thread Pool -- 66) HHH000204: Processing PersistenceUnitInfo [</div><div> name: keycloak-default</div><div> ...]</div><div>14:38:50,690 INFO [org.hibernate.Version] (ServerService Thread Pool -- 66) HHH000412: Hibernate Core {4.3.10.Final}</div><div>14:38:50,691 INFO [org.hibernate.cfg.Environment] (ServerService Thread Pool -- 66) HHH000206: hibernate.properties not found</div><div>14:38:50,693 INFO [org.hibernate.cfg.Environment] (ServerService Thread Pool -- 66) HHH000021: Bytecode provider name : javassist</div><div>14:38:50,842 INFO [org.hibernate.annotations.common.Version] (ServerService Thread Pool -- 66) HCANN000001: Hibernate Commons Annotations {4.0.5.Final}</div><div>14:38:51,794 INFO [org.hibernate.dialect.Dialect] (ServerService Thread Pool -- 66) HHH000400: Using dialect: org.hibernate.dialect.PostgreSQL9Dialect</div><div>14:38:51,803 INFO [org.hibernate.engine.jdbc.internal.LobCreatorBuilder] (ServerService Thread Pool -- 66) HHH000424: Disabling contextual LOB creation as createClob() method threw error : java.lang.reflect.InvocationTargetException</div><div>14:38:52,120 INFO [org.hibernate.hql.internal.ast.ASTQueryTranslatorFactory] (ServerService Thread Pool -- 66) HHH000397: Using ASTQueryTranslatorFactory</div><div>14:38:52,156 INFO [org.hibernate.validator.internal.util.Version] (ServerService Thread Pool -- 66) HV000001: Hibernate Validator 5.1.3.Final</div><div>14:38:53,706 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 66) WFLYCLINF0002: Started offlineSessions cache from keycloak container</div><div>14:38:53,748 INFO [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 66) Deploying javax.ws.rs.core.Application: class org.keycloak.services.resources.KeycloakApplication</div><div>14:38:53,750 INFO [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 66) Adding class resource org.keycloak.services.resources.WelcomeResource from Application class org.keycloak.services.resources.KeycloakApplication</div><div>14:38:53,750 INFO [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 66) Adding class resource org.keycloak.services.resources.JsResource from Application class org.keycloak.services.resources.KeycloakApplication</div><div>14:38:53,750 INFO [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 66) Adding class resource org.keycloak.services.resources.QRCodeResource from Application class org.keycloak.services.resources.KeycloakApplication</div><div>14:38:53,750 INFO [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 66) Adding class resource org.keycloak.services.resources.ThemeResource from Application class org.keycloak.services.resources.KeycloakApplication</div><div>14:38:53,751 INFO [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 66) Adding singleton resource org.keycloak.services.resources.RealmsResource from Application class org.keycloak.services.resources.KeycloakApplication</div><div>14:38:53,751 INFO [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 66) Adding singleton resource org.keycloak.services.resources.ServerVersionResource from Application class org.keycloak.services.resources.KeycloakApplication</div><div>14:38:53,751 INFO [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 66) Adding singleton resource org.keycloak.services.resources.admin.AdminRoot from Application class org.keycloak.services.resources.KeycloakApplication</div><div>14:38:53,751 INFO [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 66) Adding provider singleton org.keycloak.services.util.ObjectMapperResolver from Application class org.keycloak.services.resources.KeycloakApplication</div><div>14:38:53,752 INFO [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 66) Adding provider singleton org.keycloak.services.resources.ModelExceptionMapper from Application class org.keycloak.services.resources.KeycloakApplication</div><div>14:38:53,824 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 66) WFLYUT0021: Registered web context: /auth</div><div>14:38:53,920 INFO [org.jboss.as.server] (ServerService Thread Pool -- 61) WFLYSRV0010: Deployed "keycloak-server.war" (runtime-name : "keycloak-server.war")</div><div>14:38:54,021 INFO [<a href="http://org.jboss.as" target="_blank">org.jboss.as</a>] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on <a href="http://10.1.7.103:9990/management" target="_blank">http://10.1.7.103:9990/management</a></div><div>14:38:54,021 INFO [<a href="http://org.jboss.as" target="_blank">org.jboss.as</a>] (Controller Boot Thread) WFLYSRV0051: Admin console listening on <a href="http://10.1.7.103:9990" target="_blank">http://10.1.7.103:9990</a></div><div>14:38:54,022 INFO [<a href="http://org.jboss.as" target="_blank">org.jboss.as</a>] (Controller Boot Thread) WFLYSRV0025: Keycloak 1.7.0.Final (WildFly Core 1.0.2.Final) started in 9388ms - Started 349 of 613 services (353 services are lazy, passive or on-demand)</div><div><br></div><div><br></div><div>The logs for server 2 are : </div><div>=========================================================================</div><div><br></div><div> JBoss Bootstrap Environment</div><div><br></div><div> JBOSS_HOME: /opt/keycloak-1.7.0.Final</div><div><br></div><div> JAVA: /usr/lib/jvm/jre/bin/java</div><div><br></div><div> JAVA_OPTS: -server -XX:+UseCompressedOops -server -XX:+UseCompressedOops -Xms64m -Xmx512m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true</div><div><br></div><div>=========================================================================</div><div><br></div><div>14:38:48,239 INFO [org.jboss.modules] (main) JBoss Modules version 1.4.3.Final</div><div>14:38:48,723 INFO [org.jboss.msc] (main) JBoss MSC version 1.2.6.Final</div><div>14:38:48,896 INFO [<a href="http://org.jboss.as" target="_blank">org.jboss.as</a>] (MSC service thread 1-2) WFLYSRV0049: Keycloak 1.7.0.Final (WildFly Core 1.0.2.Final) starting</div><div>14:38:50,979 INFO [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 8) WFLYCTL0028: Attribute 'job-repository-type' in the resource at address '/subsystem=batch' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.</div><div>14:38:50,983 INFO [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 11) WFLYCTL0028: Attribute 'enabled' in the resource at address '/subsystem=datasources/data-source=PgDskeycloak' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.</div><div>14:38:50,986 INFO [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 11) WFLYCTL0028: Attribute 'enabled' in the resource at address '/subsystem=datasources/data-source=ExampleDS' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.</div><div>14:38:51,010 INFO [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 14) WFLYCTL0028: Attribute 'default-stack' in the resource at address '/subsystem=jgroups' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.</div><div>14:38:51,044 INFO [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 11) WFLYCTL0028: Attribute 'enabled' in the resource at address '/subsystem=datasources/data-source=KeycloakDS' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.</div><div>14:38:51,452 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http)</div><div>14:38:51,499 INFO [org.xnio] (MSC service thread 1-1) XNIO version 3.3.1.Final</div><div>14:38:51,520 INFO [org.xnio.nio] (MSC service thread 1-1) XNIO NIO Implementation Version 3.3.1.Final</div><div>14:38:51,590 INFO [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 35) WFLYJCA0005: Deploying non-JDBC-compliant driver class org.postgresql.Driver (version 9.4)</div><div>14:38:51,603 INFO [<a href="http://org.wildfly.extension.io" target="_blank">org.wildfly.extension.io</a>] (ServerService Thread Pool -- 38) WFLYIO001: Worker 'default' has auto-configured to 2 core threads with 16 task threads based on your 1 available processors</div><div>14:38:51,601 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 39) WFLYCLINF0001: Activating Infinispan subsystem.</div><div>14:38:51,634 INFO [org.jboss.as.clustering.jgroups] (ServerService Thread Pool -- 43) WFLYCLJG0001: Activating JGroups subsystem.</div><div>14:38:51,694 INFO [org.jboss.as.naming] (ServerService Thread Pool -- 49) WFLYNAM0001: Activating Naming Subsystem</div><div>14:38:51,666 INFO [org.jboss.as.jsf] (ServerService Thread Pool -- 46) WFLYJSF0007: Activated the following JSF Implementations: [main]</div><div>14:38:51,696 INFO [org.jboss.as.connector] (MSC service thread 1-2) WFLYJCA0009: Starting JCA Subsystem (IronJacamar 1.2.5.Final)</div><div>14:38:51,932 INFO [org.jboss.as.webservices] (ServerService Thread Pool -- 59) WFLYWS0002: Activating WebServices Extension</div><div>14:38:51,970 INFO [org.jboss.remoting] (MSC service thread 1-1) JBoss Remoting version 4.0.9.Final</div><div>14:38:51,975 INFO [org.jboss.as.security] (ServerService Thread Pool -- 56) WFLYSEC0002: Activating Security Subsystem</div><div>14:38:51,972 INFO [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 35) WFLYJCA0004: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3)</div><div>14:38:51,971 WARN [org.jboss.as.txn] (ServerService Thread Pool -- 57) WFLYTX0013: Node identifier property is set to the default value. Please make sure it is unique.</div><div>14:38:52,140 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 58) WFLYUT0003: Undertow 1.2.9.Final starting</div><div>14:38:52,187 INFO [org.jboss.as.security] (MSC service thread 1-2) WFLYSEC0001: Current PicketBox version=4.9.2.Final</div><div>14:38:52,224 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-1) WFLYJCA0018: Started Driver service with driver-name = postgresql</div><div>14:38:52,225 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-1) WFLYJCA0018: Started Driver service with driver-name = h2</div><div>14:38:52,368 INFO [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0003: Undertow 1.2.9.Final starting</div><div>14:38:52,369 INFO [org.jboss.as.naming] (MSC service thread 1-2) WFLYNAM0003: Starting Naming Service</div><div>14:38:52,471 INFO [org.jboss.as.mail.extension] (MSC service thread 1-2) WFLYMAIL0001: Bound mail session [java:jboss/mail/Default]</div><div>14:38:52,710 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 58) WFLYUT0014: Creating file handler for path /opt/keycloak-1.7.0.Final/welcome-content</div><div>14:38:52,864 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0012: Started server default-server.</div><div>14:38:53,133 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0006: Undertow HTTP listener default listening on /<a href="http://10.1.1.245:8080" target="_blank">10.1.1.245:8080</a></div><div>14:38:53,166 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0018: Host default-host starting</div><div>14:38:53,192 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0006: Undertow AJP listener ajp listening on /<a href="http://10.1.1.245:8009" target="_blank">10.1.1.245:8009</a></div><div>14:38:53,211 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 62) MODCLUSTER000001: Initializing mod_cluster version 1.3.1.Final</div><div>14:38:53,307 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 62) MODCLUSTER000032: Listening to proxy advertisements on /<a href="http://224.0.1.105:23364" target="_blank">224.0.1.105:23364</a></div><div>14:38:53,779 INFO [org.jboss.as.connector.subsystems.datasources.AbstractDataSourceService$AS7DataSourceDeployer] (MSC service thread 1-2) IJ020018: Enabling <validate-on-match> for java:jboss/datasources/PgDskeycloak</div><div>14:38:53,896 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-1) WFLYJCA0001: Bound data source [java:jboss/datasources/KeycloakDS]</div><div>14:38:53,903 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0001: Bound data source [java:jboss/datasources/ExampleDS]</div><div>14:38:53,909 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0001: Bound data source [java:jboss/datasources/PgDskeycloak]</div><div>14:38:54,118 INFO [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0027: Starting deployment of "keycloak-server.war" (runtime-name: "keycloak-server.war")</div><div>14:38:54,306 INFO [org.jboss.ws.common.management] (MSC service thread 1-1) JBWS022052: Starting JBoss Web Services - Stack CXF Server 5.0.0.Final</div><div>14:38:56,138 INFO [stdout] (MSC service thread 1-2)</div><div>14:38:56,138 INFO [stdout] (MSC service thread 1-2) -------------------------------------------------------------------</div><div>14:38:56,139 INFO [stdout] (MSC service thread 1-2) GMS: address=ip-10-1-1-245, cluster=ee, physical address=<a href="http://10.1.1.245:7600" target="_blank">10.1.1.245:7600</a></div><div>14:38:56,139 INFO [stdout] (MSC service thread 1-2) -------------------------------------------------------------------</div><div>14:38:56,606 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 62) ISPN000078: Starting JGroups channel keycloak</div><div>14:38:56,623 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 62) ISPN000094: Received new cluster view for channel keycloak: [ip-10-1-1-245|0] (1) [ip-10-1-1-245]</div><div>14:38:56,644 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 62) ISPN000079: Channel keycloak local address is ip-10-1-1-245, physical addresses are [<a href="http://10.1.1.245:7600" target="_blank">10.1.1.245:7600</a>]</div><div>14:38:56,651 INFO [org.infinispan.factories.GlobalComponentRegistry] (ServerService Thread Pool -- 62) ISPN000128: Infinispan version: Infinispan 'Insanely Bad Elf' 7.2.3.Final</div><div>14:38:57,044 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 65) WFLYCLINF0002: Started users cache from keycloak container</div><div>14:38:57,050 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 62) WFLYCLINF0002: Started sessions cache from keycloak container</div><div>14:38:57,055 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 63) WFLYCLINF0002: Started realms cache from keycloak container</div><div>14:38:57,059 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 64) WFLYCLINF0002: Started loginFailures cache from keycloak container</div><div>14:38:58,007 INFO [org.keycloak.services.resources.KeycloakApplication] (ServerService Thread Pool -- 64) Load config from /opt/keycloak-1.7.0.Final/standalone/configuration/keycloak-server.json</div><div>14:38:58,755 INFO [org.hibernate.jpa.internal.util.LogHelper] (ServerService Thread Pool -- 64) HHH000204: Processing PersistenceUnitInfo [</div><div> name: keycloak-default</div><div> ...]</div><div>14:38:58,812 INFO [org.hibernate.Version] (ServerService Thread Pool -- 64) HHH000412: Hibernate Core {4.3.10.Final}</div><div>14:38:58,819 INFO [org.hibernate.cfg.Environment] (ServerService Thread Pool -- 64) HHH000206: hibernate.properties not found</div><div>14:38:58,824 INFO [org.hibernate.cfg.Environment] (ServerService Thread Pool -- 64) HHH000021: Bytecode provider name : javassist</div><div>14:38:59,268 INFO [org.hibernate.annotations.common.Version] (ServerService Thread Pool -- 64) HCANN000001: Hibernate Commons Annotations {4.0.5.Final}</div><div>14:39:00,264 INFO [org.hibernate.dialect.Dialect] (ServerService Thread Pool -- 64) HHH000400: Using dialect: org.hibernate.dialect.PostgreSQL9Dialect</div><div>14:39:00,272 INFO [org.hibernate.engine.jdbc.internal.LobCreatorBuilder] (ServerService Thread Pool -- 64) HHH000424: Disabling contextual LOB creation as createClob() method threw error : java.lang.reflect.InvocationTargetException</div><div>14:39:00,602 INFO [org.hibernate.hql.internal.ast.ASTQueryTranslatorFactory] (ServerService Thread Pool -- 64) HHH000397: Using ASTQueryTranslatorFactory</div><div>14:39:00,634 INFO [org.hibernate.validator.internal.util.Version] (ServerService Thread Pool -- 64) HV000001: Hibernate Validator 5.1.3.Final</div><div>14:39:04,607 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 64) WFLYCLINF0002: Started offlineSessions cache from keycloak container</div><div>14:39:04,665 INFO [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 64) Deploying javax.ws.rs.core.Application: class org.keycloak.services.resources.KeycloakApplication</div><div>14:39:04,667 INFO [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 64) Adding class resource org.keycloak.services.resources.WelcomeResource from Application class org.keycloak.services.resources.KeycloakApplication</div><div>14:39:04,667 INFO [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 64) Adding class resource org.keycloak.services.resources.QRCodeResource from Application class org.keycloak.services.resources.KeycloakApplication</div><div>14:39:04,668 INFO [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 64) Adding class resource org.keycloak.services.resources.JsResource from Application class org.keycloak.services.resources.KeycloakApplication</div><div>14:39:04,668 INFO [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 64) Adding class resource org.keycloak.services.resources.ThemeResource from Application class org.keycloak.services.resources.KeycloakApplication</div><div>14:39:04,668 INFO [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 64) Adding provider singleton org.keycloak.services.resources.ModelExceptionMapper from Application class org.keycloak.services.resources.KeycloakApplication</div><div>14:39:04,669 INFO [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 64) Adding singleton resource org.keycloak.services.resources.RealmsResource from Application class org.keycloak.services.resources.KeycloakApplication</div><div>14:39:04,669 INFO [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 64) Adding provider singleton org.keycloak.services.util.ObjectMapperResolver from Application class org.keycloak.services.resources.KeycloakApplication</div><div>14:39:04,669 INFO [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 64) Adding singleton resource org.keycloak.services.resources.ServerVersionResource from Application class org.keycloak.services.resources.KeycloakApplication</div><div>14:39:04,669 INFO [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 64) Adding singleton resource org.keycloak.services.resources.admin.AdminRoot from Application class org.keycloak.services.resources.KeycloakApplication</div><div>14:39:04,757 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 64) WFLYUT0021: Registered web context: /auth</div><div>14:39:04,844 INFO [org.jboss.as.server] (ServerService Thread Pool -- 61) WFLYSRV0010: Deployed "keycloak-server.war" (runtime-name : "keycloak-server.war")</div><div>14:39:05,526 INFO [<a href="http://org.jboss.as" target="_blank">org.jboss.as</a>] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on <a href="http://10.1.1.245:9990/management" target="_blank">http://10.1.1.245:9990/management</a></div><div>14:39:05,527 INFO [<a href="http://org.jboss.as" target="_blank">org.jboss.as</a>] (Controller Boot Thread) WFLYSRV0051: Admin console listening on <a href="http://10.1.1.245:9990" target="_blank">http://10.1.1.245:9990</a></div><div>14:39:05,531 INFO [<a href="http://org.jboss.as" target="_blank">org.jboss.as</a>] (Controller Boot Thread) WFLYSRV0025: Keycloak 1.7.0.Final (WildFly Core 1.0.2.Final) started in 17727ms - Started 349 of 613 services (353 services are lazy, passive or on-demand)</div><div><br></div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><table width="100%" cellpadding="0" cellspacing="0" style="font-family:'Times New Roman'" class="mceItemTable"><tbody><tr><td colspan="4" height="20"></td></tr><tr><td valign="top" align="left"><table cellpadding="0" cellspacing="0" class="mceItemTable"><tbody><tr><td width="20"></td><td style="font-family:'Exo 2',helvetica,arial,sans-serif"><span style="color: #333333;"><span style="line-height:1em;font-size:20px">CHARLES-EDOUARD GAGNAIRE</span></span><br><span style="color: #6c6c6c;"><span style="line-height:1.4em;font-style:italic;font-size:14px">SysAdmin</span></span> <br><span style="font-size: small;"><a href="mailto:c.gagnaire@kreactive.com" style="text-decoration:none;line-height:2em" target="_blank"><span style="font-size: 13px; color: #b60018;">c.gagnaire@kreactive.com</span></a><br><span style="line-height: 1em; font-size: 14px; color: #333333;">p. 06.27.80.28.53</span></span><table cellpadding="0" cellspacing="0" class="mceItemTable"><tbody><tr><td height="20"></td></tr><tr><td valign="top"><span style="font-family: 'Exo 2', helvetica, arial, sans-serif; font-size: xx-small;"><span style="font-style: italic; font-size: 11px; color: #333333;">LYON "Le Capitole"</span><br><span style="font-style: italic; font-size: 10px; color: #333333;">97, cours Gambetta<br>69481 Lyon Cedex 03</span> </span><br><div><br></div><span style="font-family: 'Exo 2', helvetica, arial, sans-serif; font-size: xx-small;"><span style="font-style: italic; font-size: 11px; color: #333333;">PARIS</span><br><span style="font-style: italic; font-size: 10px; color: #333333;">16, rue de Turbigo<br>75002 Paris</span></span></td></tr></tbody></table></td></tr></tbody></table></td><td align="center" valign="middle"><img src="http://s3.amazonaws.com/signature_kreactive/kt-signatures/place1.gif" src="http://s3.amazonaws.com/signature_kreactive/kt-signatures/place1.gif"></td><td valign="middle" align="right" style="line-height:0.8em;font-family:helvetica,arial,sans-serif"><div style="width:132px;text-align:center"><a href="http://www.kreactive.com/" target="_blank"><img alt="Kreactive" width="132" height="90" border="0" src="http://s3.amazonaws.com/signature_kreactive/kt-signatures/kreactive-corporate.png" src="http://s3.amazonaws.com/signature_kreactive/kt-signatures/kreactive-corporate.png"></a><br><div><br></div><br><a href="https://www.facebook.com/kreactive" target="_blank"><img border="0" alt="Facebook" width="17" height="17" src="http://s3.amazonaws.com/signature_kreactive/kt-signatures/icon-facebook-kreactive.gif" src="http://s3.amazonaws.com/signature_kreactive/kt-signatures/icon-facebook-kreactive.gif"></a> <a href="https://twitter.com/kreactive" target="_blank"><img border="0" alt="Twitter" width="17" height="17" src="http://s3.amazonaws.com/signature_kreactive/kt-signatures/icon-twitter-kreactive.gif" src="http://s3.amazonaws.com/signature_kreactive/kt-signatures/icon-twitter-kreactive.gif"></a></div></td><td width="15"></td></tr><tr><td colspan="4" height="10"></td></tr><tr><td colspan="4" height="4" bgcolor="#B60018"></td></tr></tbody></table></div></div></div></div></div>
<br><div class="gmail_quote">2015-12-18 15:53 GMT+01:00 Alan Field <span dir="ltr"><<a href="mailto:afield@redhat.com" target="_blank">afield@redhat.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-family:times new roman,new york,times,serif;font-size:12pt;color:#000000"><div>Hey Charles,<br></div><div><br></div><div>Can you send the full logs and tell me which version of JGroups you are using?</div><div><br></div><div>Thanks,</div><div>Alan</div><div><br></div><hr><blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><b>From: </b>"charles-edouard gagnaire" <<a href="mailto:c.gagnaire@kreactive.com" target="_blank">c.gagnaire@kreactive.com</a>><br><b>To: </b><a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a><br><b>Sent: </b>Friday, December 18, 2015 9:01:12 AM<br><b>Subject: </b>[keycloak-user] Problem running keycloak cluster on EC2 with S3_ping<div><div class="h5"><br><div><br></div><div dir="ltr"><div>hi, </div><div><br></div><div>I'm having trouble configuring a Keycloak cluster running on AWS' EC2. </div><div>The database configuration is OK no problem, but i can't manage to get the invalidation cache working correctly. </div><div>I configured Infinispan to work with S3_ping plugin (the relevant part of my configuration is below). </div><div><br></div><div>When i run both server, the connection with the database is Ok, but the infinispan logs look like this : </div><div>On Server 1 : </div><div>...</div><div>11:00:17,592 INFO [stdout] (MSC service thread 1-1) GMS: address=ip-10-1-7-103, cluster=ee, physical address=<a href="http://10.1.7.103:7600" target="_blank">10.1.7.103:7600</a></div><div>...</div><div>11:00:18,057 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 62) ISPN000094: Received new cluster view for channel keycloak: [ip-10-1-7-103|0] (1) [ip-10-1-7-103]</div><div>...</div><div><br></div><div>On Server 2 : </div><div>...</div><div>11:03:41,159 INFO [stdout] (MSC service thread 1-1) GMS: address=ip-10-1-1-245, cluster=ee, physical address=<a href="http://10.1.1.245:7600" target="_blank">10.1.1.245:7600</a></div><div>...</div><div>11:03:41,783 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 62) ISPN000094: Received new cluster view for channel keycloak: [ip-10-1-1-245|0] (1) [ip-10-1-1-245]</div><div>...</div><div><br></div><div>In my S3 bucket, i have 2 files created :</div><div>402ea329-c135-f1e9-2782-02768779e02f.ip-10-1-1-245.list</div><div>a584321f-408b-b2ae-e2dd-d19333db96c4.ip-10-1-7-103.list</div><div><br></div><div>And the content of the files is like this :</div><div>File 1 : </div><div>ip-10-1-1-245 <span style="white-space:pre-wrap"> </span>402ea329-c135-f1e9-2782-02768779e02f <span style="white-space:pre-wrap"> </span><a href="http://10.1.1.245:7600" target="_blank">10.1.1.245:7600</a> <span style="white-space:pre-wrap"> </span>T</div><div><br></div><div>File 2 : </div><div>ip-10-1-7-103 <span style="white-space:pre-wrap"> </span>a584321f-408b-b2ae-e2dd-d19333db96c4 <span style="white-space:pre-wrap"> </span><a href="http://10.1.7.103:7600" target="_blank">10.1.7.103:7600</a> <span style="white-space:pre-wrap"> </span>T</div><div><br></div><div>When i read the logs, it looks like the infinispan's cache can't contact each other.</div><div>I double check my network config, and i tried connecting from one server to the other using nc (like this: nc -vvv 10.1.7.103 7600) and this works fine. </div><div><br></div><div>Is there a way to check the infinispan status of the servers?</div><div>Do you guys got any clue on how to make this works? </div><div><br></div><div>Thank you, </div><div>Charles-Edouard</div><div><br></div><div>My config looks like this : </div><div><br></div><div>- Standalone-ha.xml</div><div>...</div><div><datasources></div><div> <driver name="postgresql" module="org.postgresql"></div><div> <datasource-class>org.postgresql.Driver</datasource-class></div><div> <xa-datasource-class>org.postgresql.xa.PGXADataSource</xa-datasource-class></div><div> </driver></div><div> <datasource jndi-name="java:jboss/datasources/PgDskeycloak" pool-name="PgDskeycloak" enabled="true" use-java-context="true"></div><div> <connection-url>jdbc:postgresql://****:5432/keycloak?ApplicationName=keycloak</connection-url></div><div> <driver>postgresql</driver></div><div> <pool></div><div> <min-pool-size>5</min-pool-size></div><div> <initial-pool-size>5</initial-pool-size></div><div> <max-pool-size>100</max-pool-size></div><div> <prefill>true</prefill></div><div> </pool></div><div> <validation></div><div> <valid-connection-checker</div><div> class-name="org.jboss.jca.adapters.jdbc.extensions.postgres.PostgreSQLValidConnectionChecker"></valid-connection-checker></div><div> <exception-sorter</div><div> class-name="org.jboss.jca.adapters.jdbc.extensions.postgres.PostgreSQLExceptionSorter"></exception-sorter></div><div> </validation></div><div> <security></div><div> <user-name>****</user-name></div><div> <password>****</password></div><div> </security></div><div> </datasource></div><div>...</div><div> <stacks default="tcp"></div><div> <stack name="udp"></div><div> <transport type="UDP" socket-binding="jgroups-udp"/></div><div> <protocol type="PING"/></div><div> <protocol type="MERGE3"/></div><div> <protocol type="FD_SOCK" socket-binding="jgroups-udp-fd"/></div><div> <protocol type="FD_ALL"/></div><div> <protocol type="VERIFY_SUSPECT"/></div><div> <protocol type="pbcast.NAKACK2"/></div><div> <protocol type="UNICAST3"/></div><div> <protocol type="pbcast.STABLE"/></div><div> <protocol type="pbcast.GMS"/></div><div> <protocol type="UFC"/></div><div> <protocol type="MFC"/></div><div> <protocol type="FRAG2"/></div><div> <protocol type="RSVP"/></div><div> </stack></div><div> <stack name="tcp"></div><div> <transport type="TCP" socket-binding="jgroups-tcp"/></div><div> <protocol type="S3_PING" ></div><div> <property name="location">****</property></div><div> <property name="access_key">****</property></div><div> <property name="secret_access_key">****</property></div><div> </protocol></div><div> <!-- <protocol type="MPING" socket-binding="jgroups-mping"/> --></div><div> <protocol type="MERGE3"/></div><div> <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/></div><div> <protocol type="FD"/></div><div> <protocol type="VERIFY_SUSPECT"/></div><div> <protocol type="pbcast.NAKACK2"/></div><div> <protocol type="UNICAST3"/></div><div> <protocol type="pbcast.STABLE"/></div><div> <protocol type="pbcast.GMS"/></div><div> <protocol type="MFC"/></div><div> <protocol type="FRAG2"/></div><div> <protocol type="RSVP"/></div><div> </stack></div><div>...</div><div> <interfaces></div><div> <interface name="management"></div><div> <nic name="eth0"/></div><div> </interface></div><div> <interface name="public"></div><div> <nic name="eth0"/></div><div> </interface></div><div> <!-- TODO - only show this if the jacorb subsystem is added --></div><div> <interface name="unsecure"></div><div> <!--</div><div> ~ Used for IIOP sockets in the standard configuration.</div><div> ~ To secure JacORB you need to setup SSL</div><div> --></div><div> <nic name="eth0"/></div><div> </interface></div><div> </interfaces></div><div><br></div><div><span style="white-space:pre-wrap"> </span></div><div>- keycloak-server.json</div><div>{</div><div> "providers": [</div><div> "classpath:${jboss.server.config.dir}/providers/*"</div><div> ],</div><div><br></div><div> "admin": {</div><div> "realm": "master"</div><div> },</div><div><br></div><div> "eventsStore": {</div><div> "provider": "jpa",</div><div> "jpa": {</div><div> "exclude-events": [ "REFRESH_TOKEN" ]</div><div> }</div><div> },</div><div><br></div><div> "realm": {</div><div> "provider": "jpa"</div><div> },</div><div><br></div><div> "user": {</div><div> "provider": "jpa"</div><div> },</div><div><br></div><div> "userSessionPersister": {</div><div> "provider": "jpa"</div><div> },</div><div><br></div><div> "timer": {</div><div> "provider": "basic"</div><div> },</div><div><br></div><div> "theme": {</div><div> "default": "keycloak",</div><div> "staticMaxAge": 2592000,</div><div> "cacheTemplates": true,</div><div> "cacheThemes": true,</div><div> "folder": {</div><div> "dir": "${jboss.server.config.dir}/themes"</div><div> }</div><div> },</div><div><br></div><div> "scheduled": {</div><div> "interval": 900</div><div> },</div><div><br></div><div> "connectionsHttpClient": {</div><div> "default": {</div><div> "disable-trust-manager": true</div><div> }</div><div> },</div><div><br></div><div> "connectionsJpa": {</div><div> "default": {</div><div> "dataSource": "java:jboss/datasources/PgDskeycloak",</div><div> "databaseSchema": "update"</div><div> }</div><div> },</div><div><br></div><div> "connectionsInfinispan": {</div><div> "default" : {</div><div> "cacheContainer" : "java:jboss/infinispan/Keycloak"</div><div> }</div><div> }</div><div>}</div><div><br></div><div><div><div dir="ltr"><div><div dir="ltr"><table width="100%" cellpadding="0" cellspacing="0" style="font-family:'Times New Roman'" class="mceItemTable"><tbody><tr><td colspan="4" height="20"></td></tr><tr><td valign="top" align="left"><table cellpadding="0" cellspacing="0" class="mceItemTable"><tbody><tr><td width="20"></td><td style="font-family:'Exo 2',helvetica,arial,sans-serif"><span style="color:#333333"><span style="line-height:1em;font-size:20px">CHARLES-EDOUARD GAGNAIRE</span></span><br><span style="color:#6c6c6c"><span style="line-height:1.4em;font-style:italic;font-size:14px">SysAdmin</span></span> <br><span style="font-size:small"><a href="mailto:c.gagnaire@kreactive.com" style="text-decoration:none;line-height:2em" target="_blank"><span style="font-size:13px;color:#b60018">c.gagnaire@kreactive.com</span></a><br><span style="line-height:1em;font-size:14px;color:#333333">p. 06.27.80.28.53</span></span><table cellpadding="0" cellspacing="0" class="mceItemTable"><tbody><tr><td height="20"></td></tr><tr><td valign="top"><span style="font-family:'Exo 2',helvetica,arial,sans-serif;font-size:xx-small"><span style="font-style:italic;font-size:11px;color:#333333">LYON "Le Capitole"</span><br><span style="font-style:italic;font-size:10px;color:#333333">97, cours Gambetta<br>69481 Lyon Cedex 03</span> </span><br><div><br></div><span style="font-family:'Exo 2',helvetica,arial,sans-serif;font-size:xx-small"><span style="font-style:italic;font-size:11px;color:#333333">PARIS</span><br><span style="font-style:italic;font-size:10px;color:#333333">16, rue de Turbigo<br>75002 Paris</span></span></td></tr></tbody></table></td></tr></tbody></table></td><td align="center" valign="middle"><img src="http://s3.amazonaws.com/signature_kreactive/kt-signatures/place1.gif" src="http://s3.amazonaws.com/signature_kreactive/kt-signatures/place1.gif"></td><td valign="middle" align="right" style="line-height:0.8em;font-family:helvetica,arial,sans-serif"><div style="width:132px;text-align:center"><a href="http://www.kreactive.com/" target="_blank"><img alt="Kreactive" width="132" height="90" border="0" src="http://s3.amazonaws.com/signature_kreactive/kt-signatures/kreactive-corporate.png" src="http://s3.amazonaws.com/signature_kreactive/kt-signatures/kreactive-corporate.png"></a><br><div><br></div><br><a href="https://www.facebook.com/kreactive" target="_blank"><img border="0" alt="Facebook" width="17" height="17" src="http://s3.amazonaws.com/signature_kreactive/kt-signatures/icon-facebook-kreactive.gif" src="http://s3.amazonaws.com/signature_kreactive/kt-signatures/icon-facebook-kreactive.gif"></a> <a href="https://twitter.com/kreactive" target="_blank"><img border="0" alt="Twitter" width="17" height="17" src="http://s3.amazonaws.com/signature_kreactive/kt-signatures/icon-twitter-kreactive.gif" src="http://s3.amazonaws.com/signature_kreactive/kt-signatures/icon-twitter-kreactive.gif"></a></div></td><td width="15"></td></tr><tr><td colspan="4" height="10"></td></tr><tr><td colspan="4" height="4" bgcolor="#B60018"></td></tr></tbody></table></div></div></div></div></div>
</div>
<br></div></div><span class="">_______________________________________________<br>keycloak-user mailing list<br><a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a><br><a href="https://lists.jboss.org/mailman/listinfo/keycloak-user" target="_blank">https://lists.jboss.org/mailman/listinfo/keycloak-user</a></span></blockquote><div><br></div></div></div></blockquote></div><br></div>
</blockquote><div><br></div></div><br>_______________________________________________<br>keycloak-user mailing list<br>keycloak-user@lists.jboss.org<br>https://lists.jboss.org/mailman/listinfo/keycloak-user</blockquote><div><br></div></div></body></html>