From: "Alan Field" <afield@redhat.com>
To: "charles-edouard gagnaire" <c.gagnaire@kreactive.com>
Cc: keycloak-user@lists.jboss.org
Sent: Friday, December 18, 2015 10:21:54 AM
Subject: Re: [keycloak-user] Problem running keycloak cluster on EC2 with S3_ping

Hey Charles,

Thanks for the logs. I'm not sure what is wrong, but it looks like each server is creating a cluster of 1. I'll try it with my AWS account to see if I can figure out what is wrong.

Alan

---

From: "charles-edouard gagnaire" <c.gagnaire@kreactive.com>
To: "Alan Field" <afield@redhat.com>
Cc: keycloak-user@lists.jboss.org
Sent: Friday, December 18, 2015 10:04:53 AM
Subject: Re: [keycloak-user] Problem running keycloak cluster on EC2 with S3_ping

First i want to thank you guys for the quick answers, i was still reading the "Replace use of Infinispan with User Sessions SPI ?" discussion. 

Yes of course i can send all the logs. You'll find them below.  

The JGroups version is the one shipping with keycloak 1.7, but the problem was the same with Keycloak 1.6.

Looking at the config file, it looks like i'm using : <subsystem xmlns="urn:jboss:domain:jgroups:3.0">

I didn't mention it but i use the archive i found on Keycloak website. The archive is "keycloak-1.7.0.Final.tar.gz". 

I just untar and modify the config files, then i launched it using : /opt/keycloak-1.7.0.Final/bin/standalone.sh -c standalone-ha.xml

Thank you again for your help 

The logs for server 1 are : 

=========================================================================

  JBoss Bootstrap Environment

  JBOSS_HOME: /opt/keycloak-1.7.0.Final

  JAVA: /usr/lib/jvm/jre/bin/java

  JAVA_OPTS:  -server -XX:+UseCompressedOops  -server -XX:+UseCompressedOops -Xms64m -Xmx512m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true

=========================================================================

OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0

14:38:44,910 INFO  [org.jboss.modules] (main) JBoss Modules version 1.4.3.Final

14:38:45,091 INFO  [org.jboss.msc] (main) JBoss MSC version 1.2.6.Final

14:38:45,163 INFO  [org.jboss.as] (MSC service thread 1-2) WFLYSRV0049: Keycloak 1.7.0.Final (WildFly Core 1.0.2.Final) starting

14:38:46,358 INFO  [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 24) WFLYCTL0028: Attribute 'job-repository-type' in the resource at address '/subsystem=batch' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.

14:38:46,360 INFO  [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 14) WFLYCTL0028: Attribute 'enabled' in the resource at address '/subsystem=datasources/data-source=PgDskeycloak' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.

14:38:46,362 INFO  [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 14) WFLYCTL0028: Attribute 'enabled' in the resource at address '/subsystem=datasources/data-source=ExampleDS' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.

14:38:46,362 INFO  [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 14) WFLYCTL0028: Attribute 'enabled' in the resource at address '/subsystem=datasources/data-source=KeycloakDS' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.

14:38:46,370 INFO  [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 21) WFLYCTL0028: Attribute 'default-stack' in the resource at address '/subsystem=jgroups' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.

14:38:46,572 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http)

14:38:46,589 INFO  [org.xnio] (MSC service thread 1-4) XNIO version 3.3.1.Final

14:38:46,607 INFO  [org.xnio.nio] (MSC service thread 1-4) XNIO NIO Implementation Version 3.3.1.Final

14:38:46,655 INFO  [org.jboss.remoting] (MSC service thread 1-4) JBoss Remoting version 4.0.9.Final

14:38:46,687 INFO  [org.wildfly.extension.io] (ServerService Thread Pool -- 38) WFLYIO001: Worker 'default' has auto-configured to 4 core threads with 32 task threads based on your 2 available processors

14:38:46,685 INFO  [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 35) WFLYJCA0005: Deploying non-JDBC-compliant driver class org.postgresql.Driver (version 9.4)

14:38:46,715 INFO  [org.jboss.as.clustering.jgroups] (ServerService Thread Pool -- 43) WFLYCLJG0001: Activating JGroups subsystem.

14:38:46,724 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 39) WFLYCLINF0001: Activating Infinispan subsystem.

14:38:46,744 INFO  [org.jboss.as.connector] (MSC service thread 1-4) WFLYJCA0009: Starting JCA Subsystem (IronJacamar 1.2.5.Final)

14:38:46,746 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-2) WFLYJCA0018: Started Driver service with driver-name = postgresql

14:38:46,767 INFO  [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 35) WFLYJCA0004: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3)

14:38:46,769 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-2) WFLYJCA0018: Started Driver service with driver-name = h2

14:38:46,781 INFO  [org.jboss.as.jsf] (ServerService Thread Pool -- 46) WFLYJSF0007: Activated the following JSF Implementations: [main]

14:38:46,772 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 49) WFLYNAM0001: Activating Naming Subsystem

14:38:46,914 INFO  [org.jboss.as.security] (ServerService Thread Pool -- 56) WFLYSEC0002: Activating Security Subsystem

14:38:46,916 INFO  [org.jboss.as.security] (MSC service thread 1-1) WFLYSEC0001: Current PicketBox version=4.9.2.Final

14:38:46,932 WARN  [org.jboss.as.txn] (ServerService Thread Pool -- 57) WFLYTX0013: Node identifier property is set to the default value. Please make sure it is unique.

14:38:46,957 INFO  [org.jboss.as.webservices] (ServerService Thread Pool -- 59) WFLYWS0002: Activating WebServices Extension

14:38:46,985 INFO  [org.jboss.as.naming] (MSC service thread 1-2) WFLYNAM0003: Starting Naming Service

14:38:46,992 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-4) WFLYMAIL0001: Bound mail session [java:jboss/mail/Default]

14:38:47,115 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0003: Undertow 1.2.9.Final starting

14:38:47,119 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 58) WFLYUT0003: Undertow 1.2.9.Final starting

14:38:47,206 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 58) WFLYUT0014: Creating file handler for path /opt/keycloak-1.7.0.Final/welcome-content

14:38:47,229 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0012: Started server default-server.

14:38:47,263 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0018: Host default-host starting

14:38:47,320 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0006: Undertow AJP listener ajp listening on /10.1.7.103:8009

14:38:47,324 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0006: Undertow HTTP listener default listening on /10.1.7.103:8080

14:38:47,339 INFO  [org.jboss.modcluster] (ServerService Thread Pool -- 62) MODCLUSTER000001: Initializing mod_cluster version 1.3.1.Final

14:38:47,372 INFO  [org.jboss.modcluster] (ServerService Thread Pool -- 62) MODCLUSTER000032: Listening to proxy advertisements on /224.0.1.105:23364

14:38:47,478 INFO  [org.jboss.as.connector.subsystems.datasources.AbstractDataSourceService$AS7DataSourceDeployer] (MSC service thread 1-1) IJ020018: Enabling <validate-on-match> for java:jboss/datasources/PgDskeycloak

14:38:47,513 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0001: Bound data source [java:jboss/datasources/KeycloakDS]

14:38:47,513 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0001: Bound data source [java:jboss/datasources/ExampleDS]

14:38:47,530 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0001: Bound data source [java:jboss/datasources/PgDskeycloak]

14:38:47,673 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-4) WFLYSRV0027: Starting deployment of "keycloak-server.war" (runtime-name: "keycloak-server.war")

14:38:47,820 INFO  [org.jboss.ws.common.management] (MSC service thread 1-3) JBWS022052: Starting JBoss Web Services - Stack CXF Server 5.0.0.Final

14:38:48,898 INFO  [stdout] (MSC service thread 1-2)

14:38:48,898 INFO  [stdout] (MSC service thread 1-2) -------------------------------------------------------------------

14:38:48,898 INFO  [stdout] (MSC service thread 1-2) GMS: address=ip-10-1-7-103, cluster=ee, physical address=10.1.7.103:7600

14:38:48,899 INFO  [stdout] (MSC service thread 1-2) -------------------------------------------------------------------

14:38:49,250 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 62) ISPN000078: Starting JGroups channel keycloak

14:38:49,265 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 62) ISPN000094: Received new cluster view for channel keycloak: [ip-10-1-7-103|0] (1) [ip-10-1-7-103]

14:38:49,273 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 62) ISPN000079: Channel keycloak local address is ip-10-1-7-103, physical addresses are [10.1.7.103:7600]

14:38:49,277 INFO  [org.infinispan.factories.GlobalComponentRegistry] (ServerService Thread Pool -- 62) ISPN000128: Infinispan version: Infinispan 'Insanely Bad Elf' 7.2.3.Final

14:38:49,521 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 65) WFLYCLINF0002: Started users cache from keycloak container

14:38:49,529 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 62) WFLYCLINF0002: Started loginFailures cache from keycloak container

14:38:49,530 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 66) WFLYCLINF0002: Started sessions cache from keycloak container

14:38:49,536 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 63) WFLYCLINF0002: Started realms cache from keycloak container

14:38:50,116 INFO  [org.keycloak.services.resources.KeycloakApplication] (ServerService Thread Pool -- 66) Load config from /opt/keycloak-1.7.0.Final/standalone/configuration/keycloak-server.json

14:38:50,638 INFO  [org.hibernate.jpa.internal.util.LogHelper] (ServerService Thread Pool -- 66) HHH000204: Processing PersistenceUnitInfo [

        name: keycloak-default

        ...]

14:38:50,690 INFO  [org.hibernate.Version] (ServerService Thread Pool -- 66) HHH000412: Hibernate Core {4.3.10.Final}

14:38:50,691 INFO  [org.hibernate.cfg.Environment] (ServerService Thread Pool -- 66) HHH000206: hibernate.properties not found

14:38:50,693 INFO  [org.hibernate.cfg.Environment] (ServerService Thread Pool -- 66) HHH000021: Bytecode provider name : javassist

14:38:50,842 INFO  [org.hibernate.annotations.common.Version] (ServerService Thread Pool -- 66) HCANN000001: Hibernate Commons Annotations {4.0.5.Final}

14:38:51,794 INFO  [org.hibernate.dialect.Dialect] (ServerService Thread Pool -- 66) HHH000400: Using dialect: org.hibernate.dialect.PostgreSQL9Dialect

14:38:51,803 INFO  [org.hibernate.engine.jdbc.internal.LobCreatorBuilder] (ServerService Thread Pool -- 66) HHH000424: Disabling contextual LOB creation as createClob() method threw error : java.lang.reflect.InvocationTargetException

14:38:52,120 INFO  [org.hibernate.hql.internal.ast.ASTQueryTranslatorFactory] (ServerService Thread Pool -- 66) HHH000397: Using ASTQueryTranslatorFactory

14:38:52,156 INFO  [org.hibernate.validator.internal.util.Version] (ServerService Thread Pool -- 66) HV000001: Hibernate Validator 5.1.3.Final

14:38:53,706 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 66) WFLYCLINF0002: Started offlineSessions cache from keycloak container

14:38:53,748 INFO  [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 66) Deploying javax.ws.rs.core.Application: class org.keycloak.services.resources.KeycloakApplication

14:38:53,750 INFO  [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 66) Adding class resource org.keycloak.services.resources.WelcomeResource from Application class org.keycloak.services.resources.KeycloakApplication

14:38:53,750 INFO  [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 66) Adding class resource org.keycloak.services.resources.JsResource from Application class org.keycloak.services.resources.KeycloakApplication

14:38:53,750 INFO  [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 66) Adding class resource org.keycloak.services.resources.QRCodeResource from Application class org.keycloak.services.resources.KeycloakApplication

14:38:53,750 INFO  [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 66) Adding class resource org.keycloak.services.resources.ThemeResource from Application class org.keycloak.services.resources.KeycloakApplication

14:38:53,751 INFO  [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 66) Adding singleton resource org.keycloak.services.resources.RealmsResource from Application class org.keycloak.services.resources.KeycloakApplication

14:38:53,751 INFO  [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 66) Adding singleton resource org.keycloak.services.resources.ServerVersionResource from Application class org.keycloak.services.resources.KeycloakApplication

14:38:53,751 INFO  [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 66) Adding singleton resource org.keycloak.services.resources.admin.AdminRoot from Application class org.keycloak.services.resources.KeycloakApplication

14:38:53,751 INFO  [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 66) Adding provider singleton org.keycloak.services.util.ObjectMapperResolver from Application class org.keycloak.services.resources.KeycloakApplication

14:38:53,752 INFO  [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 66) Adding provider singleton org.keycloak.services.resources.ModelExceptionMapper from Application class org.keycloak.services.resources.KeycloakApplication

14:38:53,824 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 66) WFLYUT0021: Registered web context: /auth

14:38:53,920 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 61) WFLYSRV0010: Deployed "keycloak-server.war" (runtime-name : "keycloak-server.war")

14:38:54,021 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://10.1.7.103:9990/management

14:38:54,021 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://10.1.7.103:9990

14:38:54,022 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Keycloak 1.7.0.Final (WildFly Core 1.0.2.Final) started in 9388ms - Started 349 of 613 services (353 services are lazy, passive or on-demand)

The logs for server 2 are : 

=========================================================================

  JBoss Bootstrap Environment

  JBOSS_HOME: /opt/keycloak-1.7.0.Final

  JAVA: /usr/lib/jvm/jre/bin/java

  JAVA_OPTS:  -server -XX:+UseCompressedOops  -server -XX:+UseCompressedOops -Xms64m -Xmx512m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true

=========================================================================

14:38:48,239 INFO  [org.jboss.modules] (main) JBoss Modules version 1.4.3.Final

14:38:48,723 INFO  [org.jboss.msc] (main) JBoss MSC version 1.2.6.Final

14:38:48,896 INFO  [org.jboss.as] (MSC service thread 1-2) WFLYSRV0049: Keycloak 1.7.0.Final (WildFly Core 1.0.2.Final) starting

14:38:50,979 INFO  [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 8) WFLYCTL0028: Attribute 'job-repository-type' in the resource at address '/subsystem=batch' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.

14:38:50,983 INFO  [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 11) WFLYCTL0028: Attribute 'enabled' in the resource at address '/subsystem=datasources/data-source=PgDskeycloak' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.

14:38:50,986 INFO  [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 11) WFLYCTL0028: Attribute 'enabled' in the resource at address '/subsystem=datasources/data-source=ExampleDS' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.

14:38:51,010 INFO  [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 14) WFLYCTL0028: Attribute 'default-stack' in the resource at address '/subsystem=jgroups' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.

14:38:51,044 INFO  [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 11) WFLYCTL0028: Attribute 'enabled' in the resource at address '/subsystem=datasources/data-source=KeycloakDS' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.

14:38:51,452 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http)

14:38:51,499 INFO  [org.xnio] (MSC service thread 1-1) XNIO version 3.3.1.Final

14:38:51,520 INFO  [org.xnio.nio] (MSC service thread 1-1) XNIO NIO Implementation Version 3.3.1.Final

14:38:51,590 INFO  [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 35) WFLYJCA0005: Deploying non-JDBC-compliant driver class org.postgresql.Driver (version 9.4)

14:38:51,603 INFO  [org.wildfly.extension.io] (ServerService Thread Pool -- 38) WFLYIO001: Worker 'default' has auto-configured to 2 core threads with 16 task threads based on your 1 available processors

14:38:51,601 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 39) WFLYCLINF0001: Activating Infinispan subsystem.

14:38:51,634 INFO  [org.jboss.as.clustering.jgroups] (ServerService Thread Pool -- 43) WFLYCLJG0001: Activating JGroups subsystem.

14:38:51,694 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 49) WFLYNAM0001: Activating Naming Subsystem

14:38:51,666 INFO  [org.jboss.as.jsf] (ServerService Thread Pool -- 46) WFLYJSF0007: Activated the following JSF Implementations: [main]

14:38:51,696 INFO  [org.jboss.as.connector] (MSC service thread 1-2) WFLYJCA0009: Starting JCA Subsystem (IronJacamar 1.2.5.Final)

14:38:51,932 INFO  [org.jboss.as.webservices] (ServerService Thread Pool -- 59) WFLYWS0002: Activating WebServices Extension

14:38:51,970 INFO  [org.jboss.remoting] (MSC service thread 1-1) JBoss Remoting version 4.0.9.Final

14:38:51,975 INFO  [org.jboss.as.security] (ServerService Thread Pool -- 56) WFLYSEC0002: Activating Security Subsystem

14:38:51,972 INFO  [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 35) WFLYJCA0004: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3)

14:38:51,971 WARN  [org.jboss.as.txn] (ServerService Thread Pool -- 57) WFLYTX0013: Node identifier property is set to the default value. Please make sure it is unique.

14:38:52,140 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 58) WFLYUT0003: Undertow 1.2.9.Final starting

14:38:52,187 INFO  [org.jboss.as.security] (MSC service thread 1-2) WFLYSEC0001: Current PicketBox version=4.9.2.Final

14:38:52,224 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-1) WFLYJCA0018: Started Driver service with driver-name = postgresql

14:38:52,225 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-1) WFLYJCA0018: Started Driver service with driver-name = h2

14:38:52,368 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0003: Undertow 1.2.9.Final starting

14:38:52,369 INFO  [org.jboss.as.naming] (MSC service thread 1-2) WFLYNAM0003: Starting Naming Service

14:38:52,471 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-2) WFLYMAIL0001: Bound mail session [java:jboss/mail/Default]

14:38:52,710 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 58) WFLYUT0014: Creating file handler for path /opt/keycloak-1.7.0.Final/welcome-content

14:38:52,864 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0012: Started server default-server.

14:38:53,133 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0006: Undertow HTTP listener default listening on /10.1.1.245:8080

14:38:53,166 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0018: Host default-host starting

14:38:53,192 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0006: Undertow AJP listener ajp listening on /10.1.1.245:8009

14:38:53,211 INFO  [org.jboss.modcluster] (ServerService Thread Pool -- 62) MODCLUSTER000001: Initializing mod_cluster version 1.3.1.Final

14:38:53,307 INFO  [org.jboss.modcluster] (ServerService Thread Pool -- 62) MODCLUSTER000032: Listening to proxy advertisements on /224.0.1.105:23364

14:38:53,779 INFO  [org.jboss.as.connector.subsystems.datasources.AbstractDataSourceService$AS7DataSourceDeployer] (MSC service thread 1-2) IJ020018: Enabling <validate-on-match> for java:jboss/datasources/PgDskeycloak

14:38:53,896 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-1) WFLYJCA0001: Bound data source [java:jboss/datasources/KeycloakDS]

14:38:53,903 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0001: Bound data source [java:jboss/datasources/ExampleDS]

14:38:53,909 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0001: Bound data source [java:jboss/datasources/PgDskeycloak]

14:38:54,118 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0027: Starting deployment of "keycloak-server.war" (runtime-name: "keycloak-server.war")

14:38:54,306 INFO  [org.jboss.ws.common.management] (MSC service thread 1-1) JBWS022052: Starting JBoss Web Services - Stack CXF Server 5.0.0.Final

14:38:56,138 INFO  [stdout] (MSC service thread 1-2)

14:38:56,138 INFO  [stdout] (MSC service thread 1-2) -------------------------------------------------------------------

14:38:56,139 INFO  [stdout] (MSC service thread 1-2) GMS: address=ip-10-1-1-245, cluster=ee, physical address=10.1.1.245:7600

14:38:56,139 INFO  [stdout] (MSC service thread 1-2) -------------------------------------------------------------------

14:38:56,606 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 62) ISPN000078: Starting JGroups channel keycloak

14:38:56,623 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 62) ISPN000094: Received new cluster view for channel keycloak: [ip-10-1-1-245|0] (1) [ip-10-1-1-245]

14:38:56,644 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 62) ISPN000079: Channel keycloak local address is ip-10-1-1-245, physical addresses are [10.1.1.245:7600]

14:38:56,651 INFO  [org.infinispan.factories.GlobalComponentRegistry] (ServerService Thread Pool -- 62) ISPN000128: Infinispan version: Infinispan 'Insanely Bad Elf' 7.2.3.Final

14:38:57,044 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 65) WFLYCLINF0002: Started users cache from keycloak container

14:38:57,050 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 62) WFLYCLINF0002: Started sessions cache from keycloak container

14:38:57,055 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 63) WFLYCLINF0002: Started realms cache from keycloak container

14:38:57,059 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 64) WFLYCLINF0002: Started loginFailures cache from keycloak container

14:38:58,007 INFO  [org.keycloak.services.resources.KeycloakApplication] (ServerService Thread Pool -- 64) Load config from /opt/keycloak-1.7.0.Final/standalone/configuration/keycloak-server.json

14:38:58,755 INFO  [org.hibernate.jpa.internal.util.LogHelper] (ServerService Thread Pool -- 64) HHH000204: Processing PersistenceUnitInfo [

        name: keycloak-default

        ...]

14:38:58,812 INFO  [org.hibernate.Version] (ServerService Thread Pool -- 64) HHH000412: Hibernate Core {4.3.10.Final}

14:38:58,819 INFO  [org.hibernate.cfg.Environment] (ServerService Thread Pool -- 64) HHH000206: hibernate.properties not found

14:38:58,824 INFO  [org.hibernate.cfg.Environment] (ServerService Thread Pool -- 64) HHH000021: Bytecode provider name : javassist

14:38:59,268 INFO  [org.hibernate.annotations.common.Version] (ServerService Thread Pool -- 64) HCANN000001: Hibernate Commons Annotations {4.0.5.Final}

14:39:00,264 INFO  [org.hibernate.dialect.Dialect] (ServerService Thread Pool -- 64) HHH000400: Using dialect: org.hibernate.dialect.PostgreSQL9Dialect

14:39:00,272 INFO  [org.hibernate.engine.jdbc.internal.LobCreatorBuilder] (ServerService Thread Pool -- 64) HHH000424: Disabling contextual LOB creation as createClob() method threw error : java.lang.reflect.InvocationTargetException

14:39:00,602 INFO  [org.hibernate.hql.internal.ast.ASTQueryTranslatorFactory] (ServerService Thread Pool -- 64) HHH000397: Using ASTQueryTranslatorFactory

14:39:00,634 INFO  [org.hibernate.validator.internal.util.Version] (ServerService Thread Pool -- 64) HV000001: Hibernate Validator 5.1.3.Final

14:39:04,607 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 64) WFLYCLINF0002: Started offlineSessions cache from keycloak container

14:39:04,665 INFO  [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 64) Deploying javax.ws.rs.core.Application: class org.keycloak.services.resources.KeycloakApplication

14:39:04,667 INFO  [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 64) Adding class resource org.keycloak.services.resources.WelcomeResource from Application class org.keycloak.services.resources.KeycloakApplication

14:39:04,667 INFO  [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 64) Adding class resource org.keycloak.services.resources.QRCodeResource from Application class org.keycloak.services.resources.KeycloakApplication

14:39:04,668 INFO  [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 64) Adding class resource org.keycloak.services.resources.JsResource from Application class org.keycloak.services.resources.KeycloakApplication

14:39:04,668 INFO  [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 64) Adding class resource org.keycloak.services.resources.ThemeResource from Application class org.keycloak.services.resources.KeycloakApplication

14:39:04,668 INFO  [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 64) Adding provider singleton org.keycloak.services.resources.ModelExceptionMapper from Application class org.keycloak.services.resources.KeycloakApplication

14:39:04,669 INFO  [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 64) Adding singleton resource org.keycloak.services.resources.RealmsResource from Application class org.keycloak.services.resources.KeycloakApplication

14:39:04,669 INFO  [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 64) Adding provider singleton org.keycloak.services.util.ObjectMapperResolver from Application class org.keycloak.services.resources.KeycloakApplication

14:39:04,669 INFO  [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 64) Adding singleton resource org.keycloak.services.resources.ServerVersionResource from Application class org.keycloak.services.resources.KeycloakApplication

14:39:04,669 INFO  [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 64) Adding singleton resource org.keycloak.services.resources.admin.AdminRoot from Application class org.keycloak.services.resources.KeycloakApplication

14:39:04,757 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 64) WFLYUT0021: Registered web context: /auth

14:39:04,844 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 61) WFLYSRV0010: Deployed "keycloak-server.war" (runtime-name : "keycloak-server.war")

14:39:05,526 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://10.1.1.245:9990/management

14:39:05,527 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://10.1.1.245:9990

14:39:05,531 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Keycloak 1.7.0.Final (WildFly Core 1.0.2.Final) started in 17727ms - Started 349 of 613 services (353 services are lazy, passive or on-demand)

CHARLES-EDOUARD GAGNAIRE SysAdmin  c.gagnaire@kreactive.com p. 06.27.80.28.53 LYON "Le Capitole" 97, cours Gambetta 69481 Lyon Cedex 03  PARIS 16, rue de Turbigo 75002 Paris

2015-12-18 15:53 GMT+01:00 Alan Field <afield@redhat.com>:

Hey Charles,

Can you send the full logs and tell me which version of JGroups you are using?

Thanks,

Alan

---

From: "charles-edouard gagnaire" <c.gagnaire@kreactive.com>
To: keycloak-user@lists.jboss.org
Sent: Friday, December 18, 2015 9:01:12 AM
Subject: [keycloak-user] Problem running keycloak cluster on EC2 with S3_ping

hi, 

I'm having trouble configuring a Keycloak cluster running on AWS' EC2. 

The database configuration is OK no problem, but i can't manage to get the invalidation cache working correctly. 

I configured Infinispan to work with S3_ping plugin (the relevant part of my configuration is below). 

When i run both server, the connection with the database is Ok, but the infinispan logs look like this : 

On Server 1 : 

...

11:00:17,592 INFO  [stdout] (MSC service thread 1-1) GMS: address=ip-10-1-7-103, cluster=ee, physical address=10.1.7.103:7600

...

11:00:18,057 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 62) ISPN000094: Received new cluster view for channel keycloak: [ip-10-1-7-103|0] (1) [ip-10-1-7-103]

...

On Server 2 : 

...

11:03:41,159 INFO  [stdout] (MSC service thread 1-1) GMS: address=ip-10-1-1-245, cluster=ee, physical address=10.1.1.245:7600

...

11:03:41,783 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 62) ISPN000094: Received new cluster view for channel keycloak: [ip-10-1-1-245|0] (1) [ip-10-1-1-245]

...

In my S3 bucket, i have 2 files created :

402ea329-c135-f1e9-2782-02768779e02f.ip-10-1-1-245.list

a584321f-408b-b2ae-e2dd-d19333db96c4.ip-10-1-7-103.list

And the content of the files is like this :

File 1 : 

ip-10-1-1-245 402ea329-c135-f1e9-2782-02768779e02f 10.1.1.245:7600 T

File 2 : 

ip-10-1-7-103 a584321f-408b-b2ae-e2dd-d19333db96c4 10.1.7.103:7600 T

When i read the logs, it looks like the infinispan's cache can't contact each other.

I double check my network config, and i tried connecting from one server to the other using nc (like this: nc -vvv 10.1.7.103 7600) and this works fine. 

Is there a way to check the infinispan status of the servers?

Do you guys got any clue on how to make this works? 

Thank you, 

Charles-Edouard

My config looks like this : 

- Standalone-ha.xml

...

<datasources>

                <driver name="postgresql" module="org.postgresql">

                   <datasource-class>org.postgresql.Driver</datasource-class>

                   <xa-datasource-class>org.postgresql.xa.PGXADataSource</xa-datasource-class>

                </driver>

                <datasource jndi-name="java:jboss/datasources/PgDskeycloak" pool-name="PgDskeycloak" enabled="true" use-java-context="true">

                    <connection-url>jdbc:postgresql://****:5432/keycloak?ApplicationName=keycloak</connection-url>

                    <driver>postgresql</driver>

                    <pool>

                        <min-pool-size>5</min-pool-size>

                        <initial-pool-size>5</initial-pool-size>

                        <max-pool-size>100</max-pool-size>

                        <prefill>true</prefill>

                    </pool>

                    <validation>

                        <valid-connection-checker

                                 class-name="org.jboss.jca.adapters.jdbc.extensions.postgres.PostgreSQLValidConnectionChecker"></valid-connection-checker>

                        <exception-sorter

                                 class-name="org.jboss.jca.adapters.jdbc.extensions.postgres.PostgreSQLExceptionSorter"></exception-sorter>

                    </validation>

                    <security>

                        <user-name>****</user-name>

                        <password>****</password>

                    </security>

                </datasource>

...

            <stacks default="tcp">

                <stack name="udp">

                    <transport type="UDP" socket-binding="jgroups-udp"/>

                    <protocol type="PING"/>

                    <protocol type="MERGE3"/>

                    <protocol type="FD_SOCK" socket-binding="jgroups-udp-fd"/>

                    <protocol type="FD_ALL"/>

                    <protocol type="VERIFY_SUSPECT"/>

                    <protocol type="pbcast.NAKACK2"/>

                    <protocol type="UNICAST3"/>

                    <protocol type="pbcast.STABLE"/>

                    <protocol type="pbcast.GMS"/>

                    <protocol type="UFC"/>

                    <protocol type="MFC"/>

                    <protocol type="FRAG2"/>

                    <protocol type="RSVP"/>

                </stack>

                <stack name="tcp">

                    <transport type="TCP" socket-binding="jgroups-tcp"/>

                    <protocol type="S3_PING" >

                        <property name="location">****</property>

                        <property name="access_key">****</property>

                        <property name="secret_access_key">****</property>

                    </protocol>

                    <!-- <protocol type="MPING" socket-binding="jgroups-mping"/> -->

                    <protocol type="MERGE3"/>

                    <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>

                    <protocol type="FD"/>

                    <protocol type="VERIFY_SUSPECT"/>

                    <protocol type="pbcast.NAKACK2"/>

                    <protocol type="UNICAST3"/>

                    <protocol type="pbcast.STABLE"/>

                    <protocol type="pbcast.GMS"/>

                    <protocol type="MFC"/>

                    <protocol type="FRAG2"/>

                    <protocol type="RSVP"/>

                </stack>

...

    <interfaces>

        <interface name="management">

                <nic name="eth0"/>

        </interface>

        <interface name="public">

            <nic name="eth0"/>

        </interface>

        <!-- TODO - only show this if the jacorb subsystem is added  -->

        <interface name="unsecure">

            <!--

              ~  Used for IIOP sockets in the standard configuration.

              ~                  To secure JacORB you need to setup SSL

              -->

            <nic name="eth0"/>

        </interface>

    </interfaces>

- keycloak-server.json

{

    "providers": [

        "classpath:${jboss.server.config.dir}/providers/*"

    ],

    "admin": {

        "realm": "master"

    },

    "eventsStore": {

        "provider": "jpa",

        "jpa": {

            "exclude-events": [ "REFRESH_TOKEN" ]

        }

    },

    "realm": {

        "provider": "jpa"

    },

    "user": {

        "provider": "jpa"

    },

    "userSessionPersister": {

        "provider": "jpa"

    },

    "timer": {

        "provider": "basic"

    },

    "theme": {

        "default": "keycloak",

        "staticMaxAge": 2592000,

        "cacheTemplates": true,

        "cacheThemes": true,

        "folder": {

          "dir": "${jboss.server.config.dir}/themes"

        }

    },

    "scheduled": {

        "interval": 900

    },

    "connectionsHttpClient": {

        "default": {

            "disable-trust-manager": true

        }

    },

    "connectionsJpa": {

        "default": {

            "dataSource": "java:jboss/datasources/PgDskeycloak",

            "databaseSchema": "update"

        }

    },

    "connectionsInfinispan": {

        "default" : {

            "cacheContainer" : "java:jboss/infinispan/Keycloak"

        }

    }

}

CHARLES-EDOUARD GAGNAIRE SysAdmin  c.gagnaire@kreactive.com p. 06.27.80.28.53 LYON "Le Capitole" 97, cours Gambetta 69481 Lyon Cedex 03  PARIS 16, rue de Turbigo 75002 Paris

_______________________________________________
keycloak-user mailing list
keycloak-user@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-user

_______________________________________________
keycloak-user mailing list
keycloak-user@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-user