[jboss-jira] [JBoss JIRA] (WFLY-11535) Cluster fails to merge if instances started simultaneously

[Ian Rodgers (Jira)]

    [ https://issues.jboss.org/browse/WFLY-11535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13685716#comment-13685716 ] 

Ian Rodgers commented on WFLY-11535:
------------------------------------

[~pferraro] - Yes, we're using Keycloak 4.6.0 which uses Wildfly 14 under the covers - https://www.keycloak.org/docs/latest/release_notes/index.html#keycloak-4-6-0-final
According to the logs it includes JGroups 4.0.13 which supposedly fixes JGRP-2281.


=========================================================================
JBoss Bootstrap Environment
JBOSS_HOME: /opt/jboss/keycloak
JAVA: /usr/lib/jvm/java/bin/java
JAVA_OPTS: -server -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
=========================================================================
INFO [org.jboss.modules] (main) JBoss Modules version 1.8.6.Final
INFO [org.jboss.msc] (main) JBoss MSC version 1.4.3.Final
INFO [org.jboss.threads] (main) JBoss Threads version 2.3.2.Final
INFO [org.jboss.as] (MSC service thread 1-1) WFLYSRV0049: Keycloak 4.6.0.Final (WildFly Core 6.0.2.Final) starting
INFO [org.wildfly.security] (ServerService Thread Pool -- 20) ELY00001: WildFly Elytron version 1.6.0.Final
INFO [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/core-service=management/management-interface=http-interface' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the depr
INFO [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 4) WFLYCTL0028: Attribute 'default-stack' in the resource at address '/subsystem=jgroups' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
INFO [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 9) WFLYCTL0028: Attribute 'permissions' in the resource at address '/subsystem=elytron/constant-permission-mapper=constant-permission-mapper' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn mo
INFO [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 25) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/subsystem=undertow/server=default-server/https-listener=https' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more abo
INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http)
INFO [org.xnio] (MSC service thread 1-1) XNIO version 3.6.5.Final
INFO [org.xnio.nio] (MSC service thread 1-1) XNIO NIO Implementation Version 3.6.5.Final
INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 34) WFLYCLINF0001: Activating Infinispan subsystem.
INFO [org.jboss.as.clustering.jgroups] (ServerService Thread Pool -- 38) WFLYCLJG0001: Activating JGroups subsystem. JGroups version 4.0.13
INFO [org.jboss.as.naming] (ServerService Thread Pool -- 43) WFLYNAM0001: Activating Naming Subsystem
INFO [org.wildfly.extension.io] (ServerService Thread Pool -- 35) WFLYIO001: Worker 'default' has auto-configured to 2 core threads with 16 task threads based on your 1 available processors
WARN [org.jboss.as.txn] (ServerService Thread Pool -- 48) WFLYTX0013: The node-identifier attribute on the /subsystem=transactions is set to the default value. This is a danger for environments running multiple servers. Please make sure the attribute value is unique.
INFO [org.jboss.as.jaxrs] (ServerService Thread Pool -- 36) WFLYRS0016: RESTEasy version 3.6.1.Final
INFO [org.jboss.as.security] (ServerService Thread Pool -- 46) WFLYSEC0002: Activating Security Subsystem
INFO [org.jboss.as.mail.extension] (MSC service thread 1-2) WFLYMAIL0002: Unbound mail session [java:jboss/mail/Default]
INFO [org.jboss.as.connector] (MSC service thread 1-2) WFLYJCA0009: Starting JCA Subsystem (WildFly/IronJacamar 1.4.11.Final)
INFO [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 29) WFLYJCA0004: Deploying JDBC-compliant driver class org.h2.Driver (version 1.4)
INFO [org.jboss.as.security] (MSC service thread 1-2) WFLYSEC0001: Current PicketBox version=5.0.3.Final
INFO [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 29) WFLYJCA0005: Deploying non-JDBC-compliant driver class org.postgresql.Driver (version 42.2)
INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0003: Undertow 2.0.13.Final starting


> Cluster fails to merge if instances started simultaneously
> ----------------------------------------------------------
>
>                 Key: WFLY-11535
>                 URL: https://issues.jboss.org/browse/WFLY-11535
>             Project: WildFly
>          Issue Type: Bug
>          Components: Clustering
>    Affects Versions: 14.0.0.Final
>         Environment: Keycloak 4.6.0 official docker image, AWS ECS cluster
>            Reporter: Ian Rodgers
>            Assignee: Paul Ferraro
>            Priority: Major
>         Attachments: standalone-ha.xml
>
>
> Our Keycloak docker cluster has four instances, clustered using Jgroups/Infinispan as per the standalone-ha.xml. If you start them all simultaneously the "Receive new cluster" logs indicate four separate clusters, each with a single member. They never get merged into the proper single cluster of four members. It seems to be the merging that has changed. The application then fails (we are not using sticky sessions, and each member is ignorant of the sessions on the other members).
> We can only start the cluster by first starting one instance, then when it is running, starting the other three. The logs then indicate the creation of a single cluster which subsequent instances join.
> This is consistent behaviour, and when we revert back to v.4.5.0, the issue goes away, Unfortunately we need 4.6.0 for an important fix.
> On 4.5.0 we get the message "Received new, MERGED cluster view for channel ejb: MergeView::" when it detects a number of subgroups to merge. This never appears in 4.6.0.



--
This message was sent by Atlassian Jira
(v7.12.1#712002)