3:02 p.m.
[
https://issues.jboss.org/browse/JGRP-2300?page=com.atlassian.jira.plugin....
]
Eric Thompson commented on JGRP-2300:
-------------------------------------
Hello [~sebastian.laskawiec]! In the dynamic port mapping example this is a typical DNS
entry:
{code}
$ dig jgroups.dev.auth.example.com.svc.cluster.local SRV
; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.68.rc1.58.amzn1 <<>>
jgroups.dev.auth.example.com.svc.cluster.local SRV
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16690
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;jgroups.dev.auth.example.com.svc.cluster.local. IN SRV
;; ANSWER SECTION:
jgroups.dev.auth.example.com.svc.cluster.local. 10 IN SRV 1 1 32921
9ec82e3f-3a0e-4e30-b785-17879c63cd7d.jgroups.dev.auth.example.com.svc.cluster.local.
jgroups.dev.auth.example.com.svc.cluster.local. 10 IN SRV 1 1 32923
60b5a820-9678-4bd2-84c6-00061a52bde0.jgroups.dev.auth.example.com.svc.cluster.local.
jgroups.dev.auth.example.com.svc.cluster.local. 10 IN SRV 1 1 32915
9d9d78d0-8919-4b91-9df8-2e4e65afedae.jgroups.dev.auth.example.com.svc.cluster.local.
jgroups.dev.auth.example.com.svc.cluster.local. 10 IN SRV 1 1 32917
161f3d66-f1e3-46f4-a44f-ebda925a25c6.jgroups.dev.auth.example.com.svc.cluster.local.
;; Query time: 2 msec
;; SERVER: 10.42.3.2#53(10.42.3.2)
;; WHEN: Fri Sep 21 01:45:44 2018
;; MSG SIZE rcvd: 481
{code}
You can see ports of 32921, 32923, 32915, and 32917. But the containerized JVM is unaware
of this mapping and will refer to itself as <external_ip>:7600 when making a service
discovery request. This is a new address and so the receiving node adds it to the
discovered_hosts variable.
Anyway, it fails to work and results in endless redirects in the front-end and no way to
load the realm login page.
DNS_PING in AWS ECS cannot cluster with dynamic port mappings
-------------------------------------------------------------
Key: JGRP-2300
URL: https://issues.jboss.org/browse/JGRP-2300
Project: JGroups
Issue Type: Bug
Affects Versions: 4.0.16
Environment: AWS ECS Cluster with DNS based service discovery using
jboss/keycloak:latest containers
Reporter: Eric Thompson
Assignee: Sebastian Łaskawiec
Priority: Critical
Fix For: 4.0.16
When running an ECS cluster with jboss/keycloak:latest containers dynamic port mapping of
all ports is required to allow more than one container to run per EC2 instance. Using SRV
based service discovery records will allow each node to find the rest of the nodes, but
when a discovery request is sent the receiving node sees the sender as IP:7600 instead of
the dynamic port. It then sees this as a "new" node and tries to send discovery
requests to it. And somehow it is also getting node IDs and trying to send requests to
those!
See the following log, there are only 4 actual nodes and the each have a different 5
digit port number:
{code}
### Service discovery with dynamic port mapping
2018-10-10 20:17:44,178 DEBUG [org.jgroups.protocols.dns.DNS_PING]
(thread-240,ejb,17081c624290) Performing discovery of the following hosts
[10.42.3.44:7600, 10.42.3.56:32949, 10.42.3.56:32951, 10.42.3.44:32954, c5b479b7b6d5,
10.42.3.44:32952, 10.42.3.56:7600, 17081c624290, 63976b7fae70, 557cbd7891a2]
2018-10-10 20:17:44,178 DEBUG [org.jgroups.protocols.dns.DNS_PING]
(thread-240,ejb,17081c624290) 17081c624290: sending discovery request to 10.42.3.44:7600
2018-10-10 20:17:44,179 DEBUG [org.jgroups.protocols.dns.DNS_PING]
(thread-240,ejb,17081c624290) 17081c624290: sending discovery request to 10.42.3.56:32949
2018-10-10 20:17:44,179 DEBUG [org.jgroups.protocols.dns.DNS_PING]
(thread-240,ejb,17081c624290) 17081c624290: sending discovery request to 10.42.3.56:32951
2018-10-10 20:17:44,180 DEBUG [org.jgroups.protocols.dns.DNS_PING]
(thread-240,ejb,17081c624290) 17081c624290: sending discovery request to 10.42.3.44:32954
2018-10-10 20:17:44,181 DEBUG [org.jgroups.protocols.dns.DNS_PING]
(thread-240,ejb,17081c624290) 17081c624290: sending discovery request to c5b479b7b6d5
2018-10-10 20:17:44,181 DEBUG [org.jgroups.protocols.dns.DNS_PING]
(thread-240,ejb,17081c624290) 17081c624290: sending discovery request to 10.42.3.44:32952
2018-10-10 20:17:44,181 DEBUG [org.jgroups.protocols.dns.DNS_PING]
(thread-240,ejb,17081c624290) 17081c624290: sending discovery request to 10.42.3.56:7600
2018-10-10 20:17:44,182 DEBUG [org.jgroups.protocols.dns.DNS_PING]
(thread-237,ejb,17081c624290) Received discovery from: 17081c624290, IP: 10.42.3.56:7600
2018-10-10 20:17:44,182 DEBUG [org.jgroups.protocols.dns.DNS_PING]
(thread-240,ejb,17081c624290) 17081c624290: sending discovery request to 17081c624290
2018-10-10 20:17:44,182 DEBUG [org.jgroups.protocols.dns.DNS_PING]
(thread-238,ejb,17081c624290) Received discovery from: 17081c624290, IP: 10.42.3.56:7600
2018-10-10 20:17:44,182 DEBUG [org.jgroups.protocols.dns.DNS_PING]
(thread-240,ejb,17081c624290) 17081c624290: sending discovery request to 63976b7fae70
2018-10-10 20:17:44,183 DEBUG [org.jgroups.protocols.dns.DNS_PING]
(thread-240,ejb,17081c624290) 17081c624290: sending discovery request to 557cbd7891a2
2018-10-10 20:17:44,187 WARN [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,17081c624290)
JGRP000032: 17081c624290: no physical address for c5b479b7b6d5, dropping message
{code}
This code seems to be part of the problem in this case:
https://github.com/belaban/JGroups/blob/87d15ec848aa3d482ae792ef152f7e36e...
See that code uses the incoming address and adds it to the discocvered_hosts, but those
addresses are ALWAYS inaccurate in this case.
Because this is what the recipient of the service discovery request sees (ie: all the
ports are the default 7600):
{code}
2018-10-10 20:35:15,229 DEBUG [org.jgroups.protocols.dns.DNS_PING]
(thread-350,ejb,17081c624290) Received discovery from: 63976b7fae70, IP: 10.42.3.44:7600
2018-10-10 20:35:15,231 DEBUG [org.jgroups.protocols.dns.DNS_PING]
(thread-350,ejb,17081c624290) Received discovery from: 63976b7fae70, IP: 10.42.3.44:7600
2018-10-10 20:35:15,232 DEBUG [org.jgroups.protocols.dns.DNS_PING]
(thread-397,ejb,17081c624290) Received discovery from: 63976b7fae70, IP: 10.42.3.44:7600
2018-10-10 20:35:15,233 DEBUG [org.jgroups.protocols.dns.DNS_PING]
(thread-350,ejb,17081c624290) Received discovery from: 63976b7fae70, IP: 10.42.3.44:7600
2018-10-10 20:35:17,234 DEBUG [org.jgroups.protocols.dns.DNS_PING]
(thread-350,ejb,17081c624290) Received discovery from: 63976b7fae70, IP: 10.42.3.44:7600
2018-10-10 20:35:17,236 DEBUG [org.jgroups.protocols.dns.DNS_PING]
(thread-350,ejb,17081c624290) Received discovery from: 63976b7fae70, IP: 10.42.3.44:7600
2018-10-10 20:35:17,238 DEBUG [org.jgroups.protocols.dns.DNS_PING]
(thread-397,ejb,17081c624290) Received discovery from: 63976b7fae70, IP: 10.42.3.44:7600
2018-10-10 20:35:17,238 DEBUG [org.jgroups.protocols.dns.DNS_PING]
(thread-350,ejb,17081c624290) Received discovery from: 63976b7fae70, IP: 10.42.3.44:7600
2018-10-10 20:35:19,239 DEBUG [org.jgroups.protocols.dns.DNS_PING]
(thread-350,ejb,17081c624290) Received discovery from: 63976b7fae70, IP: 10.42.3.44:7600
2018-10-10 20:35:19,240 DEBUG [org.jgroups.protocols.dns.DNS_PING]
(thread-350,ejb,17081c624290) Received discovery from: 63976b7fae70, IP: 10.42.3.44:7600
2018-10-10 20:35:19,242 DEBUG [org.jgroups.protocols.dns.DNS_PING]
(thread-237,ejb,17081c624290) Received discovery from: 63976b7fae70, IP: 10.42.3.44:7600
2018-10-10 20:35:19,243 DEBUG [org.jgroups.protocols.dns.DNS_PING]
(thread-350,ejb,17081c624290) Received discovery from: 63976b7fae70, IP: 10.42.3.44:7600
2018-10-10 20:35:21,246 DEBUG [org.jgroups.protocols.dns.DNS_PING]
(thread-350,ejb,17081c624290) Received discovery from: 63976b7fae70, IP: 10.42.3.44:7600
2018-10-10 20:35:21,247 DEBUG [org.jgroups.protocols.dns.DNS_PING]
(thread-350,ejb,17081c624290) Received discovery from: 63976b7fae70, IP: 10.42.3.44:7600
2018-10-10 20:35:21,253 DEBUG [org.jgroups.protocols.dns.DNS_PING]
(thread-237,ejb,17081c624290) Received discovery from: 63976b7fae70, IP: 10.42.3.44:7600
2018-10-10 20:35:21,253 DEBUG [org.jgroups.protocols.dns.DNS_PING]
(thread-350,ejb,17081c624290) Received discovery from: 63976b7fae70, IP: 10.42.3.44:7600
2018-10-10 20:35:23,247 DEBUG [org.jgroups.protocols.dns.DNS_PING]
(thread-350,ejb,17081c624290) Received discovery from: 63976b7fae70, IP: 10.42.3.44:7600
2018-10-10 20:35:23,249 DEBUG [org.jgroups.protocols.dns.DNS_PING]
(thread-350,ejb,17081c624290) Received discovery from: 63976b7fae70, IP: 10.42.3.44:7600
2018-10-10 20:35:23,251 DEBUG [org.jgroups.protocols.dns.DNS_PING]
(thread-237,ejb,17081c624290) Received discovery from: 63976b7fae70, IP: 10.42.3.44:7600
2018-10-10 20:35:23,251 DEBUG [org.jgroups.protocols.dns.DNS_PING]
(thread-350,ejb,17081c624290) Received discovery from: 63976b7fae70, IP: 10.42.3.44:7600
2018-10-10 20:35:25,252 DEBUG [org.jgroups.protocols.dns.DNS_PING]
(thread-237,ejb,17081c624290) Received discovery from: 63976b7fae70, IP: 10.42.3.44:7600
2018-10-10 20:35:25,253 DEBUG [org.jgroups.protocols.dns.DNS_PING]
(thread-237,ejb,17081c624290) Received discovery from: 63976b7fae70, IP: 10.42.3.44:7600
2018-10-10 20:35:25,255 DEBUG [org.jgroups.protocols.dns.DNS_PING]
(thread-237,ejb,17081c624290) Received discovery from: 63976b7fae70, IP: 10.42.3.44:7600
2018-10-10 20:35:25,256 DEBUG [org.jgroups.protocols.dns.DNS_PING]
(thread-237,ejb,17081c624290) Received discovery from: 63976b7fae70, IP: 10.42.3.44:7600
{code}
In this state the cluster never seems to work properly and the Keycloak interface breaks
in many frustrating ways.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
2825
days inactive
2825
days old
0 comments
1 participants
participants (1)
-
Eric Thompson (Jira)