[JBoss JIRA] (MODCLUSTER-487) Default AdvertiseBindAddress value should not be NULL (UDP Multicast on Linux systems with more NICs)
6:44 a.m.
[
https://issues.jboss.org/browse/MODCLUSTER-487?page=com.atlassian.jira.pl...
]
Bogdan Sikora edited comment on MODCLUSTER-487 at 5/25/16 7:43 AM:
-------------------------------------------------------------------
It's not working
mod_cluster.conf
{noformat}
AdvertiseGroup [ff02::a]:23364
AdvertiseBindAddress [2620:52:0:105f:40b0:50ff:feb4:403]:23364
{noformat}
{noformat}
[hudson@dhcp180-123 sikora]$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 10.16.183.254 0.0.0.0 UG 100 0 0 eth0
ns2.eng.bos.red 10.16.183.254 255.255.255.255 UGH 100 0 0 eth0
10.16.176.0 0.0.0.0 255.255.248.0 U 0 0 0 eth0.1
10.16.176.0 0.0.0.0 255.255.248.0 U 0 0 0 eth0.2
10.16.176.0 0.0.0.0 255.255.248.0 U 100 0 0 eth0
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
{noformat}
{noformat}
[hudson@dhcp180-123 bin]$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen
1000
link/ether 52:56:00:00:00:08 brd ff:ff:ff:ff:ff:ff
inet 10.16.180.123/21 brd 10.16.183.255 scope global dynamic eth0
valid_lft 49446sec preferred_lft 49446sec
inet6 2620:52:0:105f:5056:ff:fe00:8/64 scope global noprefixroute dynamic
valid_lft 2591970sec preferred_lft 604770sec
inet6 fe80::5056:ff:fe00:8/64 scope link
valid_lft forever preferred_lft forever
3: eth0.1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state
UNKNOWN
link/ether 42:b0:50:b4:04:03 brd ff:ff:ff:ff:ff:ff
inet 10.16.180.136/21 brd 10.16.183.255 scope global dynamic eth0.1
valid_lft 73077sec preferred_lft 73077sec
inet6 2620:52:0:105f:40b0:50ff:feb4:403/64 scope global mngtmpaddr dynamic
valid_lft 2591965sec preferred_lft 604765sec
inet6 fe80::40b0:50ff:feb4:403/64 scope link
valid_lft forever preferred_lft forever
4: eth0.2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state
UNKNOWN
link/ether 42:22:b2:07:e9:5c brd ff:ff:ff:ff:ff:ff
inet 10.16.180.59/21 brd 10.16.183.255 scope global dynamic eth0.2
valid_lft 40730sec preferred_lft 40730sec
inet6 2620:52:0:105f:4022:b2ff:fe07:e95c/64 scope global noprefixroute dynamic
valid_lft 2591970sec preferred_lft 604770sec
inet6 fe80::4022:b2ff:fe07:e95c/64 scope link
valid_lft forever preferred_lft forever
5: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
link/ether 02:42:12:f9:c0:b2 brd ff:ff:ff:ff:ff:ff
inet 172.17.42.1/16 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80::42:12ff:fef9:c0b2/64 scope link
valid_lft forever preferred_lft forever
{noformat}
standalone-ha.xml
{noformat}
<interfaces>
<interface name="management">
<inet-address value="2620:52:0:105f:40b0:50ff:feb4:403"/>
</interface>
<interface name="public">
<inet-address value="2620:52:0:105f:40b0:50ff:feb4:403"/>
</interface>
<interface name="private">
<inet-address value="2620:52:0:105f:40b0:50ff:feb4:403"/>
</interface>
<interface name="karel">
<inet-address value="2620:52:0:105f:5056:ff:fe00:8"/>
</interface>
</interfaces>
<socket-binding-group name="standard-sockets"
default-interface="public"
port-offset="${jboss.socket.binding.port-offset:0}">
<socket-binding name="management-http"
interface="management" port="${jboss.management.http.port:9990}"/>
<socket-binding name="management-https"
interface="management"
port="${jboss.management.https.port:9993}"/>
<socket-binding name="ajp"
port="${jboss.ajp.port:8009}"/>
<socket-binding name="http"
port="${jboss.http.port:8080}"/>
<socket-binding name="https"
port="${jboss.https.port:8443}"/>
<socket-binding name="jgroups-mping" interface="private"
port="0" multicast-address="ff02::2"
multicast-port="45700"/>
<socket-binding name="jgroups-tcp" interface="private"
port="7600"/>
<socket-binding name="jgroups-tcp-fd" interface="private"
port="57600"/>
<socket-binding name="jgroups-udp" interface="private"
port="55200" multicast-address="ff02::2"
multicast-port="45688"/>
<socket-binding name="jgroups-udp-fd" interface="private"
port="54200"/>
<socket-binding name="modcluster" interface="public"
port="0" multicast-address="ff02::a"
multicast-port="23364"/>
<socket-binding name="txn-recovery-environment"
port="4712"/>
<socket-binding name="txn-status-manager" port="4713"/>
<outbound-socket-binding name="mail-smtp">
<remote-destination host="localhost" port="25"/>
</outbound-socket-binding>
</socket-binding-group>
{noformat}
was (Author: bsikora):
mod_cluster.conf
{noformat}
AdvertiseGroup [ff02::a]:23364
AdvertiseBindAddress [2620:52:0:105f:40b0:50ff:feb4:403]:23364
{noformat}
{noformat}
[hudson@dhcp180-123 sikora]$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 10.16.183.254 0.0.0.0 UG 100 0 0 eth0
ns2.eng.bos.red 10.16.183.254 255.255.255.255 UGH 100 0 0 eth0
10.16.176.0 0.0.0.0 255.255.248.0 U 0 0 0 eth0.1
10.16.176.0 0.0.0.0 255.255.248.0 U 0 0 0 eth0.2
10.16.176.0 0.0.0.0 255.255.248.0 U 100 0 0 eth0
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
{noformat}
{noformat}
[hudson@dhcp180-123 bin]$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen
1000
link/ether 52:56:00:00:00:08 brd ff:ff:ff:ff:ff:ff
inet 10.16.180.123/21 brd 10.16.183.255 scope global dynamic eth0
valid_lft 49446sec preferred_lft 49446sec
inet6 2620:52:0:105f:5056:ff:fe00:8/64 scope global noprefixroute dynamic
valid_lft 2591970sec preferred_lft 604770sec
inet6 fe80::5056:ff:fe00:8/64 scope link
valid_lft forever preferred_lft forever
3: eth0.1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state
UNKNOWN
link/ether 42:b0:50:b4:04:03 brd ff:ff:ff:ff:ff:ff
inet 10.16.180.136/21 brd 10.16.183.255 scope global dynamic eth0.1
valid_lft 73077sec preferred_lft 73077sec
inet6 2620:52:0:105f:40b0:50ff:feb4:403/64 scope global mngtmpaddr dynamic
valid_lft 2591965sec preferred_lft 604765sec
inet6 fe80::40b0:50ff:feb4:403/64 scope link
valid_lft forever preferred_lft forever
4: eth0.2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state
UNKNOWN
link/ether 42:22:b2:07:e9:5c brd ff:ff:ff:ff:ff:ff
inet 10.16.180.59/21 brd 10.16.183.255 scope global dynamic eth0.2
valid_lft 40730sec preferred_lft 40730sec
inet6 2620:52:0:105f:4022:b2ff:fe07:e95c/64 scope global noprefixroute dynamic
valid_lft 2591970sec preferred_lft 604770sec
inet6 fe80::4022:b2ff:fe07:e95c/64 scope link
valid_lft forever preferred_lft forever
5: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
link/ether 02:42:12:f9:c0:b2 brd ff:ff:ff:ff:ff:ff
inet 172.17.42.1/16 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80::42:12ff:fef9:c0b2/64 scope link
valid_lft forever preferred_lft forever
{noformat}
standalone-ha.xml
{noformat}
<interfaces>
<interface name="management">
<inet-address value="2620:52:0:105f:40b0:50ff:feb4:403"/>
</interface>
<interface name="public">
<inet-address value="2620:52:0:105f:40b0:50ff:feb4:403"/>
</interface>
<interface name="private">
<inet-address value="2620:52:0:105f:40b0:50ff:feb4:403"/>
</interface>
<interface name="karel">
<inet-address value="2620:52:0:105f:5056:ff:fe00:8"/>
</interface>
</interfaces>
<socket-binding-group name="standard-sockets"
default-interface="public"
port-offset="${jboss.socket.binding.port-offset:0}">
<socket-binding name="management-http"
interface="management" port="${jboss.management.http.port:9990}"/>
<socket-binding name="management-https"
interface="management"
port="${jboss.management.https.port:9993}"/>
<socket-binding name="ajp"
port="${jboss.ajp.port:8009}"/>
<socket-binding name="http"
port="${jboss.http.port:8080}"/>
<socket-binding name="https"
port="${jboss.https.port:8443}"/>
<socket-binding name="jgroups-mping" interface="private"
port="0" multicast-address="ff02::2"
multicast-port="45700"/>
<socket-binding name="jgroups-tcp" interface="private"
port="7600"/>
<socket-binding name="jgroups-tcp-fd" interface="private"
port="57600"/>
<socket-binding name="jgroups-udp" interface="private"
port="55200" multicast-address="ff02::2"
multicast-port="45688"/>
<socket-binding name="jgroups-udp-fd" interface="private"
port="54200"/>
<socket-binding name="modcluster" interface="public"
port="0" multicast-address="ff02::a"
multicast-port="23364"/>
<socket-binding name="txn-recovery-environment"
port="4712"/>
<socket-binding name="txn-status-manager" port="4713"/>
<outbound-socket-binding name="mail-smtp">
<remote-destination host="localhost" port="25"/>
</outbound-socket-binding>
</socket-binding-group>
{noformat}
Default AdvertiseBindAddress value should not be NULL (UDP Multicast
on Linux systems with more NICs)
-----------------------------------------------------------------------------------------------------
Key: MODCLUSTER-487
URL: https://issues.jboss.org/browse/MODCLUSTER-487
Project: mod_cluster
Issue Type: Bug
Components: Native (httpd modules)
Affects Versions: 1.2.11.Final, 1.3.1.Final
Environment: Linux, multiple NICs environment
Reporter: Michal Karm Babacek
Assignee: Michal Karm Babacek
Priority: Critical
Attachments: advertise-linux3_x86_64.zip, advertise-windows_x86.zip,
Advertize.class
Credit where it's due: the issue was first spotted by [~rhatlapa].
h3. Problem
It appears that trying to send to all interfaces with {{NULL}} or {{"0.0.0.0"}}
-- the default {{bindaddr}} when no {{AdvertiseBindAddress}} is set -- in the following
statement actually picks the first non-loopback interface and sends to it.
{code}
if ((rv = apr_sockaddr_info_get(&ma_listen_sa, bindaddr,
ma_mgroup_sa->family, bindport,
APR_UNSPEC, pool)) != APR_SUCCESS) {
ap_log_error(APLOG_MARK, APLOG_ERR, rv, s,
"mod_advertise: ma_group_join apr_sockaddr_info_get(%s:%d)
failed", bindaddr, bindport);
{code}
The result is that there is no datagram on other interfaces. Surprisingly, this is not
deterministic though: After dozens or hundreds of messages, eventually one datagram
reaches another interface.
h3. Impact
Picture this simple scenario: There are two interfaces, e.g.
{noformat}
enp1s0 10.16.88.187
enp2s0 172.18.0.1
{noformat}
listed in this exact order with {{ip addr show}}.
One has an EAP 7 (Wildfly 10) instance with mod_cluster bound to {{172.18.0.1}} IP
address, which implies {{enp2s0}} interface.
Furthermore, one has an Apache HTTP Server instance with mod_cluster bound to
{{172.18.0.1}} IP address, i.e. MCMP VirtualHost and main VirtualHost all Listen on this
IP address.
Result: Without advertising, using an explicit {{proxy-list}}, all is well. MCMP works,
requests work, balancing works.
On the other hand, relying on advertisement, it could take EAP 7 (Wildfly 10) *minutes*
to register with the balancer.
The reason is that a vast majority of UDP Multicast datagrams arrives at enp1s0 and EAP 7
(Wildfly 10) doesn't see them.
h3. Reproducer
Lemme demonstrate with a recently refactored
[advertise.c|https://github.com/Karm/mod_cluster/tree/advertise-native-tes...]
utility for sending datagrams and the well known
[Advertize.java|https://raw.githubusercontent.com/modcluster/mod_cluster/m...]
utility for receiving them.
Your your convenience, here are binaries built from the aforementioned sources:
* Advertize java utility: [^Advertize.class]
* advertise native utility (Linux3 x86_64): [^advertise-linux3_x86_64.zip]
* advertise native utility (WIndows x86): [^advertise-windows_x86.zip]
h3. Demonstration on Linux
h4. System
{noformat}
[mbabacek@perf09 ~]$ ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:18:8b:7a:46:04 brd ff:ff:ff:ff:ff:ff
inet 10.16.88.187/21 brd 10.16.95.255 scope global enp1s0
valid_lft forever preferred_lft forever
inet 10.16.93.253/21 brd 10.16.95.255 scope global secondary enp1s0
valid_lft forever preferred_lft forever
3: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:18:8b:7a:46:05 brd ff:ff:ff:ff:ff:ff
inet 172.17.72.254/19 brd 172.17.95.255 scope global enp2s0
valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
link/ether 02:42:07:ab:74:f9 brd ff:ff:ff:ff:ff:ff
inet 172.18.0.1/16 scope global docker0
valid_lft forever preferred_lft forever
{noformat}
h4. Java
{noformat}
[mbabacek@perf09 ~]$ java -version
openjdk version "1.8.0_71"
OpenJDK Runtime Environment (build 1.8.0_71-b15)
OpenJDK 64-Bit Server VM (build 25.71-b15, mixed mode)
{noformat}
h4.Advertise SENT
{noformat}
[mbabacek@perf09 ~]$ date;./advertise -a 224.0.1.102 -p 33364
Mon Mar 21 12:39:51 EDT 2016
UDP Multicast address to send datagrams to. Value: 224.0.1.102
UDP Multicast port. Value: 33364
IP address of the NIC to bound to. Value: NULL
apr_socket_bind on 0.0.0.0:0
apr_mcast_join on 0.0.0.0:0
apr_socket_sendto to 224.0.1.102:33364
{noformat}
h4. Advertize RECEIVED
YES (/)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364
Linux like OS
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 16:39:51 GMT
received from /10.16.88.187:38907
{noformat}
YES (/)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 10.16.88.187
Linux like OS
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 16:39:51 GMT
received from /10.16.88.187:38907
{noformat}
NO (x)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 172.17.72.254
Linux like OS
ready waiting...
{noformat}
YES (/)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 0.0.0.0
Linux like OS
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 16:39:51 GMT
received from /10.16.88.187:38907
{noformat}
And now let's take a look at {{172.17.72.254}}, i.e. {{enp2s0}}
h4. Advertise SENT
{noformat}
[mbabacek@perf09 ~]$ date;./advertise -a 224.0.1.102 -p 33364 -n 172.17.72.254
Mon Mar 21 12:42:57 EDT 2016
UDP Multicast address to send datagrams to. Value: 224.0.1.102
UDP Multicast port. Value: 33364
IP address of the NIC to bound to. Value: 172.17.72.254
apr_socket_bind on 172.17.72.254:0
apr_mcast_join on 172.17.72.254:0
apr_socket_sendto to 224.0.1.102:33364
{noformat}
h4. Advertize RECEIVED
NO (x)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364
Linux like OS
ready waiting...
{noformat}
NO (x)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 10.16.88.187
Linux like OS
ready waiting...
{noformat}
YES (/)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 172.17.72.254
Linux like OS
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 16:42:57 GMT
received from /172.17.72.254:35452
{noformat}
NO (x)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 0.0.0.0
Linux like OS
ready waiting...
{noformat}
h3. Demonstration on Windows
One could note that the problem doesn't exist on Windows. All interfaces receive
advertising.
h4. Advertise SENT
{noformat}
C:\Users\karm\advertise-build
λ advertise.exe -a 224.0.1.102 -p 33364
UDP Multicast address to send datagrams to. Value: 224.0.1.102
UDP Multicast port. Value: 33364
IP address of the NIC to bound to. Value: NULL
apr_socket_bind on 0.0.0.0:0
apr_mcast_join on 0.0.0.0:0
apr_socket_sendto to 224.0.1.102:33364
{noformat}
h4. Advertize RECEIVED
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:07:50 GMT
received from /192.168.122.52:61805
{noformat}
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364
192.168.122.52
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:07:50 GMT
received from /192.168.122.52:61805
{noformat}
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364
192.168.122.199
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:07:50 GMT
received from /192.168.122.52:61805
{noformat}
h4. Advertise SENT
{noformat}
C:\Users\karm\advertise-build
λ advertise.exe -a 224.0.1.102 -p 33364 -n 192.168.122.199
UDP Multicast address to send datagrams to. Value: 224.0.1.102
UDP Multicast port. Value: 33364
IP address of the NIC to bound to. Value: 192.168.122.199
apr_socket_bind on 192.168.122.199:0
apr_mcast_join on 192.168.122.199:0
apr_socket_sendto to 224.0.1.102:33364
{noformat}
h4. Advertize RECEIVED
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:09:55 GMT
received from /192.168.122.199:52781
{noformat}
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364
192.168.122.52
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:09:55 GMT
received from /192.168.122.199:52781
{noformat}
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364
192.168.122.199
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:09:55 GMT
received from /192.168.122.199:52781
{noformat}
h3. Suggestion
Ideas? :) [~jfclere], [~rhusar]
I suggest setting {{bindaddr}} (AdvertiseBindAddress) default to main_server's
address or MCMP enabled vhost instead of NULL. I'll post a PR for evaluation.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
3148
days inactive
3148
days old
mod_cluster-issues@lists.jboss.org
0 comments
1 participants
participants (1)
-
Bogdan Sikora (JIRA)