[JBoss JIRA] (MODCLUSTER-478) Session draining always takes maximum configured timeout since listener is always notified before session is removed
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/MODCLUSTER-478?page=com.atlassian.jira.pl... ]
Radoslav Husar updated MODCLUSTER-478:
--------------------------------------
Status: Resolved (was: Pull Request Sent)
Fix Version/s: 1.2.12.Final
1.3.3.Final
Resolution: Done
Merged 1.3.x and 1.2.x branches now.
> Session draining always takes maximum configured timeout since listener is always notified before session is removed
> --------------------------------------------------------------------------------------------------------------------
>
> Key: MODCLUSTER-478
> URL: https://issues.jboss.org/browse/MODCLUSTER-478
> Project: mod_cluster
> Issue Type: Bug
> Components: Core & Container Integration (Java)
> Affects Versions: 1.2.11.Final, 1.3.1.Final
> Environment: -JBoss EAP 6
> Reporter: Aaron Ogburn
> Assignee: Radoslav Husar
> Fix For: 1.2.12.Final, 1.3.3.Final
>
>
> mod_cluster session draining does not properly end when an active session is invalidated. The NotifyOnDestroySessionListener is not invoked because it is not properly added. That listener is added as a ApplicationLifecycleListeners, but JBossWeb 7 invokes ApplicationSessionLifecycleListeners instead upon session expiration.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 8 months
- 1
- 0
[JBoss JIRA] (MODCLUSTER-487) Default AdvertiseBindAddress value should not be NULL (UDP Multicast on Linux systems with more NICs)
by Michal Karm Babacek (JIRA)
[ https://issues.jboss.org/browse/MODCLUSTER-487?page=com.atlassian.jira.pl... ]
Michal Karm Babacek updated MODCLUSTER-487:
-------------------------------------------
Description:
Credit where it's due: the issue was first spotted by [~rhatlapa].
h3. Problem
It appears that trying to send to all interfaces with {{NULL}} or {{"0.0.0.0"}} -- the default {{bindaddr}} when no {{AdvertiseBindAddress}} is set -- in the following statement actually picks the first non-loopback interface and sends to it.
{code}
if ((rv = apr_sockaddr_info_get(&ma_listen_sa, bindaddr,
ma_mgroup_sa->family, bindport,
APR_UNSPEC, pool)) != APR_SUCCESS) {
ap_log_error(APLOG_MARK, APLOG_ERR, rv, s,
"mod_advertise: ma_group_join apr_sockaddr_info_get(%s:%d) failed", bindaddr, bindport);
{code}
The result is that there is no datagram on other interfaces. Surprisingly, this is not deterministic though: After dozens or hundreds of messages, eventually one datagram reaches another interface.
h3. Impact
Picture this simple scenario: There are two interfaces, e.g.
{noformat}
enp1s0 10.16.88.187
enp2s0 172.18.0.1
{noformat}
listed in this exact order with {{ip addr show}}.
One has an EAP 7 (Wildfly 10) instance with mod_cluster bound to {{172.18.0.1}} IP address, which implies {{enp2s0}} interface.
Furthermore, one has an Apache HTTP Server instance with mod_cluster bound to {{172.18.0.1}} IP address, i.e. MCMP VirtualHost and main VirtualHost all Listen on this IP address.
Result: Without advertising, using an explicit {{proxy-list}}, all is well. MCMP works, requests work, balancing works.
On the other hand, relying on advertisement, it could take EAP 7 (Wildfly 10) *minutes* to register with the balancer.
The reason is that a vast majority of UDP Multicast datagrams arrives at enp1s0 and EAP 7 (Wildfly 10) doesn't see them.
h3. Reproducer
Lemme demonstrate with a recently refactored [advertise.c|https://github.com/Karm/mod_cluster/tree/advertise-native-tes...] utility for sending datagrams and the well known [Advertize.java|https://raw.githubusercontent.com/modcluster/mod_cluster/m...] utility for receiving them.
Your your convenience, here are binaries built from the aforementioned sources:
* Advertize java utility: [^Advertize.class]
* advertise native utility (Linux3 x86_64): [^advertise-linux3_x86_64.zip]
* advertise native utility (WIndows x86): [^advertise-windows_x86.zip]
h3. Demonstration on Linux
h4. System
{noformat}
[mbabacek@perf09 ~]$ ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:18:8b:7a:46:04 brd ff:ff:ff:ff:ff:ff
inet 10.16.88.187/21 brd 10.16.95.255 scope global enp1s0
valid_lft forever preferred_lft forever
inet 10.16.93.253/21 brd 10.16.95.255 scope global secondary enp1s0
valid_lft forever preferred_lft forever
3: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:18:8b:7a:46:05 brd ff:ff:ff:ff:ff:ff
inet 172.17.72.254/19 brd 172.17.95.255 scope global enp2s0
valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
link/ether 02:42:07:ab:74:f9 brd ff:ff:ff:ff:ff:ff
inet 172.18.0.1/16 scope global docker0
valid_lft forever preferred_lft forever
{noformat}
h4. Java
{noformat}
[mbabacek@perf09 ~]$ java -version
openjdk version "1.8.0_71"
OpenJDK Runtime Environment (build 1.8.0_71-b15)
OpenJDK 64-Bit Server VM (build 25.71-b15, mixed mode)
{noformat}
h4.Advertise SENT
{noformat}
[mbabacek@perf09 ~]$ date;./advertise -a 224.0.1.102 -p 33364
Mon Mar 21 12:39:51 EDT 2016
UDP Multicast address to send datagrams to. Value: 224.0.1.102
UDP Multicast port. Value: 33364
IP address of the NIC to bound to. Value: NULL
apr_socket_bind on 0.0.0.0:0
apr_mcast_join on 0.0.0.0:0
apr_socket_sendto to 224.0.1.102:33364
{noformat}
h4. Advertize RECEIVED
YES (/)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364
Linux like OS
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 16:39:51 GMT
received from /10.16.88.187:38907
{noformat}
YES (/)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 10.16.88.187
Linux like OS
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 16:39:51 GMT
received from /10.16.88.187:38907
{noformat}
NO (x)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 172.17.72.254
Linux like OS
ready waiting...
{noformat}
YES (/)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 0.0.0.0
Linux like OS
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 16:39:51 GMT
received from /10.16.88.187:38907
{noformat}
And now let's take a look at {{172.17.72.254}}, i.e. {{enp2s0}}
h4. Advertise SENT
{noformat}
[mbabacek@perf09 ~]$ date;./advertise -a 224.0.1.102 -p 33364 -n 172.17.72.254
Mon Mar 21 12:42:57 EDT 2016
UDP Multicast address to send datagrams to. Value: 224.0.1.102
UDP Multicast port. Value: 33364
IP address of the NIC to bound to. Value: 172.17.72.254
apr_socket_bind on 172.17.72.254:0
apr_mcast_join on 172.17.72.254:0
apr_socket_sendto to 224.0.1.102:33364
{noformat}
h4. Advertize RECEIVED
NO (x)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364
Linux like OS
ready waiting...
{noformat}
NO (x)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 10.16.88.187
Linux like OS
ready waiting...
{noformat}
YES (/)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 172.17.72.254
Linux like OS
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 16:42:57 GMT
received from /172.17.72.254:35452
{noformat}
NO (x)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 0.0.0.0
Linux like OS
ready waiting...
{noformat}
h3. Demonstration on Windows
One could note that the problem doesn't exist on Windows. All interfaces receive advertising.
h4. Advertise SENT
{noformat}
C:\Users\karm\advertise-build
λ advertise.exe -a 224.0.1.102 -p 33364
UDP Multicast address to send datagrams to. Value: 224.0.1.102
UDP Multicast port. Value: 33364
IP address of the NIC to bound to. Value: NULL
apr_socket_bind on 0.0.0.0:0
apr_mcast_join on 0.0.0.0:0
apr_socket_sendto to 224.0.1.102:33364
{noformat}
h4. Advertize RECEIVED
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:07:50 GMT
received from /192.168.122.52:61805
{noformat}
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.52
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:07:50 GMT
received from /192.168.122.52:61805
{noformat}
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.199
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:07:50 GMT
received from /192.168.122.52:61805
{noformat}
h4. Advertise SENT
{noformat}
C:\Users\karm\advertise-build
λ advertise.exe -a 224.0.1.102 -p 33364 -n 192.168.122.199
UDP Multicast address to send datagrams to. Value: 224.0.1.102
UDP Multicast port. Value: 33364
IP address of the NIC to bound to. Value: 192.168.122.199
apr_socket_bind on 192.168.122.199:0
apr_mcast_join on 192.168.122.199:0
apr_socket_sendto to 224.0.1.102:33364
{noformat}
h4. Advertize RECEIVED
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:09:55 GMT
received from /192.168.122.199:52781
{noformat}
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.52
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:09:55 GMT
received from /192.168.122.199:52781
{noformat}
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.199
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:09:55 GMT
received from /192.168.122.199:52781
{noformat}
h3. Suggestion
Ideas? :) [~jfclere], [~rhusar]
I suggest setting {{bindaddr}} (AdvertiseBindAddress) default to main_server's address or MCMP enabled vhost instead of NULL. I'll post a PR for evaluation.
was:
Credit where it's due: the issue was first spotted by [~rhatlapa].
h3. Problem
It appears that trying to send to all interfaces with {{NULL}} or {{"0.0.0.0"}} -- the default of in the following statement actually picks the first non-loopback interface and sends to it.
The result is that there is no datagram on other interfaces. Surprisingly, this is not deterministic though: After dozens or hundreds of messages, eventually one datagram reaches another interface.
h3. Impact
Picture this simple scenario: There are two interfaces, e.g.
{noformat}
enp1s0 10.16.88.187
enp2s0 172.18.0.1
{noformat}
listed in this exact order with {{ip addr show}}.
One has an EAP 7 (Wildfly 10) instance with mod_cluster bound to {{172.18.0.1}} IP address, which implies {{enp2s0}} interface.
Furthermore, one has an Apache HTTP Server instance with mod_cluster bound to {{172.18.0.1}} IP address, i.e. MCMP VirtualHost and main VirtualHost all Listen on this IP address.
Result: Without advertising, using an explicit {{proxy-list}}, all is well. MCMP works, requests work, balancing works.
On the other hand, relying on advertisement, it could take EAP 7 (Wildfly 10) *minutes* to register with the balancer.
The reason is that a vast majority of UDP Multicast datagrams arrives at enp1s0 and EAP 7 (Wildfly 10) doesn't see them.
h3. Reproducer
Lemme demonstrate with a recently refactored [advertise.c|https://github.com/Karm/mod_cluster/tree/advertise-native-tes...] utility for sending datagrams and the well known [Advertize.java|https://raw.githubusercontent.com/modcluster/mod_cluster/m...] utility for receiving them.
Your your convenience, here are binaries built from the aforementioned sources:
* Advertize java utility: [^Advertize.class]
* advertise native utility (Linux3 x86_64): [^advertise-linux3_x86_64.zip]
* advertise native utility (WIndows x86): [^advertise-windows_x86.zip]
h3. Demonstration on Linux
h4. System
{noformat}
[mbabacek@perf09 ~]$ ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:18:8b:7a:46:04 brd ff:ff:ff:ff:ff:ff
inet 10.16.88.187/21 brd 10.16.95.255 scope global enp1s0
valid_lft forever preferred_lft forever
inet 10.16.93.253/21 brd 10.16.95.255 scope global secondary enp1s0
valid_lft forever preferred_lft forever
3: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:18:8b:7a:46:05 brd ff:ff:ff:ff:ff:ff
inet 172.17.72.254/19 brd 172.17.95.255 scope global enp2s0
valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
link/ether 02:42:07:ab:74:f9 brd ff:ff:ff:ff:ff:ff
inet 172.18.0.1/16 scope global docker0
valid_lft forever preferred_lft forever
{noformat}
h4. Java
{noformat}
[mbabacek@perf09 ~]$ java -version
openjdk version "1.8.0_71"
OpenJDK Runtime Environment (build 1.8.0_71-b15)
OpenJDK 64-Bit Server VM (build 25.71-b15, mixed mode)
{noformat}
h4.Advertise SENT
{noformat}
[mbabacek@perf09 ~]$ date;./advertise -a 224.0.1.102 -p 33364
Mon Mar 21 12:39:51 EDT 2016
UDP Multicast address to send datagrams to. Value: 224.0.1.102
UDP Multicast port. Value: 33364
IP address of the NIC to bound to. Value: NULL
apr_socket_bind on 0.0.0.0:0
apr_mcast_join on 0.0.0.0:0
apr_socket_sendto to 224.0.1.102:33364
{noformat}
h4. Advertize RECEIVED
YES (/)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364
Linux like OS
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 16:39:51 GMT
received from /10.16.88.187:38907
{noformat}
YES (/)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 10.16.88.187
Linux like OS
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 16:39:51 GMT
received from /10.16.88.187:38907
{noformat}
NO (x)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 172.17.72.254
Linux like OS
ready waiting...
{noformat}
YES (/)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 0.0.0.0
Linux like OS
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 16:39:51 GMT
received from /10.16.88.187:38907
{noformat}
And now let's take a look at {{172.17.72.254}}, i.e. {{enp2s0}}
h4. Advertise SENT
{noformat}
[mbabacek@perf09 ~]$ date;./advertise -a 224.0.1.102 -p 33364 -n 172.17.72.254
Mon Mar 21 12:42:57 EDT 2016
UDP Multicast address to send datagrams to. Value: 224.0.1.102
UDP Multicast port. Value: 33364
IP address of the NIC to bound to. Value: 172.17.72.254
apr_socket_bind on 172.17.72.254:0
apr_mcast_join on 172.17.72.254:0
apr_socket_sendto to 224.0.1.102:33364
{noformat}
h4. Advertize RECEIVED
NO (x)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364
Linux like OS
ready waiting...
{noformat}
NO (x)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 10.16.88.187
Linux like OS
ready waiting...
{noformat}
YES (/)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 172.17.72.254
Linux like OS
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 16:42:57 GMT
received from /172.17.72.254:35452
{noformat}
NO (x)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 0.0.0.0
Linux like OS
ready waiting...
{noformat}
h3. Demonstration on Windows
One could note that the problem doesn't exist on Windows. All interfaces receive advertising.
h4. Advertise SENT
{noformat}
C:\Users\karm\advertise-build
λ advertise.exe -a 224.0.1.102 -p 33364
UDP Multicast address to send datagrams to. Value: 224.0.1.102
UDP Multicast port. Value: 33364
IP address of the NIC to bound to. Value: NULL
apr_socket_bind on 0.0.0.0:0
apr_mcast_join on 0.0.0.0:0
apr_socket_sendto to 224.0.1.102:33364
{noformat}
h4. Advertize RECEIVED
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:07:50 GMT
received from /192.168.122.52:61805
{noformat}
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.52
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:07:50 GMT
received from /192.168.122.52:61805
{noformat}
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.199
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:07:50 GMT
received from /192.168.122.52:61805
{noformat}
h4. Advertise SENT
{noformat}
C:\Users\karm\advertise-build
λ advertise.exe -a 224.0.1.102 -p 33364 -n 192.168.122.199
UDP Multicast address to send datagrams to. Value: 224.0.1.102
UDP Multicast port. Value: 33364
IP address of the NIC to bound to. Value: 192.168.122.199
apr_socket_bind on 192.168.122.199:0
apr_mcast_join on 192.168.122.199:0
apr_socket_sendto to 224.0.1.102:33364
{noformat}
h4. Advertize RECEIVED
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:09:55 GMT
received from /192.168.122.199:52781
{noformat}
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.52
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:09:55 GMT
received from /192.168.122.199:52781
{noformat}
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.199
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:09:55 GMT
received from /192.168.122.199:52781
{noformat}
h3. Suggestion
Ideas? :) [~jfclere], [~rhusar]
I suggest setting {{bindaddr}} (AdvertiseBindAddress) default to main_server's address or MCMP enabled vhost instead of NULL.
> Default AdvertiseBindAddress value should not be NULL (UDP Multicast on Linux systems with more NICs)
> -----------------------------------------------------------------------------------------------------
>
> Key: MODCLUSTER-487
> URL: https://issues.jboss.org/browse/MODCLUSTER-487
> Project: mod_cluster
> Issue Type: Bug
> Components: Native (httpd modules)
> Affects Versions: 1.2.11.Final, 1.3.1.Final
> Environment: Linux, multiple NICs environment
> Reporter: Michal Karm Babacek
> Assignee: Michal Karm Babacek
> Priority: Critical
> Attachments: advertise-linux3_x86_64.zip, advertise-windows_x86.zip, Advertize.class
>
>
> Credit where it's due: the issue was first spotted by [~rhatlapa].
> h3. Problem
> It appears that trying to send to all interfaces with {{NULL}} or {{"0.0.0.0"}} -- the default {{bindaddr}} when no {{AdvertiseBindAddress}} is set -- in the following statement actually picks the first non-loopback interface and sends to it.
> {code}
> if ((rv = apr_sockaddr_info_get(&ma_listen_sa, bindaddr,
> ma_mgroup_sa->family, bindport,
> APR_UNSPEC, pool)) != APR_SUCCESS) {
> ap_log_error(APLOG_MARK, APLOG_ERR, rv, s,
> "mod_advertise: ma_group_join apr_sockaddr_info_get(%s:%d) failed", bindaddr, bindport);
> {code}
> The result is that there is no datagram on other interfaces. Surprisingly, this is not deterministic though: After dozens or hundreds of messages, eventually one datagram reaches another interface.
> h3. Impact
> Picture this simple scenario: There are two interfaces, e.g.
> {noformat}
> enp1s0 10.16.88.187
> enp2s0 172.18.0.1
> {noformat}
> listed in this exact order with {{ip addr show}}.
> One has an EAP 7 (Wildfly 10) instance with mod_cluster bound to {{172.18.0.1}} IP address, which implies {{enp2s0}} interface.
> Furthermore, one has an Apache HTTP Server instance with mod_cluster bound to {{172.18.0.1}} IP address, i.e. MCMP VirtualHost and main VirtualHost all Listen on this IP address.
> Result: Without advertising, using an explicit {{proxy-list}}, all is well. MCMP works, requests work, balancing works.
> On the other hand, relying on advertisement, it could take EAP 7 (Wildfly 10) *minutes* to register with the balancer.
> The reason is that a vast majority of UDP Multicast datagrams arrives at enp1s0 and EAP 7 (Wildfly 10) doesn't see them.
> h3. Reproducer
> Lemme demonstrate with a recently refactored [advertise.c|https://github.com/Karm/mod_cluster/tree/advertise-native-tes...] utility for sending datagrams and the well known [Advertize.java|https://raw.githubusercontent.com/modcluster/mod_cluster/m...] utility for receiving them.
> Your your convenience, here are binaries built from the aforementioned sources:
> * Advertize java utility: [^Advertize.class]
> * advertise native utility (Linux3 x86_64): [^advertise-linux3_x86_64.zip]
> * advertise native utility (WIndows x86): [^advertise-windows_x86.zip]
> h3. Demonstration on Linux
> h4. System
> {noformat}
> [mbabacek@perf09 ~]$ ip addr show
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> inet 127.0.0.1/8 scope host lo
> valid_lft forever preferred_lft forever
> 2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
> link/ether 00:18:8b:7a:46:04 brd ff:ff:ff:ff:ff:ff
> inet 10.16.88.187/21 brd 10.16.95.255 scope global enp1s0
> valid_lft forever preferred_lft forever
> inet 10.16.93.253/21 brd 10.16.95.255 scope global secondary enp1s0
> valid_lft forever preferred_lft forever
> 3: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
> link/ether 00:18:8b:7a:46:05 brd ff:ff:ff:ff:ff:ff
> inet 172.17.72.254/19 brd 172.17.95.255 scope global enp2s0
> valid_lft forever preferred_lft forever
> 4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
> link/ether 02:42:07:ab:74:f9 brd ff:ff:ff:ff:ff:ff
> inet 172.18.0.1/16 scope global docker0
> valid_lft forever preferred_lft forever
> {noformat}
> h4. Java
> {noformat}
> [mbabacek@perf09 ~]$ java -version
> openjdk version "1.8.0_71"
> OpenJDK Runtime Environment (build 1.8.0_71-b15)
> OpenJDK 64-Bit Server VM (build 25.71-b15, mixed mode)
> {noformat}
> h4.Advertise SENT
> {noformat}
> [mbabacek@perf09 ~]$ date;./advertise -a 224.0.1.102 -p 33364
> Mon Mar 21 12:39:51 EDT 2016
> UDP Multicast address to send datagrams to. Value: 224.0.1.102
> UDP Multicast port. Value: 33364
> IP address of the NIC to bound to. Value: NULL
> apr_socket_bind on 0.0.0.0:0
> apr_mcast_join on 0.0.0.0:0
> apr_socket_sendto to 224.0.1.102:33364
> {noformat}
> h4. Advertize RECEIVED
> YES (/)
> {noformat}
> [mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364
> Linux like OS
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 16:39:51 GMT
> received from /10.16.88.187:38907
> {noformat}
> YES (/)
> {noformat}
> [mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 10.16.88.187
> Linux like OS
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 16:39:51 GMT
> received from /10.16.88.187:38907
> {noformat}
> NO (x)
> {noformat}
> [mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 172.17.72.254
> Linux like OS
> ready waiting...
> {noformat}
> YES (/)
> {noformat}
> [mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 0.0.0.0
> Linux like OS
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 16:39:51 GMT
> received from /10.16.88.187:38907
> {noformat}
> And now let's take a look at {{172.17.72.254}}, i.e. {{enp2s0}}
> h4. Advertise SENT
> {noformat}
> [mbabacek@perf09 ~]$ date;./advertise -a 224.0.1.102 -p 33364 -n 172.17.72.254
> Mon Mar 21 12:42:57 EDT 2016
> UDP Multicast address to send datagrams to. Value: 224.0.1.102
> UDP Multicast port. Value: 33364
> IP address of the NIC to bound to. Value: 172.17.72.254
> apr_socket_bind on 172.17.72.254:0
> apr_mcast_join on 172.17.72.254:0
> apr_socket_sendto to 224.0.1.102:33364
> {noformat}
> h4. Advertize RECEIVED
> NO (x)
> {noformat}
> [mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364
> Linux like OS
> ready waiting...
> {noformat}
> NO (x)
> {noformat}
> [mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 10.16.88.187
> Linux like OS
> ready waiting...
> {noformat}
> YES (/)
> {noformat}
> [mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 172.17.72.254
> Linux like OS
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 16:42:57 GMT
> received from /172.17.72.254:35452
> {noformat}
> NO (x)
> {noformat}
> [mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 0.0.0.0
> Linux like OS
> ready waiting...
> {noformat}
> h3. Demonstration on Windows
> One could note that the problem doesn't exist on Windows. All interfaces receive advertising.
> h4. Advertise SENT
> {noformat}
> C:\Users\karm\advertise-build
> λ advertise.exe -a 224.0.1.102 -p 33364
> UDP Multicast address to send datagrams to. Value: 224.0.1.102
> UDP Multicast port. Value: 33364
> IP address of the NIC to bound to. Value: NULL
> apr_socket_bind on 0.0.0.0:0
> apr_mcast_join on 0.0.0.0:0
> apr_socket_sendto to 224.0.1.102:33364
> {noformat}
> h4. Advertize RECEIVED
> YES (/)
> {noformat}
> C:\Users\karm\WORKSPACE
> λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 18:07:50 GMT
> received from /192.168.122.52:61805
> {noformat}
> YES (/)
> {noformat}
> C:\Users\karm\WORKSPACE
> λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.52
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 18:07:50 GMT
> received from /192.168.122.52:61805
> {noformat}
> YES (/)
> {noformat}
> C:\Users\karm\WORKSPACE
> λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.199
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 18:07:50 GMT
> received from /192.168.122.52:61805
> {noformat}
> h4. Advertise SENT
> {noformat}
> C:\Users\karm\advertise-build
> λ advertise.exe -a 224.0.1.102 -p 33364 -n 192.168.122.199
> UDP Multicast address to send datagrams to. Value: 224.0.1.102
> UDP Multicast port. Value: 33364
> IP address of the NIC to bound to. Value: 192.168.122.199
> apr_socket_bind on 192.168.122.199:0
> apr_mcast_join on 192.168.122.199:0
> apr_socket_sendto to 224.0.1.102:33364
> {noformat}
> h4. Advertize RECEIVED
> YES (/)
> {noformat}
> C:\Users\karm\WORKSPACE
> λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 18:09:55 GMT
> received from /192.168.122.199:52781
> {noformat}
> YES (/)
> {noformat}
> C:\Users\karm\WORKSPACE
> λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.52
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 18:09:55 GMT
> received from /192.168.122.199:52781
> {noformat}
> YES (/)
> {noformat}
> C:\Users\karm\WORKSPACE
> λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.199
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 18:09:55 GMT
> received from /192.168.122.199:52781
> {noformat}
> h3. Suggestion
> Ideas? :) [~jfclere], [~rhusar]
> I suggest setting {{bindaddr}} (AdvertiseBindAddress) default to main_server's address or MCMP enabled vhost instead of NULL. I'll post a PR for evaluation.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 8 months
- 1
- 0
[JBoss JIRA] (MODCLUSTER-487) Default AdvertiseBindAddress value should not be NULL (UDP Multicast on Linux systems with more NICs)
by Michal Karm Babacek (JIRA)
[ https://issues.jboss.org/browse/MODCLUSTER-487?page=com.atlassian.jira.pl... ]
Michal Karm Babacek updated MODCLUSTER-487:
-------------------------------------------
Description:
Credit where it's due: the issue was first spotted by [~rhatlapa].
h3. Problem
It appears that trying to send to all interfaces with {{NULL}} or {{"0.0.0.0"}} -- the default of in the following statement actually picks the first non-loopback interface and sends to it.
The result is that there is no datagram on other interfaces. Surprisingly, this is not deterministic though: After dozens or hundreds of messages, eventually one datagram reaches another interface.
h3. Impact
Picture this simple scenario: There are two interfaces, e.g.
{noformat}
enp1s0 10.16.88.187
enp2s0 172.18.0.1
{noformat}
listed in this exact order with {{ip addr show}}.
One has an EAP 7 (Wildfly 10) instance with mod_cluster bound to {{172.18.0.1}} IP address, which implies {{enp2s0}} interface.
Furthermore, one has an Apache HTTP Server instance with mod_cluster bound to {{172.18.0.1}} IP address, i.e. MCMP VirtualHost and main VirtualHost all Listen on this IP address.
Result: Without advertising, using an explicit {{proxy-list}}, all is well. MCMP works, requests work, balancing works.
On the other hand, relying on advertisement, it could take EAP 7 (Wildfly 10) *minutes* to register with the balancer.
The reason is that a vast majority of UDP Multicast datagrams arrives at enp1s0 and EAP 7 (Wildfly 10) doesn't see them.
h3. Reproducer
Lemme demonstrate with a recently refactored [advertise.c|https://github.com/Karm/mod_cluster/tree/advertise-native-tes...] utility for sending datagrams and the well known [Advertize.java|https://raw.githubusercontent.com/modcluster/mod_cluster/m...] utility for receiving them.
Your your convenience, here are binaries built from the aforementioned sources:
* Advertize java utility: [^Advertize.class]
* advertise native utility (Linux3 x86_64): [^advertise-linux3_x86_64.zip]
* advertise native utility (WIndows x86): [^advertise-windows_x86.zip]
h3. Demonstration on Linux
h4. System
{noformat}
[mbabacek@perf09 ~]$ ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:18:8b:7a:46:04 brd ff:ff:ff:ff:ff:ff
inet 10.16.88.187/21 brd 10.16.95.255 scope global enp1s0
valid_lft forever preferred_lft forever
inet 10.16.93.253/21 brd 10.16.95.255 scope global secondary enp1s0
valid_lft forever preferred_lft forever
3: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:18:8b:7a:46:05 brd ff:ff:ff:ff:ff:ff
inet 172.17.72.254/19 brd 172.17.95.255 scope global enp2s0
valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
link/ether 02:42:07:ab:74:f9 brd ff:ff:ff:ff:ff:ff
inet 172.18.0.1/16 scope global docker0
valid_lft forever preferred_lft forever
{noformat}
h4. Java
{noformat}
[mbabacek@perf09 ~]$ java -version
openjdk version "1.8.0_71"
OpenJDK Runtime Environment (build 1.8.0_71-b15)
OpenJDK 64-Bit Server VM (build 25.71-b15, mixed mode)
{noformat}
h4.Advertise SENT
{noformat}
[mbabacek@perf09 ~]$ date;./advertise -a 224.0.1.102 -p 33364
Mon Mar 21 12:39:51 EDT 2016
UDP Multicast address to send datagrams to. Value: 224.0.1.102
UDP Multicast port. Value: 33364
IP address of the NIC to bound to. Value: NULL
apr_socket_bind on 0.0.0.0:0
apr_mcast_join on 0.0.0.0:0
apr_socket_sendto to 224.0.1.102:33364
{noformat}
h4. Advertize RECEIVED
YES (/)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364
Linux like OS
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 16:39:51 GMT
received from /10.16.88.187:38907
{noformat}
YES (/)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 10.16.88.187
Linux like OS
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 16:39:51 GMT
received from /10.16.88.187:38907
{noformat}
NO (x)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 172.17.72.254
Linux like OS
ready waiting...
{noformat}
YES (/)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 0.0.0.0
Linux like OS
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 16:39:51 GMT
received from /10.16.88.187:38907
{noformat}
And now let's take a look at {{172.17.72.254}}, i.e. {{enp2s0}}
h4. Advertise SENT
{noformat}
[mbabacek@perf09 ~]$ date;./advertise -a 224.0.1.102 -p 33364 -n 172.17.72.254
Mon Mar 21 12:42:57 EDT 2016
UDP Multicast address to send datagrams to. Value: 224.0.1.102
UDP Multicast port. Value: 33364
IP address of the NIC to bound to. Value: 172.17.72.254
apr_socket_bind on 172.17.72.254:0
apr_mcast_join on 172.17.72.254:0
apr_socket_sendto to 224.0.1.102:33364
{noformat}
h4. Advertize RECEIVED
NO (x)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364
Linux like OS
ready waiting...
{noformat}
NO (x)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 10.16.88.187
Linux like OS
ready waiting...
{noformat}
YES (/)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 172.17.72.254
Linux like OS
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 16:42:57 GMT
received from /172.17.72.254:35452
{noformat}
NO (x)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 0.0.0.0
Linux like OS
ready waiting...
{noformat}
h3. Demonstration on Windows
One could note that the problem doesn't exist on Windows. All interfaces receive advertising.
h4. Advertise SENT
{noformat}
C:\Users\karm\advertise-build
λ advertise.exe -a 224.0.1.102 -p 33364
UDP Multicast address to send datagrams to. Value: 224.0.1.102
UDP Multicast port. Value: 33364
IP address of the NIC to bound to. Value: NULL
apr_socket_bind on 0.0.0.0:0
apr_mcast_join on 0.0.0.0:0
apr_socket_sendto to 224.0.1.102:33364
{noformat}
h4. Advertize RECEIVED
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:07:50 GMT
received from /192.168.122.52:61805
{noformat}
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.52
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:07:50 GMT
received from /192.168.122.52:61805
{noformat}
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.199
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:07:50 GMT
received from /192.168.122.52:61805
{noformat}
h4. Advertise SENT
{noformat}
C:\Users\karm\advertise-build
λ advertise.exe -a 224.0.1.102 -p 33364 -n 192.168.122.199
UDP Multicast address to send datagrams to. Value: 224.0.1.102
UDP Multicast port. Value: 33364
IP address of the NIC to bound to. Value: 192.168.122.199
apr_socket_bind on 192.168.122.199:0
apr_mcast_join on 192.168.122.199:0
apr_socket_sendto to 224.0.1.102:33364
{noformat}
h4. Advertize RECEIVED
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:09:55 GMT
received from /192.168.122.199:52781
{noformat}
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.52
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:09:55 GMT
received from /192.168.122.199:52781
{noformat}
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.199
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:09:55 GMT
received from /192.168.122.199:52781
{noformat}
h3. Suggestion
Ideas? :) [~jfclere], [~rhusar]
I suggest setting {{bindaddr}} (AdvertiseBindAddress) default to main_server's address or MCMP enabled vhost instead of NULL.
was:
Credit where it's due: the issue was first spotted by [~rhatlapa].
h3. Problem
It appears that trying to send to all interfaces with {{NULL}} or {{"0.0.0.0"}} in the following statement actually picks the first non-loopback interface and sends to it.
The result is that there is no datagram on other interfaces. Surprisingly, this is not deterministic though: After dozens or hundreds of messages, eventually one datagram reaches another interface.
{code}
if ((rv = apr_sockaddr_info_get(&ma_niface_sa, NULL,
ma_mgroup_sa->family, 0,
APR_UNSPEC, pool)) != APR_SUCCESS) {
ap_log_error(APLOG_MARK, APLOG_ERR, rv, s,
"mod_advertise: ma_group_join apr_sockaddr_info_get(0.0.0.0:0) failed");
return rv;
}
{code}
h3. Impact
Picture this simple scenario: There are two interfaces, e.g.
{noformat}
enp1s0 10.16.88.187
enp2s0 172.18.0.1
{noformat}
listed in this exact order with {{ip addr show}}.
One has an EAP 7 (Wildfly 10) instance with mod_cluster bound to {{172.18.0.1}} IP address, which implies {{enp2s0}} interface.
Furthermore, one has an Apache HTTP Server instance with mod_cluster bound to {{172.18.0.1}} IP address, i.e. MCMP VirtualHost and main VirtualHost all Listen on this IP address.
Result: Without advertising, using an explicit {{proxy-list}}, all is well. MCMP works, requests work, balancing works.
On the other hand, relying on advertisement, it could take EAP 7 (Wildfly 10) *minutes* to register with the balancer.
The reason is that a vast majority of UDP Multicast datagrams arrives at enp1s0 and EAP 7 (Wildfly 10) doesn't see them.
h3. Reproducer
Lemme demonstrate with a recently refactored [advertise.c|https://github.com/Karm/mod_cluster/tree/advertise-native-tes...] utility for sending datagrams and the well known [Advertize.java|https://raw.githubusercontent.com/modcluster/mod_cluster/m...] utility for receiving them.
Your your convenience, here are binaries built from the aforementioned sources:
* Advertize java utility: [^Advertize.class]
* advertise native utility (Linux3 x86_64): [^advertise-linux3_x86_64.zip]
* advertise native utility (WIndows x86): [^advertise-windows_x86.zip]
h3. Demonstration on Linux
h4. System
{noformat}
[mbabacek@perf09 ~]$ ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:18:8b:7a:46:04 brd ff:ff:ff:ff:ff:ff
inet 10.16.88.187/21 brd 10.16.95.255 scope global enp1s0
valid_lft forever preferred_lft forever
inet 10.16.93.253/21 brd 10.16.95.255 scope global secondary enp1s0
valid_lft forever preferred_lft forever
3: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:18:8b:7a:46:05 brd ff:ff:ff:ff:ff:ff
inet 172.17.72.254/19 brd 172.17.95.255 scope global enp2s0
valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
link/ether 02:42:07:ab:74:f9 brd ff:ff:ff:ff:ff:ff
inet 172.18.0.1/16 scope global docker0
valid_lft forever preferred_lft forever
{noformat}
h4. Java
{noformat}
[mbabacek@perf09 ~]$ java -version
openjdk version "1.8.0_71"
OpenJDK Runtime Environment (build 1.8.0_71-b15)
OpenJDK 64-Bit Server VM (build 25.71-b15, mixed mode)
{noformat}
h4.Advertise SENT
{noformat}
[mbabacek@perf09 ~]$ date;./advertise -a 224.0.1.102 -p 33364
Mon Mar 21 12:39:51 EDT 2016
UDP Multicast address to send datagrams to. Value: 224.0.1.102
UDP Multicast port. Value: 33364
IP address of the NIC to bound to. Value: NULL
apr_socket_bind on 0.0.0.0:0
apr_mcast_join on 0.0.0.0:0
apr_socket_sendto to 224.0.1.102:33364
{noformat}
h4. Advertize RECEIVED
YES (/)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364
Linux like OS
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 16:39:51 GMT
received from /10.16.88.187:38907
{noformat}
YES (/)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 10.16.88.187
Linux like OS
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 16:39:51 GMT
received from /10.16.88.187:38907
{noformat}
NO (x)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 172.17.72.254
Linux like OS
ready waiting...
{noformat}
YES (/)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 0.0.0.0
Linux like OS
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 16:39:51 GMT
received from /10.16.88.187:38907
{noformat}
And now let's take a look at {{172.17.72.254}}, i.e. {{enp2s0}}
h4. Advertise SENT
{noformat}
[mbabacek@perf09 ~]$ date;./advertise -a 224.0.1.102 -p 33364 -n 172.17.72.254
Mon Mar 21 12:42:57 EDT 2016
UDP Multicast address to send datagrams to. Value: 224.0.1.102
UDP Multicast port. Value: 33364
IP address of the NIC to bound to. Value: 172.17.72.254
apr_socket_bind on 172.17.72.254:0
apr_mcast_join on 172.17.72.254:0
apr_socket_sendto to 224.0.1.102:33364
{noformat}
h4. Advertize RECEIVED
NO (x)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364
Linux like OS
ready waiting...
{noformat}
NO (x)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 10.16.88.187
Linux like OS
ready waiting...
{noformat}
YES (/)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 172.17.72.254
Linux like OS
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 16:42:57 GMT
received from /172.17.72.254:35452
{noformat}
NO (x)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 0.0.0.0
Linux like OS
ready waiting...
{noformat}
h3. Demonstration on Windows
One could note that the problem doesn't exist on Windows. All interfaces receive advertising.
h4. Advertise SENT
{noformat}
C:\Users\karm\advertise-build
λ advertise.exe -a 224.0.1.102 -p 33364
UDP Multicast address to send datagrams to. Value: 224.0.1.102
UDP Multicast port. Value: 33364
IP address of the NIC to bound to. Value: NULL
apr_socket_bind on 0.0.0.0:0
apr_mcast_join on 0.0.0.0:0
apr_socket_sendto to 224.0.1.102:33364
{noformat}
h4. Advertize RECEIVED
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:07:50 GMT
received from /192.168.122.52:61805
{noformat}
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.52
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:07:50 GMT
received from /192.168.122.52:61805
{noformat}
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.199
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:07:50 GMT
received from /192.168.122.52:61805
{noformat}
h4. Advertise SENT
{noformat}
C:\Users\karm\advertise-build
λ advertise.exe -a 224.0.1.102 -p 33364 -n 192.168.122.199
UDP Multicast address to send datagrams to. Value: 224.0.1.102
UDP Multicast port. Value: 33364
IP address of the NIC to bound to. Value: 192.168.122.199
apr_socket_bind on 192.168.122.199:0
apr_mcast_join on 192.168.122.199:0
apr_socket_sendto to 224.0.1.102:33364
{noformat}
h4. Advertize RECEIVED
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:09:55 GMT
received from /192.168.122.199:52781
{noformat}
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.52
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:09:55 GMT
received from /192.168.122.199:52781
{noformat}
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.199
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:09:55 GMT
received from /192.168.122.199:52781
{noformat}
Ideas? :) [~jfclere], [~rhusar]
I suggest adding {{bindaddr}} from AdvertiseBindAddress instead of NULL or perhaps adding a configuration directive that lets one to pick either MCMP VirtualHost ip address or default -- NULL.
> Default AdvertiseBindAddress value should not be NULL (UDP Multicast on Linux systems with more NICs)
> -----------------------------------------------------------------------------------------------------
>
> Key: MODCLUSTER-487
> URL: https://issues.jboss.org/browse/MODCLUSTER-487
> Project: mod_cluster
> Issue Type: Bug
> Components: Native (httpd modules)
> Affects Versions: 1.2.11.Final, 1.3.1.Final
> Environment: Linux, multiple NICs environment
> Reporter: Michal Karm Babacek
> Assignee: Michal Karm Babacek
> Priority: Critical
> Attachments: advertise-linux3_x86_64.zip, advertise-windows_x86.zip, Advertize.class
>
>
> Credit where it's due: the issue was first spotted by [~rhatlapa].
> h3. Problem
> It appears that trying to send to all interfaces with {{NULL}} or {{"0.0.0.0"}} -- the default of in the following statement actually picks the first non-loopback interface and sends to it.
> The result is that there is no datagram on other interfaces. Surprisingly, this is not deterministic though: After dozens or hundreds of messages, eventually one datagram reaches another interface.
> h3. Impact
> Picture this simple scenario: There are two interfaces, e.g.
> {noformat}
> enp1s0 10.16.88.187
> enp2s0 172.18.0.1
> {noformat}
> listed in this exact order with {{ip addr show}}.
> One has an EAP 7 (Wildfly 10) instance with mod_cluster bound to {{172.18.0.1}} IP address, which implies {{enp2s0}} interface.
> Furthermore, one has an Apache HTTP Server instance with mod_cluster bound to {{172.18.0.1}} IP address, i.e. MCMP VirtualHost and main VirtualHost all Listen on this IP address.
> Result: Without advertising, using an explicit {{proxy-list}}, all is well. MCMP works, requests work, balancing works.
> On the other hand, relying on advertisement, it could take EAP 7 (Wildfly 10) *minutes* to register with the balancer.
> The reason is that a vast majority of UDP Multicast datagrams arrives at enp1s0 and EAP 7 (Wildfly 10) doesn't see them.
> h3. Reproducer
> Lemme demonstrate with a recently refactored [advertise.c|https://github.com/Karm/mod_cluster/tree/advertise-native-tes...] utility for sending datagrams and the well known [Advertize.java|https://raw.githubusercontent.com/modcluster/mod_cluster/m...] utility for receiving them.
> Your your convenience, here are binaries built from the aforementioned sources:
> * Advertize java utility: [^Advertize.class]
> * advertise native utility (Linux3 x86_64): [^advertise-linux3_x86_64.zip]
> * advertise native utility (WIndows x86): [^advertise-windows_x86.zip]
> h3. Demonstration on Linux
> h4. System
> {noformat}
> [mbabacek@perf09 ~]$ ip addr show
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> inet 127.0.0.1/8 scope host lo
> valid_lft forever preferred_lft forever
> 2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
> link/ether 00:18:8b:7a:46:04 brd ff:ff:ff:ff:ff:ff
> inet 10.16.88.187/21 brd 10.16.95.255 scope global enp1s0
> valid_lft forever preferred_lft forever
> inet 10.16.93.253/21 brd 10.16.95.255 scope global secondary enp1s0
> valid_lft forever preferred_lft forever
> 3: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
> link/ether 00:18:8b:7a:46:05 brd ff:ff:ff:ff:ff:ff
> inet 172.17.72.254/19 brd 172.17.95.255 scope global enp2s0
> valid_lft forever preferred_lft forever
> 4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
> link/ether 02:42:07:ab:74:f9 brd ff:ff:ff:ff:ff:ff
> inet 172.18.0.1/16 scope global docker0
> valid_lft forever preferred_lft forever
> {noformat}
> h4. Java
> {noformat}
> [mbabacek@perf09 ~]$ java -version
> openjdk version "1.8.0_71"
> OpenJDK Runtime Environment (build 1.8.0_71-b15)
> OpenJDK 64-Bit Server VM (build 25.71-b15, mixed mode)
> {noformat}
> h4.Advertise SENT
> {noformat}
> [mbabacek@perf09 ~]$ date;./advertise -a 224.0.1.102 -p 33364
> Mon Mar 21 12:39:51 EDT 2016
> UDP Multicast address to send datagrams to. Value: 224.0.1.102
> UDP Multicast port. Value: 33364
> IP address of the NIC to bound to. Value: NULL
> apr_socket_bind on 0.0.0.0:0
> apr_mcast_join on 0.0.0.0:0
> apr_socket_sendto to 224.0.1.102:33364
> {noformat}
> h4. Advertize RECEIVED
> YES (/)
> {noformat}
> [mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364
> Linux like OS
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 16:39:51 GMT
> received from /10.16.88.187:38907
> {noformat}
> YES (/)
> {noformat}
> [mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 10.16.88.187
> Linux like OS
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 16:39:51 GMT
> received from /10.16.88.187:38907
> {noformat}
> NO (x)
> {noformat}
> [mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 172.17.72.254
> Linux like OS
> ready waiting...
> {noformat}
> YES (/)
> {noformat}
> [mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 0.0.0.0
> Linux like OS
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 16:39:51 GMT
> received from /10.16.88.187:38907
> {noformat}
> And now let's take a look at {{172.17.72.254}}, i.e. {{enp2s0}}
> h4. Advertise SENT
> {noformat}
> [mbabacek@perf09 ~]$ date;./advertise -a 224.0.1.102 -p 33364 -n 172.17.72.254
> Mon Mar 21 12:42:57 EDT 2016
> UDP Multicast address to send datagrams to. Value: 224.0.1.102
> UDP Multicast port. Value: 33364
> IP address of the NIC to bound to. Value: 172.17.72.254
> apr_socket_bind on 172.17.72.254:0
> apr_mcast_join on 172.17.72.254:0
> apr_socket_sendto to 224.0.1.102:33364
> {noformat}
> h4. Advertize RECEIVED
> NO (x)
> {noformat}
> [mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364
> Linux like OS
> ready waiting...
> {noformat}
> NO (x)
> {noformat}
> [mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 10.16.88.187
> Linux like OS
> ready waiting...
> {noformat}
> YES (/)
> {noformat}
> [mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 172.17.72.254
> Linux like OS
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 16:42:57 GMT
> received from /172.17.72.254:35452
> {noformat}
> NO (x)
> {noformat}
> [mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 0.0.0.0
> Linux like OS
> ready waiting...
> {noformat}
> h3. Demonstration on Windows
> One could note that the problem doesn't exist on Windows. All interfaces receive advertising.
> h4. Advertise SENT
> {noformat}
> C:\Users\karm\advertise-build
> λ advertise.exe -a 224.0.1.102 -p 33364
> UDP Multicast address to send datagrams to. Value: 224.0.1.102
> UDP Multicast port. Value: 33364
> IP address of the NIC to bound to. Value: NULL
> apr_socket_bind on 0.0.0.0:0
> apr_mcast_join on 0.0.0.0:0
> apr_socket_sendto to 224.0.1.102:33364
> {noformat}
> h4. Advertize RECEIVED
> YES (/)
> {noformat}
> C:\Users\karm\WORKSPACE
> λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 18:07:50 GMT
> received from /192.168.122.52:61805
> {noformat}
> YES (/)
> {noformat}
> C:\Users\karm\WORKSPACE
> λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.52
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 18:07:50 GMT
> received from /192.168.122.52:61805
> {noformat}
> YES (/)
> {noformat}
> C:\Users\karm\WORKSPACE
> λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.199
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 18:07:50 GMT
> received from /192.168.122.52:61805
> {noformat}
> h4. Advertise SENT
> {noformat}
> C:\Users\karm\advertise-build
> λ advertise.exe -a 224.0.1.102 -p 33364 -n 192.168.122.199
> UDP Multicast address to send datagrams to. Value: 224.0.1.102
> UDP Multicast port. Value: 33364
> IP address of the NIC to bound to. Value: 192.168.122.199
> apr_socket_bind on 192.168.122.199:0
> apr_mcast_join on 192.168.122.199:0
> apr_socket_sendto to 224.0.1.102:33364
> {noformat}
> h4. Advertize RECEIVED
> YES (/)
> {noformat}
> C:\Users\karm\WORKSPACE
> λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 18:09:55 GMT
> received from /192.168.122.199:52781
> {noformat}
> YES (/)
> {noformat}
> C:\Users\karm\WORKSPACE
> λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.52
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 18:09:55 GMT
> received from /192.168.122.199:52781
> {noformat}
> YES (/)
> {noformat}
> C:\Users\karm\WORKSPACE
> λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.199
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 18:09:55 GMT
> received from /192.168.122.199:52781
> {noformat}
> h3. Suggestion
> Ideas? :) [~jfclere], [~rhusar]
> I suggest setting {{bindaddr}} (AdvertiseBindAddress) default to main_server's address or MCMP enabled vhost instead of NULL.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 8 months
- 1
- 0
[JBoss JIRA] (MODCLUSTER-487) Default AdvertiseBindAddress value should not be NULL (UDP Multicast on Linux systems with more NICs)
by Michal Karm Babacek (JIRA)
[ https://issues.jboss.org/browse/MODCLUSTER-487?page=com.atlassian.jira.pl... ]
Michal Karm Babacek updated MODCLUSTER-487:
-------------------------------------------
Summary: Default AdvertiseBindAddress value should not be NULL (UDP Multicast on Linux systems with more NICs) (was: Advertise: UDP Multicast on Linux systems with more NICs: datagrams could be delayed or lost)
> Default AdvertiseBindAddress value should not be NULL (UDP Multicast on Linux systems with more NICs)
> -----------------------------------------------------------------------------------------------------
>
> Key: MODCLUSTER-487
> URL: https://issues.jboss.org/browse/MODCLUSTER-487
> Project: mod_cluster
> Issue Type: Bug
> Components: Native (httpd modules)
> Affects Versions: 1.2.11.Final, 1.3.1.Final
> Environment: Linux, multiple NICs environment
> Reporter: Michal Karm Babacek
> Assignee: Michal Karm Babacek
> Priority: Critical
> Attachments: advertise-linux3_x86_64.zip, advertise-windows_x86.zip, Advertize.class
>
>
> Credit where it's due: the issue was first spotted by [~rhatlapa].
> h3. Problem
> It appears that trying to send to all interfaces with {{NULL}} or {{"0.0.0.0"}} in the following statement actually picks the first non-loopback interface and sends to it.
> The result is that there is no datagram on other interfaces. Surprisingly, this is not deterministic though: After dozens or hundreds of messages, eventually one datagram reaches another interface.
> {code}
> if ((rv = apr_sockaddr_info_get(&ma_niface_sa, NULL,
> ma_mgroup_sa->family, 0,
> APR_UNSPEC, pool)) != APR_SUCCESS) {
> ap_log_error(APLOG_MARK, APLOG_ERR, rv, s,
> "mod_advertise: ma_group_join apr_sockaddr_info_get(0.0.0.0:0) failed");
> return rv;
> }
> {code}
> h3. Impact
> Picture this simple scenario: There are two interfaces, e.g.
> {noformat}
> enp1s0 10.16.88.187
> enp2s0 172.18.0.1
> {noformat}
> listed in this exact order with {{ip addr show}}.
> One has an EAP 7 (Wildfly 10) instance with mod_cluster bound to {{172.18.0.1}} IP address, which implies {{enp2s0}} interface.
> Furthermore, one has an Apache HTTP Server instance with mod_cluster bound to {{172.18.0.1}} IP address, i.e. MCMP VirtualHost and main VirtualHost all Listen on this IP address.
> Result: Without advertising, using an explicit {{proxy-list}}, all is well. MCMP works, requests work, balancing works.
> On the other hand, relying on advertisement, it could take EAP 7 (Wildfly 10) *minutes* to register with the balancer.
> The reason is that a vast majority of UDP Multicast datagrams arrives at enp1s0 and EAP 7 (Wildfly 10) doesn't see them.
> h3. Reproducer
> Lemme demonstrate with a recently refactored [advertise.c|https://github.com/Karm/mod_cluster/tree/advertise-native-tes...] utility for sending datagrams and the well known [Advertize.java|https://raw.githubusercontent.com/modcluster/mod_cluster/m...] utility for receiving them.
> Your your convenience, here are binaries built from the aforementioned sources:
> * Advertize java utility: [^Advertize.class]
> * advertise native utility (Linux3 x86_64): [^advertise-linux3_x86_64.zip]
> * advertise native utility (WIndows x86): [^advertise-windows_x86.zip]
> h3. Demonstration on Linux
> h4. System
> {noformat}
> [mbabacek@perf09 ~]$ ip addr show
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> inet 127.0.0.1/8 scope host lo
> valid_lft forever preferred_lft forever
> 2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
> link/ether 00:18:8b:7a:46:04 brd ff:ff:ff:ff:ff:ff
> inet 10.16.88.187/21 brd 10.16.95.255 scope global enp1s0
> valid_lft forever preferred_lft forever
> inet 10.16.93.253/21 brd 10.16.95.255 scope global secondary enp1s0
> valid_lft forever preferred_lft forever
> 3: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
> link/ether 00:18:8b:7a:46:05 brd ff:ff:ff:ff:ff:ff
> inet 172.17.72.254/19 brd 172.17.95.255 scope global enp2s0
> valid_lft forever preferred_lft forever
> 4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
> link/ether 02:42:07:ab:74:f9 brd ff:ff:ff:ff:ff:ff
> inet 172.18.0.1/16 scope global docker0
> valid_lft forever preferred_lft forever
> {noformat}
> h4. Java
> {noformat}
> [mbabacek@perf09 ~]$ java -version
> openjdk version "1.8.0_71"
> OpenJDK Runtime Environment (build 1.8.0_71-b15)
> OpenJDK 64-Bit Server VM (build 25.71-b15, mixed mode)
> {noformat}
> h4.Advertise SENT
> {noformat}
> [mbabacek@perf09 ~]$ date;./advertise -a 224.0.1.102 -p 33364
> Mon Mar 21 12:39:51 EDT 2016
> UDP Multicast address to send datagrams to. Value: 224.0.1.102
> UDP Multicast port. Value: 33364
> IP address of the NIC to bound to. Value: NULL
> apr_socket_bind on 0.0.0.0:0
> apr_mcast_join on 0.0.0.0:0
> apr_socket_sendto to 224.0.1.102:33364
> {noformat}
> h4. Advertize RECEIVED
> YES (/)
> {noformat}
> [mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364
> Linux like OS
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 16:39:51 GMT
> received from /10.16.88.187:38907
> {noformat}
> YES (/)
> {noformat}
> [mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 10.16.88.187
> Linux like OS
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 16:39:51 GMT
> received from /10.16.88.187:38907
> {noformat}
> NO (x)
> {noformat}
> [mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 172.17.72.254
> Linux like OS
> ready waiting...
> {noformat}
> YES (/)
> {noformat}
> [mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 0.0.0.0
> Linux like OS
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 16:39:51 GMT
> received from /10.16.88.187:38907
> {noformat}
> And now let's take a look at {{172.17.72.254}}, i.e. {{enp2s0}}
> h4. Advertise SENT
> {noformat}
> [mbabacek@perf09 ~]$ date;./advertise -a 224.0.1.102 -p 33364 -n 172.17.72.254
> Mon Mar 21 12:42:57 EDT 2016
> UDP Multicast address to send datagrams to. Value: 224.0.1.102
> UDP Multicast port. Value: 33364
> IP address of the NIC to bound to. Value: 172.17.72.254
> apr_socket_bind on 172.17.72.254:0
> apr_mcast_join on 172.17.72.254:0
> apr_socket_sendto to 224.0.1.102:33364
> {noformat}
> h4. Advertize RECEIVED
> NO (x)
> {noformat}
> [mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364
> Linux like OS
> ready waiting...
> {noformat}
> NO (x)
> {noformat}
> [mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 10.16.88.187
> Linux like OS
> ready waiting...
> {noformat}
> YES (/)
> {noformat}
> [mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 172.17.72.254
> Linux like OS
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 16:42:57 GMT
> received from /172.17.72.254:35452
> {noformat}
> NO (x)
> {noformat}
> [mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 0.0.0.0
> Linux like OS
> ready waiting...
> {noformat}
> h3. Demonstration on Windows
> One could note that the problem doesn't exist on Windows. All interfaces receive advertising.
> h4. Advertise SENT
> {noformat}
> C:\Users\karm\advertise-build
> λ advertise.exe -a 224.0.1.102 -p 33364
> UDP Multicast address to send datagrams to. Value: 224.0.1.102
> UDP Multicast port. Value: 33364
> IP address of the NIC to bound to. Value: NULL
> apr_socket_bind on 0.0.0.0:0
> apr_mcast_join on 0.0.0.0:0
> apr_socket_sendto to 224.0.1.102:33364
> {noformat}
> h4. Advertize RECEIVED
> YES (/)
> {noformat}
> C:\Users\karm\WORKSPACE
> λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 18:07:50 GMT
> received from /192.168.122.52:61805
> {noformat}
> YES (/)
> {noformat}
> C:\Users\karm\WORKSPACE
> λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.52
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 18:07:50 GMT
> received from /192.168.122.52:61805
> {noformat}
> YES (/)
> {noformat}
> C:\Users\karm\WORKSPACE
> λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.199
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 18:07:50 GMT
> received from /192.168.122.52:61805
> {noformat}
> h4. Advertise SENT
> {noformat}
> C:\Users\karm\advertise-build
> λ advertise.exe -a 224.0.1.102 -p 33364 -n 192.168.122.199
> UDP Multicast address to send datagrams to. Value: 224.0.1.102
> UDP Multicast port. Value: 33364
> IP address of the NIC to bound to. Value: 192.168.122.199
> apr_socket_bind on 192.168.122.199:0
> apr_mcast_join on 192.168.122.199:0
> apr_socket_sendto to 224.0.1.102:33364
> {noformat}
> h4. Advertize RECEIVED
> YES (/)
> {noformat}
> C:\Users\karm\WORKSPACE
> λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 18:09:55 GMT
> received from /192.168.122.199:52781
> {noformat}
> YES (/)
> {noformat}
> C:\Users\karm\WORKSPACE
> λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.52
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 18:09:55 GMT
> received from /192.168.122.199:52781
> {noformat}
> YES (/)
> {noformat}
> C:\Users\karm\WORKSPACE
> λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.199
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 18:09:55 GMT
> received from /192.168.122.199:52781
> {noformat}
> Ideas? :) [~jfclere], [~rhusar]
> I suggest adding {{bindaddr}} from AdvertiseBindAddress instead of NULL or perhaps adding a configuration directive that lets one to pick either MCMP VirtualHost ip address or default -- NULL.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 8 months
- 1
- 0
[JBoss JIRA] (MODCLUSTER-478) Session draining always takes maximum configured timeout since listener is always notified before session is removed
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/MODCLUSTER-478?page=com.atlassian.jira.pl... ]
RH Bugzilla Integration commented on MODCLUSTER-478:
----------------------------------------------------
Mike McCune <mmccune(a)redhat.com> changed the Status of [bug 1314792|https://bugzilla.redhat.com/show_bug.cgi?id=1314792] from MODIFIED to POST
> Session draining always takes maximum configured timeout since listener is always notified before session is removed
> --------------------------------------------------------------------------------------------------------------------
>
> Key: MODCLUSTER-478
> URL: https://issues.jboss.org/browse/MODCLUSTER-478
> Project: mod_cluster
> Issue Type: Bug
> Components: Core & Container Integration (Java)
> Affects Versions: 1.2.11.Final, 1.3.1.Final
> Environment: -JBoss EAP 6
> Reporter: Aaron Ogburn
> Assignee: Radoslav Husar
>
> mod_cluster session draining does not properly end when an active session is invalidated. The NotifyOnDestroySessionListener is not invoked because it is not properly added. That listener is added as a ApplicationLifecycleListeners, but JBossWeb 7 invokes ApplicationSessionLifecycleListeners instead upon session expiration.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 8 months
- 1
- 0
[JBoss JIRA] (MODCLUSTER-478) Session draining always takes maximum configured timeout since listener is always notified before session is removed
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/MODCLUSTER-478?page=com.atlassian.jira.pl... ]
RH Bugzilla Integration commented on MODCLUSTER-478:
----------------------------------------------------
Sat6QE Jenkins <sat6-jenkins(a)redhat.com> changed the Status of [bug 1314792|https://bugzilla.redhat.com/show_bug.cgi?id=1314792] from POST to MODIFIED
> Session draining always takes maximum configured timeout since listener is always notified before session is removed
> --------------------------------------------------------------------------------------------------------------------
>
> Key: MODCLUSTER-478
> URL: https://issues.jboss.org/browse/MODCLUSTER-478
> Project: mod_cluster
> Issue Type: Bug
> Components: Core & Container Integration (Java)
> Affects Versions: 1.2.11.Final, 1.3.1.Final
> Environment: -JBoss EAP 6
> Reporter: Aaron Ogburn
> Assignee: Radoslav Husar
>
> mod_cluster session draining does not properly end when an active session is invalidated. The NotifyOnDestroySessionListener is not invoked because it is not properly added. That listener is added as a ApplicationLifecycleListeners, but JBossWeb 7 invokes ApplicationSessionLifecycleListeners instead upon session expiration.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 8 months
- 1
- 0
[JBoss JIRA] (MODCLUSTER-478) Session draining always takes maximum configured timeout since listener is always notified before session is removed
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/MODCLUSTER-478?page=com.atlassian.jira.pl... ]
RH Bugzilla Integration commented on MODCLUSTER-478:
----------------------------------------------------
Brad Maxwell <bmaxwell(a)redhat.com> changed the Status of [bug 1314792|https://bugzilla.redhat.com/show_bug.cgi?id=1314792] from NEW to POST
> Session draining always takes maximum configured timeout since listener is always notified before session is removed
> --------------------------------------------------------------------------------------------------------------------
>
> Key: MODCLUSTER-478
> URL: https://issues.jboss.org/browse/MODCLUSTER-478
> Project: mod_cluster
> Issue Type: Bug
> Components: Core & Container Integration (Java)
> Affects Versions: 1.2.11.Final, 1.3.1.Final
> Environment: -JBoss EAP 6
> Reporter: Aaron Ogburn
> Assignee: Radoslav Husar
>
> mod_cluster session draining does not properly end when an active session is invalidated. The NotifyOnDestroySessionListener is not invoked because it is not properly added. That listener is added as a ApplicationLifecycleListeners, but JBossWeb 7 invokes ApplicationSessionLifecycleListeners instead upon session expiration.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 8 months
- 1
- 0
[JBoss JIRA] (MODCLUSTER-478) Session draining always takes maximum configured timeout since listener is always notified before session is removed
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/MODCLUSTER-478?page=com.atlassian.jira.pl... ]
RH Bugzilla Integration updated MODCLUSTER-478:
-----------------------------------------------
Bugzilla References: https://bugzilla.redhat.com/show_bug.cgi?id=1314792, https://bugzilla.redhat.com/show_bug.cgi?id=1321090 (was: https://bugzilla.redhat.com/show_bug.cgi?id=1314792)
> Session draining always takes maximum configured timeout since listener is always notified before session is removed
> --------------------------------------------------------------------------------------------------------------------
>
> Key: MODCLUSTER-478
> URL: https://issues.jboss.org/browse/MODCLUSTER-478
> Project: mod_cluster
> Issue Type: Bug
> Components: Core & Container Integration (Java)
> Affects Versions: 1.2.11.Final, 1.3.1.Final
> Environment: -JBoss EAP 6
> Reporter: Aaron Ogburn
> Assignee: Radoslav Husar
>
> mod_cluster session draining does not properly end when an active session is invalidated. The NotifyOnDestroySessionListener is not invoked because it is not properly added. That listener is added as a ApplicationLifecycleListeners, but JBossWeb 7 invokes ApplicationSessionLifecycleListeners instead upon session expiration.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 8 months
- 1
- 0
[JBoss JIRA] (MODCLUSTER-478) Session draining always takes maximum configured timeout since listener is always notified before session is removed
by Aaron Ogburn (JIRA)
[ https://issues.jboss.org/browse/MODCLUSTER-478?page=com.atlassian.jira.pl... ]
Aaron Ogburn updated MODCLUSTER-478:
------------------------------------
Status: Pull Request Sent (was: Pull Request Sent)
Git Pull Request: https://github.com/modcluster/mod_cluster/pull/171, https://github.com/modcluster/mod_cluster/pull/170, https://github.com/modcluster/mod_cluster/pull/166 (was: https://github.com/modcluster/mod_cluster/pull/170, https://github.com/modcluster/mod_cluster/pull/166)
Thanks, [~rhusar]! Here's a 1.2 PR too of the fix in case it's needed for EAP 6.
> Session draining always takes maximum configured timeout since listener is always notified before session is removed
> --------------------------------------------------------------------------------------------------------------------
>
> Key: MODCLUSTER-478
> URL: https://issues.jboss.org/browse/MODCLUSTER-478
> Project: mod_cluster
> Issue Type: Bug
> Components: Core & Container Integration (Java)
> Affects Versions: 1.2.11.Final, 1.3.1.Final
> Environment: -JBoss EAP 6
> Reporter: Aaron Ogburn
> Assignee: Radoslav Husar
>
> mod_cluster session draining does not properly end when an active session is invalidated. The NotifyOnDestroySessionListener is not invoked because it is not properly added. That listener is added as a ApplicationLifecycleListeners, but JBossWeb 7 invokes ApplicationSessionLifecycleListeners instead upon session expiration.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 8 months
- 1
- 0
[JBoss JIRA] (MODCLUSTER-487) Advertise: UDP Multicast on Linux systems with more NICs: datagrams could be delayed or lost
by Michal Karm Babacek (JIRA)
[ https://issues.jboss.org/browse/MODCLUSTER-487?page=com.atlassian.jira.pl... ]
Michal Karm Babacek updated MODCLUSTER-487:
-------------------------------------------
Description:
Credit where it's due: the issue was first spotted by [~rhatlapa].
h3. Problem
It appears that trying to send to all interfaces with {{NULL}} or {{"0.0.0.0"}} in the following statement actually picks the first non-loopback interface and sends to it.
The result is that there is no datagram on other interfaces. Surprisingly, this is not deterministic though: After dozens or hundreds of messages, eventually one datagram reaches another interface.
{code}
if ((rv = apr_sockaddr_info_get(&ma_niface_sa, NULL,
ma_mgroup_sa->family, 0,
APR_UNSPEC, pool)) != APR_SUCCESS) {
ap_log_error(APLOG_MARK, APLOG_ERR, rv, s,
"mod_advertise: ma_group_join apr_sockaddr_info_get(0.0.0.0:0) failed");
return rv;
}
{code}
h3. Impact
Picture this simple scenario: There are two interfaces, e.g.
{noformat}
enp1s0 10.16.88.187
enp2s0 172.18.0.1
{noformat}
listed in this exact order with {{ip addr show}}.
One has an EAP 7 (Wildfly 10) instance with mod_cluster bound to {{172.18.0.1}} IP address, which implies {{enp2s0}} interface.
Furthermore, one has an Apache HTTP Server instance with mod_cluster bound to {{172.18.0.1}} IP address, i.e. MCMP VirtualHost and main VirtualHost all Listen on this IP address.
Result: Without advertising, using an explicit {{proxy-list}}, all is well. MCMP works, requests work, balancing works.
On the other hand, relying on advertisement, it could take EAP 7 (Wildfly 10) *minutes* to register with the balancer.
The reason is that a vast majority of UDP Multicast datagrams arrives at enp1s0 and EAP 7 (Wildfly 10) doesn't see them.
h3. Reproducer
Lemme demonstrate with a recently refactored [advertise.c|https://github.com/Karm/mod_cluster/tree/advertise-native-tes...] utility for sending datagrams and the well known [Advertize.java|https://raw.githubusercontent.com/modcluster/mod_cluster/m...] utility for receiving them.
Your your convenience, here are binaries built from the aforementioned sources:
* Advertize java utility: [^Advertize.class]
* advertise native utility (Linux3 x86_64): [^advertise-linux3_x86_64.zip]
* advertise native utility (WIndows x86): [^advertise-windows_x86.zip]
h3. Demonstration on Linux
h4. System
{noformat}
[mbabacek@perf09 ~]$ ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:18:8b:7a:46:04 brd ff:ff:ff:ff:ff:ff
inet 10.16.88.187/21 brd 10.16.95.255 scope global enp1s0
valid_lft forever preferred_lft forever
inet 10.16.93.253/21 brd 10.16.95.255 scope global secondary enp1s0
valid_lft forever preferred_lft forever
3: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:18:8b:7a:46:05 brd ff:ff:ff:ff:ff:ff
inet 172.17.72.254/19 brd 172.17.95.255 scope global enp2s0
valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
link/ether 02:42:07:ab:74:f9 brd ff:ff:ff:ff:ff:ff
inet 172.18.0.1/16 scope global docker0
valid_lft forever preferred_lft forever
{noformat}
h4. Java
{noformat}
[mbabacek@perf09 ~]$ java -version
openjdk version "1.8.0_71"
OpenJDK Runtime Environment (build 1.8.0_71-b15)
OpenJDK 64-Bit Server VM (build 25.71-b15, mixed mode)
{noformat}
h4.Advertise SENT
{noformat}
[mbabacek@perf09 ~]$ date;./advertise -a 224.0.1.102 -p 33364
Mon Mar 21 12:39:51 EDT 2016
UDP Multicast address to send datagrams to. Value: 224.0.1.102
UDP Multicast port. Value: 33364
IP address of the NIC to bound to. Value: NULL
apr_socket_bind on 0.0.0.0:0
apr_mcast_join on 0.0.0.0:0
apr_socket_sendto to 224.0.1.102:33364
{noformat}
h4. Advertize RECEIVED
YES (/)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364
Linux like OS
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 16:39:51 GMT
received from /10.16.88.187:38907
{noformat}
YES (/)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 10.16.88.187
Linux like OS
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 16:39:51 GMT
received from /10.16.88.187:38907
{noformat}
NO (x)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 172.17.72.254
Linux like OS
ready waiting...
{noformat}
YES (/)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 0.0.0.0
Linux like OS
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 16:39:51 GMT
received from /10.16.88.187:38907
{noformat}
And now let's take a look at {{172.17.72.254}}, i.e. {{enp2s0}}
h4. Advertise SENT
{noformat}
[mbabacek@perf09 ~]$ date;./advertise -a 224.0.1.102 -p 33364 -n 172.17.72.254
Mon Mar 21 12:42:57 EDT 2016
UDP Multicast address to send datagrams to. Value: 224.0.1.102
UDP Multicast port. Value: 33364
IP address of the NIC to bound to. Value: 172.17.72.254
apr_socket_bind on 172.17.72.254:0
apr_mcast_join on 172.17.72.254:0
apr_socket_sendto to 224.0.1.102:33364
{noformat}
h4. Advertize RECEIVED
NO (x)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364
Linux like OS
ready waiting...
{noformat}
NO (x)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 10.16.88.187
Linux like OS
ready waiting...
{noformat}
YES (/)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 172.17.72.254
Linux like OS
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 16:42:57 GMT
received from /172.17.72.254:35452
{noformat}
NO (x)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 0.0.0.0
Linux like OS
ready waiting...
{noformat}
h3. Demonstration on Windows
One could note that the problem doesn't exist on Windows. All interfaces receive advertising.
h4. Advertise SENT
{noformat}
C:\Users\karm\advertise-build
λ advertise.exe -a 224.0.1.102 -p 33364
UDP Multicast address to send datagrams to. Value: 224.0.1.102
UDP Multicast port. Value: 33364
IP address of the NIC to bound to. Value: NULL
apr_socket_bind on 0.0.0.0:0
apr_mcast_join on 0.0.0.0:0
apr_socket_sendto to 224.0.1.102:33364
{noformat}
h4. Advertize RECEIVED
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:07:50 GMT
received from /192.168.122.52:61805
{noformat}
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.52
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:07:50 GMT
received from /192.168.122.52:61805
{noformat}
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.199
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:07:50 GMT
received from /192.168.122.52:61805
{noformat}
h4. Advertise SENT
{noformat}
C:\Users\karm\advertise-build
λ advertise.exe -a 224.0.1.102 -p 33364 -n 192.168.122.199
UDP Multicast address to send datagrams to. Value: 224.0.1.102
UDP Multicast port. Value: 33364
IP address of the NIC to bound to. Value: 192.168.122.199
apr_socket_bind on 192.168.122.199:0
apr_mcast_join on 192.168.122.199:0
apr_socket_sendto to 224.0.1.102:33364
{noformat}
h4. Advertize RECEIVED
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:09:55 GMT
received from /192.168.122.199:52781
{noformat}
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.52
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:09:55 GMT
received from /192.168.122.199:52781
{noformat}
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.199
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:09:55 GMT
received from /192.168.122.199:52781
{noformat}
Ideas? :) [~jfclere], [~rhusar]
I suggest adding {{bindaddr}} from AdvertiseBindAddress instead of NULL or perhaps adding a configuration directive that lets one to pick either MCMP VirtualHost ip address or default -- NULL.
was:
Credit where it's due: the issue was first spotted by [~rhatlapa].
h3. Problem
It appears that trying to send to all interfaces with {{NULL}} or {{"0.0.0.0"}} in the following statement actually picks the first non-loopback interface and sends to it.
The result is that there is no datagram on other interfaces. Surprisingly, this is not deterministic though: After dozens or hundreds of messages, eventually one datagram reaches another interface.
{code}
if ((rv = apr_sockaddr_info_get(&ma_niface_sa, NULL,
ma_mgroup_sa->family, 0,
APR_UNSPEC, pool)) != APR_SUCCESS) {
ap_log_error(APLOG_MARK, APLOG_ERR, rv, s,
"mod_advertise: ma_group_join apr_sockaddr_info_get(0.0.0.0:0) failed");
return rv;
}
{code}
h3. Impact
Picture this simple scenario: There are two interfaces, e.g.
{noformat}
enp1s0 10.16.88.187
enp2s0 172.18.0.1
{noformat}
listed in this exact order with {{ip addr show}}.
One has an EAP 7 (Wildfly 10) instance with mod_cluster bound to {{172.18.0.1}} IP address, which implies {{enp2s0}} interface.
Furthermore, one has an Apache HTTP Server instance with mod_cluster bound to {{172.18.0.1}} IP address, i.e. MCMP VirtualHost and main VirtualHost all Listen on this IP address.
Result: Without advertising, using an explicit {{proxy-list}}, all is well. MCMP works, requests work, balancing works.
On the other hand, relying on advertisement, it could take EAP 7 (Wildfly 10) *minutes* to register with the balancer.
The reason is that a vast majority of UDP Multicast datagrams arrives at enp1s0 and EAP 7 (Wildfly 10) doesn't see them.
h3. Reproducer
Lemme demonstrate with a recently refactored [advertise.c|https://github.com/Karm/mod_cluster/tree/advertise-native-tes...] utility for sending datagrams and the well known [Advertize.java|https://raw.githubusercontent.com/modcluster/mod_cluster/m...] utility for receiving them.
Your your convenience, here are binaries built from the aforementioned sources:
* Advertize java utility: [^Advertize.class]
* advertise native utility (Linux3 x86_64): [^advertise-linux3_x86_64.zip]
* advertise native utility (WIndows x86): [^advertise-windows_x86.zip]
h3. Demonstration on Linux
h4. System
{noformat}
[mbabacek@perf09 ~]$ ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:18:8b:7a:46:04 brd ff:ff:ff:ff:ff:ff
inet 10.16.88.187/21 brd 10.16.95.255 scope global enp1s0
valid_lft forever preferred_lft forever
inet 10.16.93.253/21 brd 10.16.95.255 scope global secondary enp1s0
valid_lft forever preferred_lft forever
3: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:18:8b:7a:46:05 brd ff:ff:ff:ff:ff:ff
inet 172.17.72.254/19 brd 172.17.95.255 scope global enp2s0
valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
link/ether 02:42:07:ab:74:f9 brd ff:ff:ff:ff:ff:ff
inet 172.18.0.1/16 scope global docker0
valid_lft forever preferred_lft forever
{noformat}
h4. Java
{noformat}
[mbabacek@perf09 ~]$ java -version
openjdk version "1.8.0_71"
OpenJDK Runtime Environment (build 1.8.0_71-b15)
OpenJDK 64-Bit Server VM (build 25.71-b15, mixed mode)
{noformat}
h4.Advertise SENT
{noformat}
[mbabacek@perf09 ~]$ date;./advertise -a 224.0.1.102 -p 33364
Mon Mar 21 12:39:51 EDT 2016
UDP Multicast address to send datagrams to. Value: 224.0.1.102
UDP Multicast port. Value: 33364
IP address of the NIC to bound to. Value: NULL
apr_socket_bind on 0.0.0.0:0
apr_mcast_join on 0.0.0.0:0
apr_socket_sendto to 224.0.1.102:33364
{noformat}
h4. Advertize RECEIVED
YES (/)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364
Linux like OS
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 16:39:51 GMT
received from /10.16.88.187:38907
{noformat}
YES (/)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 10.16.88.187
Linux like OS
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 16:39:51 GMT
received from /10.16.88.187:38907
{noformat}
NO (x)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 172.17.72.254
Linux like OS
ready waiting...
{noformat}
YES (/)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 0.0.0.0
Linux like OS
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 16:39:51 GMT
received from /10.16.88.187:38907
{noformat}
And now let's take a look at {{172.17.72.254}}, i.e. {{enp2s0}}
h4. Advertise SENT
{noformat}
[mbabacek@perf09 ~]$ date;./advertise -a 224.0.1.102 -p 33364 -n 172.17.72.254
Mon Mar 21 12:42:57 EDT 2016
UDP Multicast address to send datagrams to. Value: 224.0.1.102
UDP Multicast port. Value: 33364
IP address of the NIC to bound to. Value: 172.17.72.254
apr_socket_bind on 172.17.72.254:0
apr_mcast_join on 172.17.72.254:0
apr_socket_sendto to 224.0.1.102:33364
{noformat}
h4. Advertize RECEIVED
NO (x)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364
Linux like OS
ready waiting...
{noformat}
NO (x)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 10.16.88.187
Linux like OS
ready waiting...
{noformat}
YES (/)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 172.17.72.254
Linux like OS
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 16:42:57 GMT
received from /172.17.72.254:35452
{noformat}
NO (x)
{noformat}
[mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 0.0.0.0
Linux like OS
ready waiting...
{noformat}
h3. Demonstration on Windows
One could note that the problem doesn't exist on Windows. All interfaces receive advertising.
h4. Advertise SENT
{noformat}
C:\Users\karm\advertise-build
λ advertise.exe -a 224.0.1.102 -p 33364
UDP Multicast address to send datagrams to. Value: 224.0.1.102
UDP Multicast port. Value: 33364
IP address of the NIC to bound to. Value: NULL
apr_socket_bind on 0.0.0.0:0
apr_mcast_join on 0.0.0.0:0
apr_socket_sendto to 224.0.1.102:33364
{noformat}
h4. Advertize RECEIVED
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:07:50 GMT
received from /192.168.122.52:61805
{noformat}
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.52
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:07:50 GMT
received from /192.168.122.52:61805
{noformat}
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.199
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:07:50 GMT
received from /192.168.122.52:61805
{noformat}
h4. Advertise SENT
{noformat}
C:\Users\karm\advertise-build
λ advertise.exe -a 224.0.1.102 -p 33364 -n 192.168.122.199
UDP Multicast address to send datagrams to. Value: 224.0.1.102
UDP Multicast port. Value: 33364
IP address of the NIC to bound to. Value: 192.168.122.199
apr_socket_bind on 192.168.122.199:0
apr_mcast_join on 192.168.122.199:0
apr_socket_sendto to 224.0.1.102:33364
{noformat}
h4. Advertize RECEIVED
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:09:55 GMT
received from /192.168.122.199:52781
{noformat}
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.52
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:09:55 GMT
received from /192.168.122.199:52781
{noformat}
YES (/)
{noformat}
C:\Users\karm\WORKSPACE
λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.199
ready waiting...
received: Advertize !!! Mon, 21 Mar 2016 18:09:55 GMT
received from /192.168.122.199:52781
{noformat}
Ideas? :) [~jfclere], [~rhusar]
I suggest adding {{bindaddr}} instead of NULL or perhaps adding a configuration directive that lets one to pick either MCMP VirtualHost ip address or default -- NULL.
> Advertise: UDP Multicast on Linux systems with more NICs: datagrams could be delayed or lost
> --------------------------------------------------------------------------------------------
>
> Key: MODCLUSTER-487
> URL: https://issues.jboss.org/browse/MODCLUSTER-487
> Project: mod_cluster
> Issue Type: Bug
> Components: Native (httpd modules)
> Affects Versions: 1.2.11.Final, 1.3.1.Final
> Environment: Linux, multiple NICs environment
> Reporter: Michal Karm Babacek
> Assignee: Michal Karm Babacek
> Priority: Critical
> Attachments: advertise-linux3_x86_64.zip, advertise-windows_x86.zip, Advertize.class
>
>
> Credit where it's due: the issue was first spotted by [~rhatlapa].
> h3. Problem
> It appears that trying to send to all interfaces with {{NULL}} or {{"0.0.0.0"}} in the following statement actually picks the first non-loopback interface and sends to it.
> The result is that there is no datagram on other interfaces. Surprisingly, this is not deterministic though: After dozens or hundreds of messages, eventually one datagram reaches another interface.
> {code}
> if ((rv = apr_sockaddr_info_get(&ma_niface_sa, NULL,
> ma_mgroup_sa->family, 0,
> APR_UNSPEC, pool)) != APR_SUCCESS) {
> ap_log_error(APLOG_MARK, APLOG_ERR, rv, s,
> "mod_advertise: ma_group_join apr_sockaddr_info_get(0.0.0.0:0) failed");
> return rv;
> }
> {code}
> h3. Impact
> Picture this simple scenario: There are two interfaces, e.g.
> {noformat}
> enp1s0 10.16.88.187
> enp2s0 172.18.0.1
> {noformat}
> listed in this exact order with {{ip addr show}}.
> One has an EAP 7 (Wildfly 10) instance with mod_cluster bound to {{172.18.0.1}} IP address, which implies {{enp2s0}} interface.
> Furthermore, one has an Apache HTTP Server instance with mod_cluster bound to {{172.18.0.1}} IP address, i.e. MCMP VirtualHost and main VirtualHost all Listen on this IP address.
> Result: Without advertising, using an explicit {{proxy-list}}, all is well. MCMP works, requests work, balancing works.
> On the other hand, relying on advertisement, it could take EAP 7 (Wildfly 10) *minutes* to register with the balancer.
> The reason is that a vast majority of UDP Multicast datagrams arrives at enp1s0 and EAP 7 (Wildfly 10) doesn't see them.
> h3. Reproducer
> Lemme demonstrate with a recently refactored [advertise.c|https://github.com/Karm/mod_cluster/tree/advertise-native-tes...] utility for sending datagrams and the well known [Advertize.java|https://raw.githubusercontent.com/modcluster/mod_cluster/m...] utility for receiving them.
> Your your convenience, here are binaries built from the aforementioned sources:
> * Advertize java utility: [^Advertize.class]
> * advertise native utility (Linux3 x86_64): [^advertise-linux3_x86_64.zip]
> * advertise native utility (WIndows x86): [^advertise-windows_x86.zip]
> h3. Demonstration on Linux
> h4. System
> {noformat}
> [mbabacek@perf09 ~]$ ip addr show
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> inet 127.0.0.1/8 scope host lo
> valid_lft forever preferred_lft forever
> 2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
> link/ether 00:18:8b:7a:46:04 brd ff:ff:ff:ff:ff:ff
> inet 10.16.88.187/21 brd 10.16.95.255 scope global enp1s0
> valid_lft forever preferred_lft forever
> inet 10.16.93.253/21 brd 10.16.95.255 scope global secondary enp1s0
> valid_lft forever preferred_lft forever
> 3: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
> link/ether 00:18:8b:7a:46:05 brd ff:ff:ff:ff:ff:ff
> inet 172.17.72.254/19 brd 172.17.95.255 scope global enp2s0
> valid_lft forever preferred_lft forever
> 4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
> link/ether 02:42:07:ab:74:f9 brd ff:ff:ff:ff:ff:ff
> inet 172.18.0.1/16 scope global docker0
> valid_lft forever preferred_lft forever
> {noformat}
> h4. Java
> {noformat}
> [mbabacek@perf09 ~]$ java -version
> openjdk version "1.8.0_71"
> OpenJDK Runtime Environment (build 1.8.0_71-b15)
> OpenJDK 64-Bit Server VM (build 25.71-b15, mixed mode)
> {noformat}
> h4.Advertise SENT
> {noformat}
> [mbabacek@perf09 ~]$ date;./advertise -a 224.0.1.102 -p 33364
> Mon Mar 21 12:39:51 EDT 2016
> UDP Multicast address to send datagrams to. Value: 224.0.1.102
> UDP Multicast port. Value: 33364
> IP address of the NIC to bound to. Value: NULL
> apr_socket_bind on 0.0.0.0:0
> apr_mcast_join on 0.0.0.0:0
> apr_socket_sendto to 224.0.1.102:33364
> {noformat}
> h4. Advertize RECEIVED
> YES (/)
> {noformat}
> [mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364
> Linux like OS
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 16:39:51 GMT
> received from /10.16.88.187:38907
> {noformat}
> YES (/)
> {noformat}
> [mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 10.16.88.187
> Linux like OS
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 16:39:51 GMT
> received from /10.16.88.187:38907
> {noformat}
> NO (x)
> {noformat}
> [mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 172.17.72.254
> Linux like OS
> ready waiting...
> {noformat}
> YES (/)
> {noformat}
> [mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 0.0.0.0
> Linux like OS
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 16:39:51 GMT
> received from /10.16.88.187:38907
> {noformat}
> And now let's take a look at {{172.17.72.254}}, i.e. {{enp2s0}}
> h4. Advertise SENT
> {noformat}
> [mbabacek@perf09 ~]$ date;./advertise -a 224.0.1.102 -p 33364 -n 172.17.72.254
> Mon Mar 21 12:42:57 EDT 2016
> UDP Multicast address to send datagrams to. Value: 224.0.1.102
> UDP Multicast port. Value: 33364
> IP address of the NIC to bound to. Value: 172.17.72.254
> apr_socket_bind on 172.17.72.254:0
> apr_mcast_join on 172.17.72.254:0
> apr_socket_sendto to 224.0.1.102:33364
> {noformat}
> h4. Advertize RECEIVED
> NO (x)
> {noformat}
> [mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364
> Linux like OS
> ready waiting...
> {noformat}
> NO (x)
> {noformat}
> [mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 10.16.88.187
> Linux like OS
> ready waiting...
> {noformat}
> YES (/)
> {noformat}
> [mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 172.17.72.254
> Linux like OS
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 16:42:57 GMT
> received from /172.17.72.254:35452
> {noformat}
> NO (x)
> {noformat}
> [mbabacek@perf09 ~]$ java Advertize 224.0.1.102 33364 0.0.0.0
> Linux like OS
> ready waiting...
> {noformat}
> h3. Demonstration on Windows
> One could note that the problem doesn't exist on Windows. All interfaces receive advertising.
> h4. Advertise SENT
> {noformat}
> C:\Users\karm\advertise-build
> λ advertise.exe -a 224.0.1.102 -p 33364
> UDP Multicast address to send datagrams to. Value: 224.0.1.102
> UDP Multicast port. Value: 33364
> IP address of the NIC to bound to. Value: NULL
> apr_socket_bind on 0.0.0.0:0
> apr_mcast_join on 0.0.0.0:0
> apr_socket_sendto to 224.0.1.102:33364
> {noformat}
> h4. Advertize RECEIVED
> YES (/)
> {noformat}
> C:\Users\karm\WORKSPACE
> λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 18:07:50 GMT
> received from /192.168.122.52:61805
> {noformat}
> YES (/)
> {noformat}
> C:\Users\karm\WORKSPACE
> λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.52
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 18:07:50 GMT
> received from /192.168.122.52:61805
> {noformat}
> YES (/)
> {noformat}
> C:\Users\karm\WORKSPACE
> λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.199
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 18:07:50 GMT
> received from /192.168.122.52:61805
> {noformat}
> h4. Advertise SENT
> {noformat}
> C:\Users\karm\advertise-build
> λ advertise.exe -a 224.0.1.102 -p 33364 -n 192.168.122.199
> UDP Multicast address to send datagrams to. Value: 224.0.1.102
> UDP Multicast port. Value: 33364
> IP address of the NIC to bound to. Value: 192.168.122.199
> apr_socket_bind on 192.168.122.199:0
> apr_mcast_join on 192.168.122.199:0
> apr_socket_sendto to 224.0.1.102:33364
> {noformat}
> h4. Advertize RECEIVED
> YES (/)
> {noformat}
> C:\Users\karm\WORKSPACE
> λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 18:09:55 GMT
> received from /192.168.122.199:52781
> {noformat}
> YES (/)
> {noformat}
> C:\Users\karm\WORKSPACE
> λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.52
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 18:09:55 GMT
> received from /192.168.122.199:52781
> {noformat}
> YES (/)
> {noformat}
> C:\Users\karm\WORKSPACE
> λ "C:\Program Files\Java\jdk1.8.0_74\bin\java" Advertize 224.0.1.102 33364 192.168.122.199
> ready waiting...
> received: Advertize !!! Mon, 21 Mar 2016 18:09:55 GMT
> received from /192.168.122.199:52781
> {noformat}
> Ideas? :) [~jfclere], [~rhusar]
> I suggest adding {{bindaddr}} from AdvertiseBindAddress instead of NULL or perhaps adding a configuration directive that lets one to pick either MCMP VirtualHost ip address or default -- NULL.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 8 months
- 1
- 0