]
Antoine Cotten edited comment on MODCLUSTER-570 at 3/30/17 11:28 AM:
---------------------------------------------------------------------
I managed to reproduce on 1.3.6.Final. It's actually as simple as doing multiple
graceful restarts in a row, no need for an application server (but happens with apps
servers as well).
And most importantly, it seems specific to *Alpine Linux*.
What I did:
# Run httpd with mod_cluster
{code}docker run --name mc -p 8080:80 -p 7777:6666
antoineco/httpd-mod_cluster:1.3.6-alpine{code}
# Uncomment the mod_cluster modules in the config, and uncomment the very last line
({{Include}}) to enable the manager vhost
{code}docker exec -it mc vi conf/httpd.conf{code}
# Gracefully reload httpd:
{code}docker exec mc apachectl -k graceful{code}
# Repeat 3. two times if no app server got registered (happens right away if an app server
is registered)
The same error messages as described in the description will surface.
was (Author: antoineco):
I managed to reproduce on 1.3.6.Final. It's actually as simple as doing multiple
graceful restarts in a row, no need for an application server.
What I did:
# Run httpd with mod_cluster
{code}docker run --name mc -p 8080:80 -p 7777:6666
antoineco/httpd-mod_cluster:1.3.6-alpine{code}
# Uncomment the mod_cluster modules in the config, and uncomment the very last line
({{Include}}) to enable the manager vhost
{code}docker exec -it mc vi conf/httpd.conf{code}
# Gracefully reload httpd:
{code}docker exec mc apachectl -k graceful{code}
# Repeat 3. two times
The same error messages as described in the description will surface.
This seems to happen only the first time mod_cluster is started, if I stop httpd and start
it again later on I can't reproduce.
Multiple invalid nodes created on graceful httpd restart
--------------------------------------------------------
Key: MODCLUSTER-570
URL:
https://issues.jboss.org/browse/MODCLUSTER-570
Project: mod_cluster
Issue Type: Bug
Components: Native (httpd modules)
Affects Versions: 1.3.5.Final, 1.3.6.Final
Environment: httpd 2.4.25
Reporter: Antoine Cotten
Assignee: Jean-Frederic Clere
Priority: Minor
Attachments: mod_cluster_after_graceful.png, mod_cluster_before_graceful.png
mod_cluster creates a bunch of invalid nodes after a graceful restart of Apache httpd
({{SIGUSR1}}, {{SIGWINCH}}). The issue happens with or without backend servers in the
pool.
httpd logs:
{code}
[mpm_event:notice] [pid 1:tid 140313388616520] AH00493: SIGUSR1 received. Doing graceful
restart
[:notice] [pid 1:tid 140313388616520] Advertise initialized for process 1
[mpm_event:notice] [pid 1:tid 140313388616520] AH00489: Apache/2.4.25 (Unix)
mod_cluster/1.3.5.Final configured -- resuming normal operations
[core:notice] [pid 1:tid 140313388616520] AH00094: Command line: 'httpd -D
FOREGROUND'
[:notice] [pid 116:tid 140313388616520] Created: worker for ://: failed: Unable to parse
URL
[:notice] [pid 116:tid 140313388616520] Created: worker for ://: failed: Unable to parse
URL
[:notice] [pid 116:tid 140313388616520] Created: worker for ://: failed: Unable to parse
URL
# ... trimmed 48 lines ...
[:notice] [pid 145:tid 140313388616520] Created: worker for ://: failed: Unable to parse
URL
[:notice] [pid 145:tid 140313388616520] Created: worker for ://: failed: Unable to parse
URL
[:notice] [pid 145:tid 140313388616520] Created: worker for ://: failed: Unable to parse
URL
[:notice] [pid 145:tid 140313388616520] Created: worker for ://:\x9d\x7f failed: Unable
to parse URL
[:notice] [pid 145:tid 140313388616520] add_balancer_node: balancer safe-name
(balancer://4:6666\r\nX-Manager-Url:
/c8dae916-7642-4deb-b841-2f3435613749\r\nX-Manager-Protocol: http\r\nX-Manager-Host:
172.17.0.4\r\n\r\n) too long
{code}
Application server logs (tomcat8):
{code}
Feb 01, 2017 11:34:25 AM org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler sendRequest
ERROR: MODCLUSTER000042: Error MEM sending STATUS command to 172.17.0.4/172.17.0.4:6666,
configuration will be reset: MEM: Can't read node with "java1-depl1-1234"
JVMRoute
Feb 01, 2017 11:34:35 AM
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor processChildren
SEVERE: Exception invoking periodic operation:
java.lang.IllegalArgumentException: Node: [1],Name: ,Balancer: ,LBGroup: ,Host: ,Port:
,Type: ,Flushpackets: Off,Flushwait: 0,Ping: 0,Smax: 0,Ttl: 0,Elected: 0,Read:
0,Transfered: 0,Connected: 0,Load: 0
Node: [2],Name: ,Balancer: ,LBGroup: ,Host: ,Port: ,Type: ,Flushpackets: Off,Flushwait:
0,Ping: 0,Smax: 0,Ttl: 0,Elected: 0,Read: 0,Transfered: 0,Connected: 0,Load: 0
Node: [3],Name: ,Balancer: ,LBGroup: ,Host: ,Port: ,Type: ,Flushpackets: Off,Flushwait:
0,Ping: 0,Smax: 0,Ttl: 0,Elected: 0,Read: 0,Transfered: 0,Connected: 0,Load: 0
# ... trimmed lines ...
Node: [18],Name: ,Balancer: ,LBGroup: ,Host: ,Port: ,Type: ,Flushpackets: Off,Flushwait:
0,Ping: 0,Smax: 0,Ttl: 0,Elected: 0,Read: 0,Transfered: 0,Connected: 0,Load: 0
Node: [19],Name: ,Balancer: ,LBGroup: ,Host: ,Port: ,Type: ,Flushpackets: Off,Flushwait:
0,Ping: 0,Smax: 0,Ttl: 0,Elected: 0,Read: 0,Transfered: 0,Connected: 0,Load: 0
Vhost: [0:0:1], Alias:
Vhost: [0:0:2], Alias:
Vhost: [0:0:3], Alias:
# ... trimmed lines ...
Vhost: [0:0:19], Alias:
Vhost: [0:0:20], Alias:
Context: [0:0:1], Context: , Status: REMOVED
Context: [0:0:2], Context: , Status: REMOVED
Context: [0:0:3], Context: , Status: REMOVED
# ... trimmed lines ...
Context: [0:0:97], Context: , Status: REMOVED
Context: [0:0:98], Context: , Status: REMOVED
Context: [0:0:99], Context: , Status: REMOVED
Context: [0:0:1], Context: , Status: REMOVED
at
org.jboss.modcluster.mcmp.impl.DefaultMCMPResponseParser.parseInfoResponse(DefaultMCMPResponseParser.java:96)
at
org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.status(DefaultMCMPHandler.java:396)
at
org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.status(DefaultMCMPHandler.java:365)
at org.jboss.modcluster.ModClusterService.status(ModClusterService.java:457)
at
org.jboss.modcluster.container.catalina.CatalinaEventHandlerAdapter.lifecycleEvent(CatalinaEventHandlerAdapter.java:252)
at
org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)
at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1374)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1546)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1524)
at java.lang.Thread.run(Thread.java:745)
Feb 01, 2017 11:35:05 AM org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler sendRequest
{code}