[mod_cluster-issues] [JBoss JIRA] (MODCLUSTER-390) add failonstatus to mod_cluster

Michal Babacek (JIRA) issues at jboss.org
Thu Sep 11 07:35:19 EDT 2014 

    [ https://issues.jboss.org/browse/MODCLUSTER-390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13001320#comment-13001320 ] 

Michal Babacek edited comment on MODCLUSTER-390 at 9/11/14 7:35 AM:
--------------------------------------------------------------------

@[~jfclere] Hmm, the forced failover doesn't happen at once, it takes several requests...is that the correct behaviour? Take a look:

h3. Setup
 * Apache HTTP Server: 2.2.26 from EAP 6.3.0 distro
 * Two EAP 6.3 servers, one called _jboss-eap-6.3_ and the other being _jboss-eap-6.3-2_
 * mod_cluster modules build from the latest 1.2.x upstream

This is my mod_cluster.conf:

{code}
MemManagerFile "/dev/shm/mod_cluster-eapx/jboss-ews-2.1/httpd/cache/mod_cluster"

ProxyPass / balancer://balancerxxx stickysession=JSESSIONID|jsessionid nofailover=on failonstatus=203,204
ProxyPassReverse / balancer://balancerxxx
ProxyPreserveHost on

ServerName 192.168.122.204:2181

<IfModule manager_module>
  Listen 192.168.122.204:8847
  LogLevel debug
  <VirtualHost 192.168.122.204:8847>
    ServerName 192.168.122.204:8847
    <Directory />
      Order deny,allow
      Deny from all
      Allow from all
    </Directory>
    KeepAliveTimeout 60
    MaxKeepAliveRequests 0
    ServerAdvertise on
    AdvertiseFrequency 5
    ManagerBalancerName balancerxxx
    AdvertiseGroup 224.0.5.79:65009
    EnableMCPMReceive
    CreateBalancers 1
    <Location /mcm>
      SetHandler mod_cluster-manager
      Order deny,allow
      Deny from all
      Allow from all
    </Location>
  </VirtualHost>
</IfModule>
{code}

EAP config...

{code}
<subsystem xmlns="urn:jboss:domain:modcluster:1.2">
    <mod-cluster-config advertise-socket="modcluster" sticky-session="true"
                        sticky-session-remove="false" sticky-session-force="false"
                        connector="ajp">
        <dynamic-load-provider>
            <load-metric type="busyness"/>
        </dynamic-load-provider>
    </mod-cluster-config>
</subsystem>
{code}

h3. Test

This dead simple [servlet|https://github.com/Karm/clusterbench/blob/simplified-and-pure/clusterbench-common/src/main/java/org/jboss/test/clusterbench/common/jvmroute/CommonHTTPCodeServlet.java#L40] sets an arbitrary response return code according to the given parameter. The output is the following:

Warming up, {{http_code=200}} is not one of the defined error codes, so everything should be normal. The session is created on the first request, the subsequent ones are the same, no failover, cool...

{code}

mbabacek at localhost:~$ curl http://rhel6x86-64:8847/clusterbench/httpcode/?http_code=200 -c cookie.txt -b cookie.txt -i
HTTP/1.1 200 OK
Date: Thu, 11 Sep 2014 11:02:22 GMT
Set-Cookie: JSESSIONID=gMPIq7FfCeiPwzM2iZLyJiR4.jboss-eap-6.3; Path=/clusterbench
Content-Type: text/plain;charset=UTF-8
Content-Length: 122
Connection: close

Done. 
HTTP Code was: 200
JVM route: jboss-eap-6.3
Session ID: gMPIq7FfCeiPwzM2iZLyJiR4.jboss-eap-6.3
Session isNew: true


mbabacek at localhost:~$ curl http://rhel6x86-64:8847/clusterbench/httpcode/?http_code=200 -c cookie.txt -b cookie.txt -i
HTTP/1.1 200 OK
Date: Thu, 11 Sep 2014 11:02:24 GMT
Content-Type: text/plain;charset=UTF-8
Content-Length: 123
Connection: close

Done. 
HTTP Code was: 200
JVM route: jboss-eap-6.3
Session ID: gMPIq7FfCeiPwzM2iZLyJiR4.jboss-eap-6.3
Session isNew: false


mbabacek at localhost:~$ curl http://rhel6x86-64:8847/clusterbench/httpcode/?http_code=200 -c cookie.txt -b cookie.txt -i
HTTP/1.1 200 OK
Date: Thu, 11 Sep 2014 11:02:27 GMT
Content-Type: text/plain;charset=UTF-8
Content-Length: 123
Connection: close

Done. 
HTTP Code was: 200
JVM route: jboss-eap-6.3
Session ID: gMPIq7FfCeiPwzM2iZLyJiR4.jboss-eap-6.3
Session isNew: false
{code}
Now, let's try {{http_code=203}}, one of the predefined error codes. What I would expect is an immediate failover from _jboss-eap-6.3_ to _jboss-eap-6.3-2_, although this is what happens:
{code}
mbabacek at localhost:~$ curl http://rhel6x86-64:8847/clusterbench/httpcode/?http_code=203 -c cookie.txt -b cookie.txt -i
HTTP/1.1 203 Non-Authoritative Information
Date: Thu, 11 Sep 2014 11:02:39 GMT
Content-Type: text/plain;charset=UTF-8
Content-Length: 123
Connection: close

Done. 
HTTP Code was: 203
JVM route: jboss-eap-6.3
Session ID: gMPIq7FfCeiPwzM2iZLyJiR4.jboss-eap-6.3
Session isNew: false


mbabacek at localhost:~$ curl http://rhel6x86-64:8847/clusterbench/httpcode/?http_code=203 -c cookie.txt -b cookie.txt -i
HTTP/1.1 203 Non-Authoritative Information
Date: Thu, 11 Sep 2014 11:03:03 GMT
Content-Type: text/plain;charset=UTF-8
Content-Length: 123
Connection: close

Done. 
HTTP Code was: 203
JVM route: jboss-eap-6.3
Session ID: gMPIq7FfCeiPwzM2iZLyJiR4.jboss-eap-6.3
Session isNew: false


mbabacek at localhost:~$ curl http://rhel6x86-64:8847/clusterbench/httpcode/?http_code=203 -c cookie.txt -b cookie.txt -i
HTTP/1.1 203 Non-Authoritative Information
Date: Thu, 11 Sep 2014 11:03:16 GMT
Content-Type: text/plain;charset=UTF-8
Content-Length: 123
Connection: close

Done. 
HTTP Code was: 203
JVM route: jboss-eap-6.3
Session ID: gMPIq7FfCeiPwzM2iZLyJiR4.jboss-eap-6.3
Session isNew: false


mbabacek at localhost:~$ curl http://rhel6x86-64:8847/clusterbench/httpcode/?http_code=203 -c cookie.txt -b cookie.txt -i
HTTP/1.1 203 Non-Authoritative Information
Date: Thu, 11 Sep 2014 11:03:19 GMT
Content-Type: text/plain;charset=UTF-8
Content-Length: 123
Connection: close

Done. 
HTTP Code was: 203
JVM route: jboss-eap-6.3
Session ID: gMPIq7FfCeiPwzM2iZLyJiR4.jboss-eap-6.3
Session isNew: false


mbabacek at localhost:~$ curl http://rhel6x86-64:8847/clusterbench/httpcode/?http_code=203 -c cookie.txt -b cookie.txt -i
HTTP/1.1 203 Non-Authoritative Information
Date: Thu, 11 Sep 2014 11:03:21 GMT
Set-Cookie: JSESSIONID=gMPIq7FfCeiPwzM2iZLyJiR4.jboss-eap-6.3-2; Path=/clusterbench
Content-Type: text/plain;charset=UTF-8
Content-Length: 127
Connection: close

Done. 
HTTP Code was: 203
JVM route: jboss-eap-6.3-2
Session ID: gMPIq7FfCeiPwzM2iZLyJiR4.jboss-eap-6.3-2
Session isNew: false
{code}
There were four requests and four _HTTP 203_ responses before the failover finally occurred. Is this correct? Continuing with {{http_code=200}} looks normal, it stays on the new server:

{code}
mbabacek at localhost:~$ curl http://rhel6x86-64:8847/clusterbench/httpcode/?http_code=200 -c cookie.txt -b cookie.txt -i
HTTP/1.1 200 OK
Date: Thu, 11 Sep 2014 11:03:34 GMT
Content-Type: text/plain;charset=UTF-8
Content-Length: 127
Connection: close

Done. 
HTTP Code was: 200
JVM route: jboss-eap-6.3-2
Session ID: gMPIq7FfCeiPwzM2iZLyJiR4.jboss-eap-6.3-2
Session isNew: false


mbabacek at localhost:~$ curl http://rhel6x86-64:8847/clusterbench/httpcode/?http_code=200 -c cookie.txt -b cookie.txt -i
HTTP/1.1 200 OK
Date: Thu, 11 Sep 2014 11:03:38 GMT
Content-Type: text/plain;charset=UTF-8
Content-Length: 127
Connection: close

Done. 
HTTP Code was: 200
JVM route: jboss-eap-6.3-2
Session ID: gMPIq7FfCeiPwzM2iZLyJiR4.jboss-eap-6.3-2
Session isNew: false


mbabacek at localhost:~$ curl http://rhel6x86-64:8847/clusterbench/httpcode/?http_code=200 -c cookie.txt -b cookie.txt -i
HTTP/1.1 200 OK
Date: Thu, 11 Sep 2014 11:03:40 GMT
Content-Type: text/plain;charset=UTF-8
Content-Length: 127
Connection: close

Done. 
HTTP Code was: 200
JVM route: jboss-eap-6.3-2
Session ID: gMPIq7FfCeiPwzM2iZLyJiR4.jboss-eap-6.3-2
Session isNew: false
{code}

This is the complete [^error_log.zip]. It's noteworthy that message: {{proxy: BALANCER: (balancer://balancerxxx).  Forcing recovery for worker (ajp://192.168.122.204:8009), failonstatus 203}} seems to by duly present...

WDYT?


was (Author: mbabacek):
@[~jfclere] Hmm, the forced failover doesn't happen at once, it takes several requests...is that the correct behaviour? Take a look:

h3. Setup
 * Apache HTTP Server: 2.2.26 from EAP 6.3.0 distro
 * Two EAP 6.3 servers, one called _jboss-eap-6.3_ and the other being _jboss-eap-6.3-2_
 * mod_cluster modules build from the latest 1.2.x upstream

This is my mod_cluster.conf:

{code}
MemManagerFile "/dev/shm/mod_cluster-eapx/jboss-ews-2.1/httpd/cache/mod_cluster"

ProxyPass / balancer://balancerxxx stickysession=JSESSIONID|jsessionid nofailover=on failonstatus=203,204
ProxyPassReverse / balancer://balancerxxx
ProxyPreserveHost on

ServerName 192.168.122.204:2181

<IfModule manager_module>
  Listen 192.168.122.204:8847
  LogLevel debug
  <VirtualHost 192.168.122.204:8847>
    ServerName 192.168.122.204:8847
    <Directory />
      Order deny,allow
      Deny from all
      Allow from all
    </Directory>
    KeepAliveTimeout 60
    MaxKeepAliveRequests 0
    ServerAdvertise on
    AdvertiseFrequency 5
    ManagerBalancerName balancerxxx
    AdvertiseGroup 224.0.5.79:65009
    EnableMCPMReceive
    CreateBalancers 1
    <Location /mcm>
      SetHandler mod_cluster-manager
      Order deny,allow
      Deny from all
      Allow from all
    </Location>
  </VirtualHost>
</IfModule>
{code}

EAP config...

{code}
<subsystem xmlns="urn:jboss:domain:modcluster:1.2">
    <mod-cluster-config advertise-socket="modcluster" sticky-session="true"
                        sticky-session-remove="false" sticky-session-force="false"
                        connector="ajp">
        <dynamic-load-provider>
            <load-metric type="busyness"/>
        </dynamic-load-provider>
    </mod-cluster-config>
</subsystem>
{code}

h3. Test

This dead simple [servlet|https://github.com/Karm/clusterbench/blob/simplified-and-pure/clusterbench-common/src/main/java/org/jboss/test/clusterbench/common/jvmroute/CommonHTTPCodeServlet.java#L40] sets an arbitrary response return code according to the given parameter. The output is the following:

Warming up, {{http_code=200}} is not one of the defined error codes, so everything should be normal. The session is created on the first request, the subsequent ones are the same, no failover, cool...

{code}

mbabacek at localhost:~$ curl http://rhel6x86-64:8847/clusterbench/httpcode/?http_code=200 -c cookie.txt -b cookie.txt -i
HTTP/1.1 200 OK
Date: Thu, 11 Sep 2014 11:02:22 GMT
Set-Cookie: JSESSIONID=gMPIq7FfCeiPwzM2iZLyJiR4.jboss-eap-6.3; Path=/clusterbench
Content-Type: text/plain;charset=UTF-8
Content-Length: 122
Connection: close

Done. 
HTTP Code was: 200
JVM route: jboss-eap-6.3
Session ID: gMPIq7FfCeiPwzM2iZLyJiR4.jboss-eap-6.3
Session isNew: true


mbabacek at localhost:~$ curl http://rhel6x86-64:8847/clusterbench/httpcode/?http_code=200 -c cookie.txt -b cookie.txt -i
HTTP/1.1 200 OK
Date: Thu, 11 Sep 2014 11:02:24 GMT
Content-Type: text/plain;charset=UTF-8
Content-Length: 123
Connection: close

Done. 
HTTP Code was: 200
JVM route: jboss-eap-6.3
Session ID: gMPIq7FfCeiPwzM2iZLyJiR4.jboss-eap-6.3
Session isNew: false


mbabacek at localhost:~$ curl http://rhel6x86-64:8847/clusterbench/httpcode/?http_code=200 -c cookie.txt -b cookie.txt -i
HTTP/1.1 200 OK
Date: Thu, 11 Sep 2014 11:02:27 GMT
Content-Type: text/plain;charset=UTF-8
Content-Length: 123
Connection: close

Done. 
HTTP Code was: 200
JVM route: jboss-eap-6.3
Session ID: gMPIq7FfCeiPwzM2iZLyJiR4.jboss-eap-6.3
Session isNew: false
{code}
Now, let's try {{http_code=203}}, one of the predefined error codes. What I would expect is an immediate failover from _jboss-eap-6.3_ to _jboss-eap-6.3-2_, although this is what happens:
{code}
mbabacek at localhost:~$ curl http://rhel6x86-64:8847/clusterbench/httpcode/?http_code=203 -c cookie.txt -b cookie.txt -i
HTTP/1.1 203 Non-Authoritative Information
Date: Thu, 11 Sep 2014 11:02:39 GMT
Content-Type: text/plain;charset=UTF-8
Content-Length: 123
Connection: close

Done. 
HTTP Code was: 203
JVM route: jboss-eap-6.3
Session ID: gMPIq7FfCeiPwzM2iZLyJiR4.jboss-eap-6.3
Session isNew: false


mbabacek at localhost:~$ curl http://rhel6x86-64:8847/clusterbench/httpcode/?http_code=203 -c cookie.txt -b cookie.txt -i
HTTP/1.1 203 Non-Authoritative Information
Date: Thu, 11 Sep 2014 11:03:03 GMT
Content-Type: text/plain;charset=UTF-8
Content-Length: 123
Connection: close

Done. 
HTTP Code was: 203
JVM route: jboss-eap-6.3
Session ID: gMPIq7FfCeiPwzM2iZLyJiR4.jboss-eap-6.3
Session isNew: false


mbabacek at localhost:~$ curl http://rhel6x86-64:8847/clusterbench/httpcode/?http_code=203 -c cookie.txt -b cookie.txt -i
HTTP/1.1 203 Non-Authoritative Information
Date: Thu, 11 Sep 2014 11:03:16 GMT
Content-Type: text/plain;charset=UTF-8
Content-Length: 123
Connection: close

Done. 
HTTP Code was: 203
JVM route: jboss-eap-6.3
Session ID: gMPIq7FfCeiPwzM2iZLyJiR4.jboss-eap-6.3
Session isNew: false


mbabacek at localhost:~$ curl http://rhel6x86-64:8847/clusterbench/httpcode/?http_code=203 -c cookie.txt -b cookie.txt -i
HTTP/1.1 203 Non-Authoritative Information
Date: Thu, 11 Sep 2014 11:03:19 GMT
Content-Type: text/plain;charset=UTF-8
Content-Length: 123
Connection: close

Done. 
HTTP Code was: 203
JVM route: jboss-eap-6.3
Session ID: gMPIq7FfCeiPwzM2iZLyJiR4.jboss-eap-6.3
Session isNew: false


mbabacek at localhost:~$ curl http://rhel6x86-64:8847/clusterbench/httpcode/?http_code=203 -c cookie.txt -b cookie.txt -i
HTTP/1.1 203 Non-Authoritative Information
Date: Thu, 11 Sep 2014 11:03:21 GMT
Set-Cookie: JSESSIONID=gMPIq7FfCeiPwzM2iZLyJiR4.jboss-eap-6.3-2; Path=/clusterbench
Content-Type: text/plain;charset=UTF-8
Content-Length: 127
Connection: close

Done. 
HTTP Code was: 203
JVM route: jboss-eap-6.3-2
Session ID: gMPIq7FfCeiPwzM2iZLyJiR4.jboss-eap-6.3-2
Session isNew: false
{code}
There were four requests and four _HTTP 203_ responses before the failover finally occurred. Is this correct? Continuing with {{http_code=200}} looks normal, it stays on the new server:

{code}
mbabacek at localhost:~$ curl http://rhel6x86-64:8847/clusterbench/httpcode/?http_code=200 -c cookie.txt -b cookie.txt -i
HTTP/1.1 200 OK
Date: Thu, 11 Sep 2014 11:03:34 GMT
Content-Type: text/plain;charset=UTF-8
Content-Length: 127
Connection: close

Done. 
HTTP Code was: 200
JVM route: jboss-eap-6.3-2
Session ID: gMPIq7FfCeiPwzM2iZLyJiR4.jboss-eap-6.3-2
Session isNew: false


mbabacek at localhost:~$ curl http://rhel6x86-64:8847/clusterbench/httpcode/?http_code=200 -c cookie.txt -b cookie.txt -i
HTTP/1.1 200 OK
Date: Thu, 11 Sep 2014 11:03:38 GMT
Content-Type: text/plain;charset=UTF-8
Content-Length: 127
Connection: close

Done. 
HTTP Code was: 200
JVM route: jboss-eap-6.3-2
Session ID: gMPIq7FfCeiPwzM2iZLyJiR4.jboss-eap-6.3-2
Session isNew: false


mbabacek at localhost:~$ curl http://rhel6x86-64:8847/clusterbench/httpcode/?http_code=200 -c cookie.txt -b cookie.txt -i
HTTP/1.1 200 OK
Date: Thu, 11 Sep 2014 11:03:40 GMT
Content-Type: text/plain;charset=UTF-8
Content-Length: 127
Connection: close

Done. 
HTTP Code was: 200
JVM route: jboss-eap-6.3-2
Session ID: gMPIq7FfCeiPwzM2iZLyJiR4.jboss-eap-6.3-2
Session isNew: false
{code}

This is the complete [^error_log]. It's noteworthy that message: {{proxy: BALANCER: (balancer://balancerxxx).  Forcing recovery for worker (ajp://192.168.122.204:8009), failonstatus 203}} seems to by duly present...

WDYT?

> add failonstatus to mod_cluster
> -------------------------------
>
>                 Key: MODCLUSTER-390
>                 URL: https://issues.jboss.org/browse/MODCLUSTER-390
>             Project: mod_cluster
>          Issue Type: Enhancement
>    Affects Versions: 1.2.6.Final, 1.3.0.Final
>            Reporter: Jean-Frederic Clere
>            Assignee: Jean-Frederic Clere
>             Fix For: 1.2.8.Final, 1.3.1.Alpha2
>
>
> The feature was added in httpd-2.2.17 in the mod_proxy_balancer we just need to copy that code and find a way to configure it,



--
This message was sent by Atlassian JIRA
(v6.3.1#6329)

More information about the mod_cluster-issues mailing list