[JBoss JIRA] (MODCLUSTER-391) mod_cluster and mod_proxy integration
by Michal Babacek (JIRA)
[ https://issues.jboss.org/browse/MODCLUSTER-391?page=com.atlassian.jira.pl... ]
Michal Babacek updated MODCLUSTER-391:
--------------------------------------
Bugzilla Update: (was: Perform)
> mod_cluster and mod_proxy integration
> -------------------------------------
>
> Key: MODCLUSTER-391
> URL: https://issues.jboss.org/browse/MODCLUSTER-391
> Project: mod_cluster
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Affects Versions: 1.2.6.Final
> Environment: All platforms we build mod_cluster for.
> Reporter: Michal Babacek
> Assignee: Jean-Frederic Clere
> Labels: native_libraries
> Fix For: 1.3.1.Final, 1.2.10.Final
>
> Attachments: error_log, mod_cluster.conf, mod_proxy.conf, standalone-ha.xml
>
>
> This Jira encapsulates all concerns regarding mod_cluster - mod_proxy integration. For instance, while basic {{ProxyPass}} settings work just fine, e.g. serving some files on {{/static}} from the Apache HTTP itself:
> {code}
> ProxyPassMatch ^/static/ !
> ProxyPass / balancer://qacluster stickysession=JSESSIONID|jsessionid nofailover=on
> ProxyPassReverse / balancer://qacluster
> ProxyPreserveHost on
> {code}
> there are more complex setups, involving {{BalancerMember}} configurations, that do not work as expected. In the following example, one wanted to have {{/clusterbench}} application managed by mod_cluster, dynamically, while at the same time, in a different VirtualHost, having {{/tses}} application handled by manually created mod_proxy balancer settings.
> Attached [^mod_cluster.conf], [^mod_proxy.conf], [^standalone-ha.xml](modcluster subsystem element only) and [^error_log].
> The aforementioned setup resulted in:
> |HTTP 200|(From worker)|http://10.16.88.19:8847/clusterbench/requestinfo/|OK|(/)|
> |HTTP 404|(From httpd)|http://10.16.88.19:8847/tses/session.jsp|Expected fail|(/)|
> |HTTP 503|(From httpd)|http://10.16.88.19:2182/tses/session.jsp|Unexpected fail|(x)|
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
10 years, 4 months
[JBoss JIRA] (MODCLUSTER-391) mod_cluster and mod_proxy integration
by Michal Babacek (JIRA)
[ https://issues.jboss.org/browse/MODCLUSTER-391?page=com.atlassian.jira.pl... ]
Michal Babacek updated MODCLUSTER-391:
--------------------------------------
Comment: was deleted
(was: Kabir Khan <kkhan(a)redhat.com> changed the Status of [bug 987259|https://bugzilla.redhat.com/show_bug.cgi?id=987259] from NEW to MODIFIED)
> mod_cluster and mod_proxy integration
> -------------------------------------
>
> Key: MODCLUSTER-391
> URL: https://issues.jboss.org/browse/MODCLUSTER-391
> Project: mod_cluster
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Affects Versions: 1.2.6.Final
> Environment: All platforms we build mod_cluster for.
> Reporter: Michal Babacek
> Assignee: Jean-Frederic Clere
> Labels: native_libraries
> Fix For: 1.3.1.Final, 1.2.10.Final
>
> Attachments: error_log, mod_cluster.conf, mod_proxy.conf, standalone-ha.xml
>
>
> This Jira encapsulates all concerns regarding mod_cluster - mod_proxy integration. For instance, while basic {{ProxyPass}} settings work just fine, e.g. serving some files on {{/static}} from the Apache HTTP itself:
> {code}
> ProxyPassMatch ^/static/ !
> ProxyPass / balancer://qacluster stickysession=JSESSIONID|jsessionid nofailover=on
> ProxyPassReverse / balancer://qacluster
> ProxyPreserveHost on
> {code}
> there are more complex setups, involving {{BalancerMember}} configurations, that do not work as expected. In the following example, one wanted to have {{/clusterbench}} application managed by mod_cluster, dynamically, while at the same time, in a different VirtualHost, having {{/tses}} application handled by manually created mod_proxy balancer settings.
> Attached [^mod_cluster.conf], [^mod_proxy.conf], [^standalone-ha.xml](modcluster subsystem element only) and [^error_log].
> The aforementioned setup resulted in:
> |HTTP 200|(From worker)|http://10.16.88.19:8847/clusterbench/requestinfo/|OK|(/)|
> |HTTP 404|(From httpd)|http://10.16.88.19:8847/tses/session.jsp|Expected fail|(/)|
> |HTTP 503|(From httpd)|http://10.16.88.19:2182/tses/session.jsp|Unexpected fail|(x)|
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
10 years, 4 months
[JBoss JIRA] (MODCLUSTER-407) worker-timeout can cause httpd thread stalls
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/MODCLUSTER-407?page=com.atlassian.jira.pl... ]
RH Bugzilla Integration commented on MODCLUSTER-407:
----------------------------------------------------
Pavel Slavicek <pslavice(a)redhat.com> changed the Status of [bug 1098564|https://bugzilla.redhat.com/show_bug.cgi?id=1098564] from VERIFIED to CLOSED
> worker-timeout can cause httpd thread stalls
> --------------------------------------------
>
> Key: MODCLUSTER-407
> URL: https://issues.jboss.org/browse/MODCLUSTER-407
> Project: mod_cluster
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Affects Versions: 1.2.8.Final
> Reporter: Aaron Ogburn
> Assignee: Jean-Frederic Clere
> Fix For: 1.3.1.Final, 1.2.9.Final
>
>
> Setting a modcluster worker-timeout can stall requests and threads on the httpd side when the requests are received with workers in a down state. A stack of the problem thread looks like the following (recursive loops through mod_proxy_cluster from #160 to #2):
> #0 0x00007ff8eb547533 in select () from /lib64/libc.so.6
> #1 0x00007ff8eba39185 in apr_sleep () from /usr/lib64/libapr-1.so.0
> #2 0x00007ff8e84be0d1 in ?? () from /etc/httpd/modules/mod_proxy_cluster.so
> ...
> #160 0x00007ff8e84beb9f in ?? () from /etc/httpd/modules/mod_proxy_cluster.so
> #161 0x00007ff8e88d2116 in proxy_run_pre_request () from /etc/httpd/modules/mod_proxy.so
> #162 0x00007ff8e88d9186 in ap_proxy_pre_request () from /etc/httpd/modules/mod_proxy.so
> #163 0x00007ff8e88d63c2 in ?? () from /etc/httpd/modules/mod_proxy.so
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
10 years, 4 months
[JBoss JIRA] (MODCLUSTER-391) mod_cluster and mod_proxy integration
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/MODCLUSTER-391?page=com.atlassian.jira.pl... ]
RH Bugzilla Integration commented on MODCLUSTER-391:
----------------------------------------------------
Pavel Slavicek <pslavice(a)redhat.com> changed the Status of [bug 987259|https://bugzilla.redhat.com/show_bug.cgi?id=987259] from VERIFIED to CLOSED
> mod_cluster and mod_proxy integration
> -------------------------------------
>
> Key: MODCLUSTER-391
> URL: https://issues.jboss.org/browse/MODCLUSTER-391
> Project: mod_cluster
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Affects Versions: 1.2.6.Final
> Environment: All platforms we build mod_cluster for.
> Reporter: Michal Babacek
> Assignee: Jean-Frederic Clere
> Labels: native_libraries
> Fix For: 1.3.1.Final, 1.2.10.Final
>
> Attachments: error_log, mod_cluster.conf, mod_proxy.conf, standalone-ha.xml
>
>
> This Jira encapsulates all concerns regarding mod_cluster - mod_proxy integration. For instance, while basic {{ProxyPass}} settings work just fine, e.g. serving some files on {{/static}} from the Apache HTTP itself:
> {code}
> ProxyPassMatch ^/static/ !
> ProxyPass / balancer://qacluster stickysession=JSESSIONID|jsessionid nofailover=on
> ProxyPassReverse / balancer://qacluster
> ProxyPreserveHost on
> {code}
> there are more complex setups, involving {{BalancerMember}} configurations, that do not work as expected. In the following example, one wanted to have {{/clusterbench}} application managed by mod_cluster, dynamically, while at the same time, in a different VirtualHost, having {{/tses}} application handled by manually created mod_proxy balancer settings.
> Attached [^mod_cluster.conf], [^mod_proxy.conf], [^standalone-ha.xml](modcluster subsystem element only) and [^error_log].
> The aforementioned setup resulted in:
> |HTTP 200|(From worker)|http://10.16.88.19:8847/clusterbench/requestinfo/|OK|(/)|
> |HTTP 404|(From httpd)|http://10.16.88.19:8847/tses/session.jsp|Expected fail|(/)|
> |HTTP 503|(From httpd)|http://10.16.88.19:2182/tses/session.jsp|Unexpected fail|(x)|
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
10 years, 4 months
[JBoss JIRA] (MODCLUSTER-425) The timeout specified has expired: proxy: AJP: cping/cpong failed, ajp_ilink_receive() can't receive header
by Michal Babacek (JIRA)
[ https://issues.jboss.org/browse/MODCLUSTER-425?page=com.atlassian.jira.pl... ]
Michal Babacek commented on MODCLUSTER-425:
-------------------------------------------
* worker-timeout: Does not help
* playing with max-connections on AS7 side helps a bit
* prepending {{apr_socket_timeout_set(backend->sock, timeout);}} before {{status = ajp_handle_cping_cpong(backend->sock, r, timeout);}} has no effect
* reconnecting immediately after failure does not help {code}status = ap_proxy_connect_backend(scheme, backend, worker, r->server);
status = ajp_handle_cping_cpong(backend->sock, r, timeout);{code}
* IMHO as soon as the available connections (~2000 {{netstat -an | grep 8009}}) are exhausted, there isn't much one can do to prevent the mod_proxy_ajp timeout errors.
* TODO: Compare directly to mod_jk
> The timeout specified has expired: proxy: AJP: cping/cpong failed,ajp_ilink_receive() can't receive header
> ----------------------------------------------------------------------------------------------------------
>
> Key: MODCLUSTER-425
> URL: https://issues.jboss.org/browse/MODCLUSTER-425
> Project: mod_cluster
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Affects Versions: 1.2.6.Final, 1.2.9.Final
> Environment: Confirmed RHEL6 x86_64
> Reporter: Michal Babacek
> Assignee: Michal Babacek
>
> mod_cluster balancer used in front of GateIn Portal, after certain warm-up time, starts to show:
> {code}
> [debug] proxy_util.c(2200): proxy: connected / to perf13:8009
> [debug] mod_proxy_cluster.c(1384): proxy_cluster_try_pingpong: connected to backend
> [debug] mod_proxy_cluster.c(1108): ajp_cping_cpong: Done
> [debug] proxy_util.c(2036): proxy: ajp: has released connection for (perf13)
> [debug] proxy_util.c(2018): proxy: ajp: has acquired connection for (perf12)
> [debug] proxy_util.c(2074): proxy: connecting ajp://perf12:8009/ to perf12:8009
> [debug] proxy_util.c(2200): proxy: connected / to perf12:8009
> [debug] proxy_util.c(2451): proxy: ajp: fam 2 socket created to connect to perf12
> [debug] mod_proxy_cluster.c(1384): proxy_cluster_try_pingpong: connected to backend
> [debug] mod_proxy_cluster.c(1108): ajp_cping_cpong: Done
> [debug] proxy_util.c(2036): proxy: ajp: has released connection for (perf12)
> [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header
> [error] ajp_handle_cping_cpong: ajp_ilink_receive failed
> [error] (70007)The timeout specified has expired: proxy: AJP: cping/cpong failed to 10.16.88.190:8009 (perf12)
> [debug] proxy_util.c(2018): proxy: ajp: has acquired connection for (perf12)
> [debug] proxy_util.c(2074): proxy: connecting ajp://perf12:8009/ to perf12:8009
> [debug] proxy_util.c(2200): proxy: connected / to perf12:8009
> [debug] mod_proxy_cluster.c(1384): proxy_cluster_try_pingpong: connected to backend
> [debug] mod_proxy_cluster.c(1108): ajp_cping_cpong: Done
> [debug] proxy_util.c(2036): proxy: ajp: has released connection for (perf12)
> [debug] proxy_util.c(2018): proxy: ajp: has acquired connection for (perf13)
> [debug] proxy_util.c(2074): proxy: connecting ajp://perf13:8009/ to perf13:8009
> [debug] proxy_util.c(2200): proxy: connected / to perf13:8009
> [debug] mod_proxy_cluster.c(1384): proxy_cluster_try_pingpong: connected to backend
> [debug] mod_proxy_cluster.c(1108): ajp_cping_cpong: Done
> [debug] proxy_util.c(2036): proxy: ajp: has released connection for (perf13)
> [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header
> [error] ajp_handle_cping_cpong: ajp_ilink_receive failed
> [error] (70007)The timeout specified has expired: proxy: AJP: cping/cpong failed to 10.16.88.190:8009 (perf12)
> [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header
> [error] ajp_handle_cping_cpong: ajp_ilink_receive failed
> [error] (70007)The timeout specified has expired: proxy: AJP: cping/cpong failed to 10.16.88.190:8009 (perf12)
> [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header
> [error] ajp_handle_cping_cpong: ajp_ilink_receive failed
> [error] (70007)The timeout specified has expired: proxy: AJP: cping/cpong failed to 10.16.88.190:8009 (perf12)
> [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header
> [error] ajp_handle_cping_cpong: ajp_ilink_receive failed
> [error] (70007)The timeout specified has expired: proxy: AJP: cping/cpong failed to 10.16.88.190:8009 (perf12)
> [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header
> [error] ajp_handle_cping_cpong: ajp_ilink_receive failed
> [error] (70007)The timeout specified has expired: proxy: AJP: cping/cpong failed to 10.16.88.190:8009 (perf12)
> [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header
> {code}
> Investigation is going on...
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
10 years, 4 months