[JBoss JIRA] (MODCLUSTER-391) mod_cluster and mod_proxy integration
by Jean-Frederic Clere (JIRA)
[ https://issues.jboss.org/browse/MODCLUSTER-391?page=com.atlassian.jira.pl... ]
Jean-Frederic Clere updated MODCLUSTER-391:
-------------------------------------------
Fix Version/s: 1.2.10.Final
(was: 1.2.9.Final)
> mod_cluster and mod_proxy integration
> -------------------------------------
>
> Key: MODCLUSTER-391
> URL: https://issues.jboss.org/browse/MODCLUSTER-391
> Project: mod_cluster
> Issue Type: Bug
> Affects Versions: 1.2.6.Final
> Environment: All platforms we build mod_cluster for.
> Reporter: Michal Babacek
> Assignee: Jean-Frederic Clere
> Labels: native_libraries
> Fix For: 1.3.1.Final, 1.2.10.Final
>
> Attachments: error_log, mod_cluster.conf, mod_proxy.conf, standalone-ha.xml
>
>
> This Jira encapsulates all concerns regarding mod_cluster - mod_proxy integration. For instance, while basic {{ProxyPass}} settings work just fine, e.g. serving some files on {{/static}} from the Apache HTTP itself:
> {code}
> ProxyPassMatch ^/static/ !
> ProxyPass / balancer://qacluster stickysession=JSESSIONID|jsessionid nofailover=on
> ProxyPassReverse / balancer://qacluster
> ProxyPreserveHost on
> {code}
> there are more complex setups, involving {{BalancerMember}} configurations, that do not work as expected. In the following example, one wanted to have {{/clusterbench}} application managed by mod_cluster, dynamically, while at the same time, in a different VirtualHost, having {{/tses}} application handled by manually created mod_proxy balancer settings.
> Attached [^mod_cluster.conf], [^mod_proxy.conf], [^standalone-ha.xml](modcluster subsystem element only) and [^error_log].
> The aforementioned setup resulted in:
> |HTTP 200|(From worker)|http://10.16.88.19:8847/clusterbench/requestinfo/|OK|(/)|
> |HTTP 404|(From httpd)|http://10.16.88.19:8847/tses/session.jsp|Expected fail|(/)|
> |HTTP 503|(From httpd)|http://10.16.88.19:2182/tses/session.jsp|Unexpected fail|(x)|
--
This message was sent by Atlassian JIRA
(v6.2.3#6260)
9 years, 11 months
[JBoss JIRA] (MODCLUSTER-402) Balancer shutdown, AS:OutOfMemoryError: unable to create new native thread
by Jean-Frederic Clere (JIRA)
[ https://issues.jboss.org/browse/MODCLUSTER-402?page=com.atlassian.jira.pl... ]
Jean-Frederic Clere updated MODCLUSTER-402:
-------------------------------------------
Fix Version/s: 1.2.10.Final
(was: 1.2.9.Final)
> Balancer shutdown, AS:OutOfMemoryError: unable to create new native thread
> --------------------------------------------------------------------------
>
> Key: MODCLUSTER-402
> URL: https://issues.jboss.org/browse/MODCLUSTER-402
> Project: mod_cluster
> Issue Type: Bug
> Affects Versions: 1.2.8.Final
> Environment: HP-UX ia64, HP JDK 1.7
> Reporter: Michal Babacek
> Assignee: Michal Babacek
> Priority: Critical
> Fix For: 1.2.10.Final
>
> Attachments: OOM-server.logs.zip
>
>
> # start Apache HTTP Server with mod_cluster
> # start EAP 6.3 worker 1, jvmroute: jboss-eap-6.3
> # start EAP 6.3 worker 2, jvmroute: jboss-eap-6.3-2
> # stop 1.
> # let 2. and 3. trying to contact the balancer
> # OOM
> See two detailed logs for both machines: [^OOM-server.logs.zip].
> At the moment, I'm not sure it was really mod_cluster's fault, due to the JGroups messages. I have a hunch it might be linked to the new JBossWeb NIO...
> Assigning to myself for investigation.
--
This message was sent by Atlassian JIRA
(v6.2.3#6260)
9 years, 11 months
[JBoss JIRA] (MODCLUSTER-401) EnableOptions and SSL configuration
by Jean-Frederic Clere (JIRA)
[ https://issues.jboss.org/browse/MODCLUSTER-401?page=com.atlassian.jira.pl... ]
Jean-Frederic Clere updated MODCLUSTER-401:
-------------------------------------------
Fix Version/s: 1.2.10.Final
(was: 1.2.9.Final)
> EnableOptions and SSL configuration
> -----------------------------------
>
> Key: MODCLUSTER-401
> URL: https://issues.jboss.org/browse/MODCLUSTER-401
> Project: mod_cluster
> Issue Type: Bug
> Affects Versions: 1.2.8.Final
> Environment: HP-UX Apache HTTP Server 2.2.15, RHEL Apache HTTP Server 2.2.22, perhaps platform independent...
> Reporter: Michal Babacek
> Assignee: Jean-Frederic Clere
> Fix For: 1.2.10.Final
>
>
> As a follow up on MODCLUSTER-400 and a documentation effort for *EnableOptions* logic, I tried to add {{EnableOptions}} to the configuration so as to allow for a "cping/cpong" emulation of the famous AJP feature.
> With the following {{mod_cluster.conf / httpd.conf}} (standalone-ha.xml being the same as in MODCLUSTER-400's description):
> {code}
> +++
> Listen 10.16.92.191:2081
> +++
> MemManagerFile "/hell/workspace/hpws22/apache/cache/mod_cluster"
> ServerName 10.16.92.191:2081
> <IfModule manager_module>
> Listen 10.16.92.191:8745
> LogLevel debug
> <VirtualHost 10.16.92.191:8745>
> ServerName 10.16.92.191:8745
> <Directory />
> Order deny,allow
> Deny from all
> Allow from all
> </Directory>
> KeepAliveTimeout 60
> MaxKeepAliveRequests 0
> ServerAdvertise on
> AdvertiseFrequency 5
> ManagerBalancerName qacluster
> AdvertiseGroup 224.0.3.47:23364
> EnableOptions
> EnableMCPMReceive
> SSLEngine on
> SSLProtocol all -SSLv2 -SSLv3
> SSLCipherSuite "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS"
> SSLHonorCipherOrder on
> SSLCertificateFile /vault/server.crt
> SSLCertificateKeyFile /vault/server.key
> SSLCACertificateFile /vault/myca.crt
> SSLProxyEngine On
> SSLVerifyDepth 10
> <Location /mcm>
> SetHandler mod_cluster-manager
> Order deny,allow
> Deny from all
> Allow from all
> </Location>
> </VirtualHost>
> </IfModule>
> {code}
> one gets this [^hp-ux_error_log-EnableOptions.zip] log:
> {code}
> [debug] mod_proxy_cluster.c(1223): http_cping_cpong: received HTTP/1.1 200 OK
> [debug] mod_proxy_cluster.c(1223): http_cping_cpong: received Server: Apache-Coyote/1.1
> [debug] mod_proxy_cluster.c(1223): http_cping_cpong: received Allow: GET, HEAD, POST, PUT, DELETE, TRACE, OPTIONS
> [debug] mod_proxy_cluster.c(1223): http_cping_cpong: received Content-Length: 0
> [debug] mod_proxy_cluster.c(1223): http_cping_cpong: received Date: Fri, 02 May 2014 17:22:46 GMT
> [debug] mod_proxy_cluster.c(1223): http_cping_cpong: received Connection: close
> [debug] mod_proxy_cluster.c(1239): http_cping_cpong: Done
> [debug] proxy_util.c(2047): proxy: https: has released connection for (10.16.92.191)
> [debug] mod_manager.c(2666): manager_handler STATUS OK
> [debug] proxy_util.c(2029): proxy: https: has acquired connection for (10.16.92.191)
> [debug] proxy_util.c(2085): proxy: connecting https://10.16.92.191:8645/ to 10.16.92.191:8645
> [debug] proxy_util.c(2211): proxy: connected / to 10.16.92.191:8645
> [debug] proxy_util.c(2462): proxy: https: fam 2 socket created to connect to 10.16.92.191
> [debug] mod_proxy_cluster.c(1384): proxy_cluster_try_pingpong: connected to backend
> [error] [client 10.16.92.191] SSL Proxy requested for 10.16.92.191:2081 but not enabled [Hint: SSLProxyEngine]
> [error] proxy: https: failed to enable ssl support for 10.16.92.191:8645 (10.16.92.191)
> [debug] proxy_util.c(2047): proxy: https: has released connection for (10.16.92.191)
> {code}
> Why is the JBoss EAP residing on {{10.16.92.191:8645}} trying to request SSL Proxy on the virtual host {{10.16.92.191:2081}}? The result is {{Status: NOTOK}} on mod_cluser manager console.
> I tried to remove that {{10.16.92.191:2081}}, so as the {{10.16.92.191:8745}} is the only one ([^hp-ux_error_log-EnableOptions-single-vhost.zip]):
> {code}
> - Listen 10.16.92.191:2081
> - ServerName 10.16.92.191:2081
> {code}
> The result is a funny trial to request a proxy for the boxe's actual hostname and port 80 *no one* (netstat) is even listening on:
> {code}
> [debug] mod_proxy_cluster.c(1223): http_cping_cpong: received HTTP/1.1 200 OK
> [debug] mod_proxy_cluster.c(1223): http_cping_cpong: received Server: Apache-Coyote/1.1
> [debug] mod_proxy_cluster.c(1223): http_cping_cpong: received Allow: GET, HEAD, POST, PUT, DELETE, TRACE, OPTIONS
> [debug] mod_proxy_cluster.c(1223): http_cping_cpong: received Content-Length: 0
> [debug] mod_proxy_cluster.c(1223): http_cping_cpong: received Date: Fri, 02 May 2014 17:39:33 GMT
> [debug] mod_proxy_cluster.c(1223): http_cping_cpong: received Connection: close
> [debug] mod_proxy_cluster.c(1239): http_cping_cpong: Done
> [debug] proxy_util.c(2047): proxy: https: has released connection for (10.16.92.191)
> [debug] mod_manager.c(2666): manager_handler STATUS OK
> [debug] proxy_util.c(2029): proxy: https: has acquired connection for (10.16.92.191)
> [debug] proxy_util.c(2085): proxy: connecting https://10.16.92.191:8645/ to 10.16.92.191:8645
> [debug] proxy_util.c(2211): proxy: connected / to 10.16.92.191:8645
> [debug] proxy_util.c(2462): proxy: https: fam 2 socket created to connect to 10.16.92.191
> [debug] mod_proxy_cluster.c(1384): proxy_cluster_try_pingpong: connected to backend
> [error] [client 10.16.92.191] SSL Proxy requested for eap-perf-hpux-03.mw.lab.eng.bos.redhat.com:80 but not enabled [Hint: SSLProxyEngine]
> [error] proxy: https: failed to enable ssl support for 10.16.92.191:8645 (10.16.92.191)
> [debug] proxy_util.c(2047): proxy: https: has released connection for (10.16.92.191)
> {code}
> I tried to add: {{RequestHeader set Front-End-Https "On"}} to the configuration without any luck.
> Finally, I replicated the SSL configuration *outside* the VirtualHost:
> {code}
> MemManagerFile "/hell/workspace/hpws22/apache/cache/mod_cluster"
> Listen 10.16.92.191:2081
> ServerName 10.16.92.191:2081
> SSLEngine on
> SSLProtocol all -SSLv2 -SSLv3
> SSLCipherSuite "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !S RP !DSS"
> SSLHonorCipherOrder on
> SSLCertificateFile /vault/server.crt
> SSLCertificateKeyFile /vault/server.key
> SSLCACertificateFile /vault/myca.crt
> SSLProxyEngine On
> SSLVerifyDepth 10
> <IfModule manager_module>
> +++ the same as above +++
> </IfModule>
> {code}
> This configuration fixed the aforementioned {{failed to enable ssl support}} *and* actually helped to workaround the MODCLUSTER-400: (log: [^hp-ux_error_log-EnableOptions-SSL_everywhere.zip])
> {code}
> Fri, May 2, 2014 02:23:44 PM Request URI: /clusterbench/requestinfo
> Headers: {host=10.16.92.191:8645, user-agent=curl/7.30.0, accept=*/*, cookie=JSESSIONID=2hC9ax9LGYDvQZtH0RXdBimf.jboss-eap-6.3-2, x-forwarded-for=10.16.92.191, x-forwarded-host=10.16.92.191:8745, x-forwarded-server=10.16.92.191, connection=Keep-Alive}
> Host header: 10.16.92.191:8645
> Character encoding: null
> JVM route: jboss-eap-6.3-2
> Session ID: 2hC9ax9LGYDvQZtH0RXdBimf.jboss-eap-6.3-2
> Session isNew: false
> Fri, May 2, 2014 02:23:47 PM Request URI: /clusterbench/requestinfo
> Headers: {host=10.16.92.191:8645, user-agent=curl/7.30.0, accept=*/*, cookie=JSESSIONID=2hC9ax9LGYDvQZtH0RXdBimf.jboss-eap-6.3-2, x-forwarded-for=10.16.92.191, x-forwarded-host=10.16.92.191:8745, x-forwarded-server=10.16.92.191, connection=Keep-Alive}
> Host header: 10.16.92.191:8645
> Character encoding: null
> JVM route: jboss-eap-6.3-2
> Session ID: 2hC9ax9LGYDvQZtH0RXdBimf.jboss-eap-6.3-2
> Session isNew: false
> -- stop jboss-eap-6.3-2 -- (the same behavior with jvm kill) --
> Fri, May 2, 2014 02:23:50 PM Request URI: /clusterbench/requestinfo
> Headers: {host=10.16.92.191:8544, user-agent=curl/7.30.0, accept=*/*, cookie=JSESSIONID=2hC9ax9LGYDvQZtH0RXdBimf.jboss-eap-6.3-2, x-forwarded-for=10.16.92.191, x-forwarded-host=10.16.92.191:8745, x-forwarded-server=10.16.92.191, connection=Keep-Alive}
> Host header: 10.16.92.191:8544
> Character encoding: null
> JVM route: jboss-eap-6.3
> Session ID: 2hC9ax9LGYDvQZtH0RXdBimf.jboss-eap-6.3
> Session isNew: false
> Fri, May 2, 2014 02:23:53 PM Request URI: /clusterbench/requestinfo
> Headers: {host=10.16.92.191:8544, user-agent=curl/7.30.0, accept=*/*, cookie=JSESSIONID=2hC9ax9LGYDvQZtH0RXdBimf.jboss-eap-6.3, x-forwarded-for=10.16.92.191, x-forwarded-host=10.16.92.191:8745, x-forwarded-server=10.16.92.191, connection=Keep-Alive}
> Host header: 10.16.92.191:8544
> Character encoding: null
> JVM route: jboss-eap-6.3
> Session ID: 2hC9ax9LGYDvQZtH0RXdBimf.jboss-eap-6.3
> Session isNew: false
> Fri, May 2, 2014 02:23:56 PM Request URI: /clusterbench/requestinfo
> Headers: {host=10.16.92.191:8544, user-agent=curl/7.30.0, accept=*/*, cookie=JSESSIONID=2hC9ax9LGYDvQZtH0RXdBimf.jboss-eap-6.3, x-forwarded-for=10.16.92.191, x-forwarded-host=10.16.92.191:8745, x-forwarded-server=10.16.92.191, connection=Keep-Alive}
> Host header: 10.16.92.191:8544
> Character encoding: null
> JVM route: jboss-eap-6.3
> Session ID: 2hC9ax9LGYDvQZtH0RXdBimf.jboss-eap-6.3
> Session isNew: false
> {code}
> Why isn't the {{10.16.92.191:8745}} enough? Is it a configuration error or a ProxyPass/SSL integration bug?
--
This message was sent by Atlassian JIRA
(v6.2.3#6260)
9 years, 11 months
[JBoss JIRA] (MODCLUSTER-400) Failover with SSL breaks sticky sessions on HP-UX v11.3, hpws22
by Jean-Frederic Clere (JIRA)
[ https://issues.jboss.org/browse/MODCLUSTER-400?page=com.atlassian.jira.pl... ]
Jean-Frederic Clere updated MODCLUSTER-400:
-------------------------------------------
Fix Version/s: 1.2.10.Final
(was: 1.2.9.Final)
> Failover with SSL breaks sticky sessions on HP-UX v11.3, hpws22
> ---------------------------------------------------------------
>
> Key: MODCLUSTER-400
> URL: https://issues.jboss.org/browse/MODCLUSTER-400
> Project: mod_cluster
> Issue Type: Bug
> Affects Versions: 1.2.8.Final
> Environment: HP-UX v11.3, hpuxws22Apache B.2.2.15.15 HP-UX Apache-based Web Server, libaprutil-1.sl.3.9, libapr-1.sl.4.2
> Reporter: Michal Babacek
> Assignee: Jean-Frederic Clere
> Priority: Critical
> Fix For: 1.2.10.Final
>
> Attachments: hp-ux_error_log-ajp-failover.zip, hp-ux_error_log-ssl-failover.zip, rhel_error_log-ssl-failover.zip
>
>
> Failover with SSL breaks sticky sessions on HP-UX v11.3, hpws22
> With the following configuration on a single box:
> {panel:title=mod_cluster.conf| borderStyle=dashed}
> {code}
> MemManagerFile "/hell/workspace/hpws22/apache/cache/mod_cluster"
> ServerName 10.16.92.191:2081
> <IfModule manager_module>
> Listen 10.16.92.191:8745
> LogLevel debug
> <VirtualHost 10.16.92.191:8745>
> ServerName 10.16.92.191:8745
> <Directory />
> Order deny,allow
> Deny from all
> Allow from all
> </Directory>
> KeepAliveTimeout 60
> MaxKeepAliveRequests 0
> ServerAdvertise on
> AdvertiseFrequency 5
> ManagerBalancerName qacluster
> AdvertiseGroup 224.0.3.47:23364
> EnableMCPMReceive
> SSLEngine on
> SSLProtocol all -SSLv2 -SSLv3
> SSLCipherSuite "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS"
> SSLHonorCipherOrder on
> SSLCertificateFile /vault/certs/server.crt
> SSLCertificateKeyFile /vault/certs/server.key
> SSLCACertificateFile /vault/certs/myca.crt
> SSLProxyEngine On
> SSLVerifyDepth 10
> <Location /mcm>
> SetHandler mod_cluster-manager
> Order deny,allow
> Deny from all
> Allow from all
> </Location>
> </VirtualHost>
> </IfModule>
> {code}
> {panel}
> {panel:title=standalone-ha.xml| borderStyle=dashed}
> {code}
> +++
> <subsystem xmlns="urn:jboss:domain:web:2.1" native="false">
> <connector name="https" protocol="HTTP/1.1" socket-binding="https" scheme="https" enabled="true" secure="true">
> <ssl name="https" ca-certificate-file="/vault/certs/ca-cert.jks" certificate-key-file="/vault/certs/client-cert-key.jks" certificate-file="/vault/certs/client-cert-key.jks" password="tomcat" verify-client="false" key-alias="javaclient"
> cipher-suite="SSL_RSA_WITH_DES_CBC_SHA,SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA,SSL_RSA_EXPORT_WITH_RC4_40_MD5,SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA,SSL_RSA_WITH_RC4_128_SHA,SSL_RSA_WITH_NULL_MD5,SSL_DHE_RSA_WITH_DES_CBC_SHA,SSL_DH_anon_EXPORT_WITH_RC4_40_MD5,SSL_DH_anon_WITH_3DES_EDE_CBC_SHA,SSL_DH_anon_WITH_RC4_128_MD5,SSL_RSA_WITH_RC4_128_MD5,SSL_DHE_DSS_WITH_DES_CBC_SHA,SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA,SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA,SSL_DH_anon_WITH_DES_CBC_SHA,SSL_RSA_WITH_NULL_SHA,SSL_RSA_EXPORT_WITH_DES40_CBC_SHA" protocol="TLS"/>
> </connector>
> <connector name="ajp" protocol="AJP/1.3" scheme="http" socket-binding="ajp"/>
> <connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http"/>
> <virtual-server name="default-host" enable-welcome-root="true">
> <alias name="localhost"/>
> <alias name="example.com"/>
> </virtual-server>
> </subsystem>
> +++
> <subsystem xmlns="urn:jboss:domain:modcluster:1.2">
> <mod-cluster-config connector="https" advertise-socket="modcluster">
> <dynamic-load-provider>
> <load-metric type="busyness"/>
> </dynamic-load-provider>
> <ssl ca-certificate-file="/vault/certs/ca-cert.jks" certificate-key-file="/vault/certs/client-cert-key.jks" password="tomcat" key-alias="javaclient"
> cipher-suite="SSL_RSA_WITH_DES_CBC_SHA,SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA,SSL_RSA_EXPORT_WITH_RC4_40_MD5,SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA,SSL_RSA_WITH_RC4_128_SHA,SSL_RSA_WITH_NULL_MD5,SSL_DHE_RSA_WITH_DES_CBC_SHA,SSL_DH_anon_EXPORT_WITH_RC4_40_MD5,SSL_DH_anon_WITH_3DES_EDE_CBC_SHA,SSL_DH_anon_WITH_RC4_128_MD5,SSL_RSA_WITH_RC4_128_MD5,SSL_DHE_DSS_WITH_DES_CBC_SHA,SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA,SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA,SSL_DH_anon_WITH_DES_CBC_SHA,SSL_RSA_WITH_NULL_SHA,SSL_RSA_EXPORT_WITH_DES40_CBC_SHA" protocol="TLS"/>
> </mod-cluster-config>
> </subsystem>
> +++
> {code}
> {panel}
> One gets the following weird session loss. This log [^hp-ux_error_log-ssl-failover.zip] covers the undermentioned test servlet output:
> {code}
> echo -e "`date` `curl https://10.16.92.191:8745/clusterbench/requestinfo --cert /vault/certs/client.crt --key /vault/certs/client.key --cacert /vault/certs/myca.crt --insecure -c cookiefile.txt -b cookiefile.txt 2> /dev/null`";
> {code}
> {code}
> Wed, Apr 30, 2014 11:50:11 AM Request URI: /clusterbench/requestinfo
> Headers: {host=10.16.92.191:8544, user-agent=curl/7.30.0, accept=*/*, cookie=JSESSIONID=vjZSMs4fZ8j0h+VIQ-GLAz+F.jboss-eap-6.3, x-forwarded-for=10.16.92.191, x-forwarded-host=10.16.92.191:8745, x-forwarded-server=10.16.92.191, connection=Keep-Alive}
> Host header: 10.16.92.191:8544
> JVM route: jboss-eap-6.3
> Session ID: vjZSMs4fZ8j0h+VIQ-GLAz+F.jboss-eap-6.3
> Session isNew: false
> Wed, Apr 30, 2014 11:50:13 AM Request URI: /clusterbench/requestinfo
> Headers: {host=10.16.92.191:8544, user-agent=curl/7.30.0, accept=*/*, cookie=JSESSIONID=vjZSMs4fZ8j0h+VIQ-GLAz+F.jboss-eap-6.3, x-forwarded-for=10.16.92.191, x-forwarded-host=10.16.92.191:8745, x-forwarded-server=10.16.92.191, connection=Keep-Alive}
> Host header: 10.16.92.191:8544
> JVM route: jboss-eap-6.3
> Session ID: vjZSMs4fZ8j0h+VIQ-GLAz+F.jboss-eap-6.3
> Session isNew: false
> -- stop jboss-eap-6.3 -- (the same behavior with jvm kill) --
> Wed, Apr 30, 2014 11:50:18 AM Request URI: /clusterbench/requestinfo
> Headers: {host=10.16.92.191:8645, user-agent=curl/7.30.0, accept=*/*, cookie=JSESSIONID=vjZSMs4fZ8j0h+VIQ-GLAz+F.jboss-eap-6.3, x-forwarded-for=10.16.92.191, x-forwarded-host=10.16.92.191:8745, x-forwarded-server=10.16.92.191, connection=Keep-Alive}
> Host header: 10.16.92.191:8645
> JVM route: jboss-eap-6.3-2
> Session ID: tYnoHJhX73UYrr3QCaUikR9h.jboss-eap-6.3-2
> Session isNew: true
> Wed, Apr 30, 2014 11:50:20 AM Request URI: /clusterbench/requestinfo
> Headers: {host=10.16.92.191:8645, user-agent=curl/7.30.0, accept=*/*, cookie=JSESSIONID=tYnoHJhX73UYrr3QCaUikR9h.jboss-eap-6.3-2, x-forwarded-for=10.16.92.191, x-forwarded-host=10.16.92.191:8745, x-forwarded-server=10.16.92.191, connection=Keep-Alive}
> Host header: 10.16.92.191:8645
> JVM route: jboss-eap-6.3-2
> Session ID: tYnoHJhX73UYrr3QCaUikR9h.jboss-eap-6.3-2
> Session isNew: false
> {code}
> One could note that in the moment of fail-over from worker {{jboss-eap-6.3}} to worker {{jboss-eap-6.3-2}}, the original session {{vjZSMs4fZ8j0h+VIQ-GLAz+F}} had been lost and a new one, {{tYnoHJhX73UYrr3QCaUikR9h}} was created.
> If we comment out all the SSL settings and switch to the AJP connector, the failover seems all right though (see the [^hp-ux_error_log-ajp-failover.zip]):
> {code}
> Wed, Apr 30, 2014 12:04:11 PM Request URI: /clusterbench/requestinfo
> Headers: {user-agent=curl/7.30.0, host=10.16.92.191:8745, accept=*/*, cookie=JSESSIONID=v-hPRQD5FsZUAqZa3ZxRBXIF.jboss-eap-6.3}
> Host header: 10.16.92.191:8745
> JVM route: jboss-eap-6.3
> Session ID: v-hPRQD5FsZUAqZa3ZxRBXIF.jboss-eap-6.3
> Session isNew: false
> -- stop jboss-eap-6.3 -- (the same behavior with jvm kill) --
> Wed, Apr 30, 2014 12:04:14 PM Request URI: /clusterbench/requestinfo
> Headers: {user-agent=curl/7.30.0, host=10.16.92.191:8745, accept=*/*, cookie=JSESSIONID=v-hPRQD5FsZUAqZa3ZxRBXIF.jboss-eap-6.3}
> Host header: 10.16.92.191:8745
> JVM route: jboss-eap-6.3-2
> Session ID: v-hPRQD5FsZUAqZa3ZxRBXIF.jboss-eap-6.3-2
> Session isNew: false
> {code}
> Note that the session {{hPRQD5FsZUAqZa3ZxRBXIF}} remained the same during the failover, only a new jvmRoute were appended.
> The most bewildering thing is that this behavior is specific to {{hpuxws22Apache B.2.2.15.15 HP-UX Apache-based Web Server}},
> i.e., I've carefully followed the same scenario with the same config on RHEL 6 x86_64, Apache/2.2.22 and the session is kept both with AJP and HTTPS connectors (see [^rhel_error_log-ssl-failover.zip]).
> {code}
> Fri May 2 09:47:13 EDT 2014 Request URI: /clusterbench/requestinfo
> Headers: {host=192.168.122.204:8443, user-agent=curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.3.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2, accept=*/*, cookie=JSESSIONID=wdeSdhbahzVweKc5F9mUprJr.jboss-eap-6.3, x-forwarded-for=192.168.122.204, x-forwarded-host=192.168.122.204:8847, x-forwarded-server=192.168.122.204, connection=Keep-Alive}
> Host header: 192.168.122.204:8443
> JVM route: jboss-eap-6.3
> Session ID: wdeSdhbahzVweKc5F9mUprJr.jboss-eap-6.3
> Session isNew: false
> -- stop jboss-eap-6.3 -- (the same behavior with jvm kill) --
> Fri May 2 09:47:16 EDT 2014 Request URI: /clusterbench/requestinfo
> Headers: {host=192.168.122.204:8544, user-agent=curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.3.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2, accept=*/*, cookie=JSESSIONID=wdeSdhbahzVweKc5F9mUprJr.jboss-eap-6.3, x-forwarded-for=192.168.122.204, x-forwarded-host=192.168.122.204:8847, x-forwarded-server=192.168.122.204, connection=Keep-Alive}
> Host header: 192.168.122.204:8544
> JVM route: jboss-eap-6.3-2
> Session ID: wdeSdhbahzVweKc5F9mUprJr.jboss-eap-6.3-2
> Session isNew: false
> {code}
> To date, I don't have any more info to share.
--
This message was sent by Atlassian JIRA
(v6.2.3#6260)
9 years, 11 months
[JBoss JIRA] (MODCLUSTER-398) mod_cluster deadlock in a jboss/windows environment
by Jean-Frederic Clere (JIRA)
[ https://issues.jboss.org/browse/MODCLUSTER-398?page=com.atlassian.jira.pl... ]
Jean-Frederic Clere resolved MODCLUSTER-398.
--------------------------------------------
Fix Version/s: 1.3.1.Final
1.2.9.Final
Resolution: Done
BZ 1080047
> mod_cluster deadlock in a jboss/windows environment
> ---------------------------------------------------
>
> Key: MODCLUSTER-398
> URL: https://issues.jboss.org/browse/MODCLUSTER-398
> Project: mod_cluster
> Issue Type: Bug
> Affects Versions: 1.2.6.Final
> Environment: Windows 2008, EAP6 and EWS2.0.1
> Reporter: Marc Maurer
> Assignee: Jean-Frederic Clere
> Fix For: 1.3.1.Final, 1.2.9.Final
>
>
> Under load Apache stops serving pages, with all threads are stuck in "W : Sending reply" state. With the windows Process Explorer we then got a stacktrace from a hanging thread. We don't have debug symbols, but it's easy enough to see what's happening:
> ntoskrnl.exe!KeWaitForMultipleObjects+0xc0a
> ntoskrnl.exe!KeAcquireSpinLockAtDpcLevel+0x732
> ntoskrnl.exe!KeWaitForMutexObject+0x19f
> ntoskrnl.exe!NtDeleteFile+0x3c4
> ntoskrnl.exe!PsDereferenceKernelStack+0x35358
> ntoskrnl.exe!KeSynchronizeExecution+0x3a23
> ntdll.dll!ZwLockFile+0xa
> KERNELBASE.dll!LockFileEx+0xb2
> kernel32.dll!LockFileEx+0x1b
> libapr-1.dll!apr_file_lock+0x69 <-- here
> mod_slotmem.so+0x1318 <-- here
> mod_manager.so+0x2a11 <-- here
> mod_proxy_cluster.so+0x679e
> mod_proxy.so!proxy_run_post_request+0x4e
> mod_proxy.so!proxy_run_request_status+0x924
> libhttpd.dll!ap_run_handler+0x35
> libhttpd.dll!ap_invoke_handler+0x114
> libhttpd.dll!ap_die+0x2ea
> libhttpd.dll!ap_psignature+0x1ae8
> libhttpd.dll!ap_run_process_connection+0x35
> libhttpd.dll!ap_process_connection+0x3b
> libhttpd.dll!ap_regkey_value_remove+0x136e
> msvcrt.dll!srand+0x93
> msvcrt.dll!ftime64_s+0x1dd
> kernel32.dll!BaseThreadInitThunk+0xd
> ntdll.dll!RtlUserThreadStart+0x21
> So mod_manager is requesting a filelock on one of the lockfiles in in the MemManagerFile path. In this case it was the "manager.sessionid.sessionid.lock" file. Removing the lockfile fixed the problem.
> When bisecting the mod_cluster code, I think commit "74eeb9c026380deb8d833be53b09b3d808e02d10 - Lock in insert-update" in version 1.2.2 is the culprit. This would also explain why mod_cluster 1.2.1 is the last known working version.
> What we don't know, is which process is already holding the lock when all Apache threads start blocking on it. We are trying to figure that out. There are no obviously wrong lock/unlock slotmem call pairs in the mod_manager module, and no locks are requested within other locks as far as we can see. Therefor our best guess would be a deadlock on a thread already holding the globalmutex_lock in combination with the slotmem file locks, but that's just a guess without debugging it.
> More context can be found here: https://bugzilla.redhat.com/show_bug.cgi?id=1080047
--
This message was sent by Atlassian JIRA
(v6.2.3#6260)
9 years, 11 months
[JBoss JIRA] (MODCLUSTER-404) ModClusterService stop commands are always draining sessions
by Jean-Frederic Clere (JIRA)
[ https://issues.jboss.org/browse/MODCLUSTER-404?page=com.atlassian.jira.pl... ]
Jean-Frederic Clere updated MODCLUSTER-404:
-------------------------------------------
Status: Resolved (was: Pull Request Sent)
Fix Version/s: 1.3.1.Final
1.2.9.Final
Resolution: Done
> ModClusterService stop commands are always draining sessions
> ------------------------------------------------------------
>
> Key: MODCLUSTER-404
> URL: https://issues.jboss.org/browse/MODCLUSTER-404
> Project: mod_cluster
> Issue Type: Bug
> Affects Versions: 1.3.0.Final, 1.2.8.Final
> Reporter: Aaron Ogburn
> Assignee: Jean-Frederic Clere
> Fix For: 1.3.1.Final, 1.2.9.Final
>
>
> Using the ModClusterService stop or stopContext commands via CLI always fails to move a context to the STOPPED state after failing to drain the active sessions. So these commands then aren't very useful for quickly stopping the context when desired if you can't do it without draining.
> I think it'd make sense for any draining done by manual stops to also follow the session draining strategy used by typical stops from undeploy events.
--
This message was sent by Atlassian JIRA
(v6.2.3#6260)
9 years, 11 months
[JBoss JIRA] (MODCLUSTER-402) Balancer shutdown, AS:OutOfMemoryError: unable to create new native thread
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/MODCLUSTER-402?page=com.atlassian.jira.pl... ]
Radoslav Husar edited comment on MODCLUSTER-402 at 5/22/14 9:34 AM:
--------------------------------------------------------------------
Looking at the description, it dosn't say about the session access patern. Can you elaborate a little bit?
was (Author: rhusar):
Looking at the description, are there no sessions being created?
> Balancer shutdown, AS:OutOfMemoryError: unable to create new native thread
> --------------------------------------------------------------------------
>
> Key: MODCLUSTER-402
> URL: https://issues.jboss.org/browse/MODCLUSTER-402
> Project: mod_cluster
> Issue Type: Bug
> Affects Versions: 1.2.8.Final
> Environment: HP-UX ia64, HP JDK 1.7
> Reporter: Michal Babacek
> Assignee: Michal Babacek
> Priority: Critical
> Fix For: 1.2.9.Final
>
> Attachments: OOM-server.logs.zip
>
>
> # start Apache HTTP Server with mod_cluster
> # start EAP 6.3 worker 1, jvmroute: jboss-eap-6.3
> # start EAP 6.3 worker 2, jvmroute: jboss-eap-6.3-2
> # stop 1.
> # let 2. and 3. trying to contact the balancer
> # OOM
> See two detailed logs for both machines: [^OOM-server.logs.zip].
> At the moment, I'm not sure it was really mod_cluster's fault, due to the JGroups messages. I have a hunch it might be linked to the new JBossWeb NIO...
> Assigning to myself for investigation.
--
This message was sent by Atlassian JIRA
(v6.2.3#6260)
9 years, 11 months
[JBoss JIRA] (MODCLUSTER-407) worker-timeout can cause httpd thread stalls
by Jean-Frederic Clere (JIRA)
[ https://issues.jboss.org/browse/MODCLUSTER-407?page=com.atlassian.jira.pl... ]
Jean-Frederic Clere updated MODCLUSTER-407:
-------------------------------------------
Fix Version/s: 1.3.1.Final
> worker-timeout can cause httpd thread stalls
> --------------------------------------------
>
> Key: MODCLUSTER-407
> URL: https://issues.jboss.org/browse/MODCLUSTER-407
> Project: mod_cluster
> Issue Type: Bug
> Affects Versions: 1.2.8.Final
> Reporter: Aaron Ogburn
> Assignee: Jean-Frederic Clere
> Fix For: 1.3.1.Final, 1.2.9.Final
>
>
> Setting a modcluster worker-timeout can stall requests and threads on the httpd side when the requests are received with workers in a down state. A stack of the problem thread looks like the following (recursive loops through mod_proxy_cluster from #160 to #2):
> #0 0x00007ff8eb547533 in select () from /lib64/libc.so.6
> #1 0x00007ff8eba39185 in apr_sleep () from /usr/lib64/libapr-1.so.0
> #2 0x00007ff8e84be0d1 in ?? () from /etc/httpd/modules/mod_proxy_cluster.so
> ...
> #160 0x00007ff8e84beb9f in ?? () from /etc/httpd/modules/mod_proxy_cluster.so
> #161 0x00007ff8e88d2116 in proxy_run_pre_request () from /etc/httpd/modules/mod_proxy.so
> #162 0x00007ff8e88d9186 in ap_proxy_pre_request () from /etc/httpd/modules/mod_proxy.so
> #163 0x00007ff8e88d63c2 in ?? () from /etc/httpd/modules/mod_proxy.so
--
This message was sent by Atlassian JIRA
(v6.2.3#6260)
9 years, 11 months
[JBoss JIRA] (MODCLUSTER-407) worker-timeout can cause httpd thread stalls
by Jean-Frederic Clere (JIRA)
[ https://issues.jboss.org/browse/MODCLUSTER-407?page=com.atlassian.jira.pl... ]
Jean-Frederic Clere updated MODCLUSTER-407:
-------------------------------------------
Status: Resolved (was: Pull Request Sent)
Fix Version/s: 1.2.9.Final
Resolution: Done
> worker-timeout can cause httpd thread stalls
> --------------------------------------------
>
> Key: MODCLUSTER-407
> URL: https://issues.jboss.org/browse/MODCLUSTER-407
> Project: mod_cluster
> Issue Type: Bug
> Affects Versions: 1.2.8.Final
> Reporter: Aaron Ogburn
> Assignee: Jean-Frederic Clere
> Fix For: 1.2.9.Final
>
>
> Setting a modcluster worker-timeout can stall requests and threads on the httpd side when the requests are received with workers in a down state. A stack of the problem thread looks like the following (recursive loops through mod_proxy_cluster from #160 to #2):
> #0 0x00007ff8eb547533 in select () from /lib64/libc.so.6
> #1 0x00007ff8eba39185 in apr_sleep () from /usr/lib64/libapr-1.so.0
> #2 0x00007ff8e84be0d1 in ?? () from /etc/httpd/modules/mod_proxy_cluster.so
> ...
> #160 0x00007ff8e84beb9f in ?? () from /etc/httpd/modules/mod_proxy_cluster.so
> #161 0x00007ff8e88d2116 in proxy_run_pre_request () from /etc/httpd/modules/mod_proxy.so
> #162 0x00007ff8e88d9186 in ap_proxy_pre_request () from /etc/httpd/modules/mod_proxy.so
> #163 0x00007ff8e88d63c2 in ?? () from /etc/httpd/modules/mod_proxy.so
--
This message was sent by Atlassian JIRA
(v6.2.3#6260)
9 years, 11 months