April 2014 - mod_cluster-issues - Jboss List Archives

[JBoss JIRA] (MODCLUSTER-398) mod_cluster deadlock in a jboss/windows environment

by RH Bugzilla Integration (JIRA)

[ https://issues.jboss.org/browse/MODCLUSTER-398?page=com.atlassian.jira.pl... ] RH Bugzilla Integration updated MODCLUSTER-398: ----------------------------------------------- Bugzilla Update: Perform Bugzilla References: https://bugzilla.redhat.com/show_bug.cgi?id=1080047 > mod_cluster deadlock in a jboss/windows environment > --------------------------------------------------- > > Key: MODCLUSTER-398 > URL: https://issues.jboss.org/browse/MODCLUSTER-398 > Project: mod_cluster > Issue Type: Bug > Affects Versions: 1.2.6.Final > Environment: Windows 2008, EAP6 and EWS2.0.1 > Reporter: Marc Maurer > Assignee: Jean-Frederic Clere > > Under load Apache stops serving pages, with all threads are stuck in "W : Sending reply" state. With the windows Process Explorer we then got a stacktrace from a hanging thread. We don't have debug symbols, but it's easy enough to see what's happening: > ntoskrnl.exe!KeWaitForMultipleObjects+0xc0a > ntoskrnl.exe!KeAcquireSpinLockAtDpcLevel+0x732 > ntoskrnl.exe!KeWaitForMutexObject+0x19f > ntoskrnl.exe!NtDeleteFile+0x3c4 > ntoskrnl.exe!PsDereferenceKernelStack+0x35358 > ntoskrnl.exe!KeSynchronizeExecution+0x3a23 > ntdll.dll!ZwLockFile+0xa > KERNELBASE.dll!LockFileEx+0xb2 > kernel32.dll!LockFileEx+0x1b > libapr-1.dll!apr_file_lock+0x69 <-- here > mod_slotmem.so+0x1318 <-- here > mod_manager.so+0x2a11 <-- here > mod_proxy_cluster.so+0x679e > mod_proxy.so!proxy_run_post_request+0x4e > mod_proxy.so!proxy_run_request_status+0x924 > libhttpd.dll!ap_run_handler+0x35 > libhttpd.dll!ap_invoke_handler+0x114 > libhttpd.dll!ap_die+0x2ea > libhttpd.dll!ap_psignature+0x1ae8 > libhttpd.dll!ap_run_process_connection+0x35 > libhttpd.dll!ap_process_connection+0x3b > libhttpd.dll!ap_regkey_value_remove+0x136e > msvcrt.dll!srand+0x93 > msvcrt.dll!ftime64_s+0x1dd > kernel32.dll!BaseThreadInitThunk+0xd > ntdll.dll!RtlUserThreadStart+0x21 > So mod_manager is requesting a filelock on one of the lockfiles in in the MemManagerFile path. In this case it was the "manager.sessionid.sessionid.lock" file. Removing the lockfile fixed the problem. > When bisecting the mod_cluster code, I think commit "74eeb9c026380deb8d833be53b09b3d808e02d10 - Lock in insert-update" in version 1.2.2 is the culprit. This would also explain why mod_cluster 1.2.1 is the last known working version. > What we don't know, is which process is already holding the lock when all Apache threads start blocking on it. We are trying to figure that out. There are no obviously wrong lock/unlock slotmem call pairs in the mod_manager module, and no locks are requested within other locks as far as we can see. Therefor our best guess would be a deadlock on a thread already holding the globalmutex_lock in combination with the slotmem file locks, but that's just a guess without debugging it. > More context can be found here: https://bugzilla.redhat.com/show_bug.cgi?id=1080047 -- This message was sent by Atlassian JIRA (v6.2.3#6260)

10 years, 7 months

1
0
0 / 0

[JBoss JIRA] (MODCLUSTER-398) mod_cluster deadlock in a jboss/windows environment

by Jean-Frederic Clere (JIRA)

[ https://issues.jboss.org/browse/MODCLUSTER-398?page=com.atlassian.jira.pl... ] Jean-Frederic Clere commented on MODCLUSTER-398: ------------------------------------------------ Actually the lock occurs in a piece of code that shouldn't be called in production (it is just there for demos). A fix that activates correctly the feature has been pushed (https://github.com/modcluster/mod_cluster/commit/df8af31db7468aba37c33c6e...). > mod_cluster deadlock in a jboss/windows environment > --------------------------------------------------- > > Key: MODCLUSTER-398 > URL: https://issues.jboss.org/browse/MODCLUSTER-398 > Project: mod_cluster > Issue Type: Bug > Affects Versions: 1.2.6.Final > Environment: Windows 2008, EAP6 and EWS2.0.1 > Reporter: Marc Maurer > Assignee: Jean-Frederic Clere > > Under load Apache stops serving pages, with all threads are stuck in "W : Sending reply" state. With the windows Process Explorer we then got a stacktrace from a hanging thread. We don't have debug symbols, but it's easy enough to see what's happening: > ntoskrnl.exe!KeWaitForMultipleObjects+0xc0a > ntoskrnl.exe!KeAcquireSpinLockAtDpcLevel+0x732 > ntoskrnl.exe!KeWaitForMutexObject+0x19f > ntoskrnl.exe!NtDeleteFile+0x3c4 > ntoskrnl.exe!PsDereferenceKernelStack+0x35358 > ntoskrnl.exe!KeSynchronizeExecution+0x3a23 > ntdll.dll!ZwLockFile+0xa > KERNELBASE.dll!LockFileEx+0xb2 > kernel32.dll!LockFileEx+0x1b > libapr-1.dll!apr_file_lock+0x69 <-- here > mod_slotmem.so+0x1318 <-- here > mod_manager.so+0x2a11 <-- here > mod_proxy_cluster.so+0x679e > mod_proxy.so!proxy_run_post_request+0x4e > mod_proxy.so!proxy_run_request_status+0x924 > libhttpd.dll!ap_run_handler+0x35 > libhttpd.dll!ap_invoke_handler+0x114 > libhttpd.dll!ap_die+0x2ea > libhttpd.dll!ap_psignature+0x1ae8 > libhttpd.dll!ap_run_process_connection+0x35 > libhttpd.dll!ap_process_connection+0x3b > libhttpd.dll!ap_regkey_value_remove+0x136e > msvcrt.dll!srand+0x93 > msvcrt.dll!ftime64_s+0x1dd > kernel32.dll!BaseThreadInitThunk+0xd > ntdll.dll!RtlUserThreadStart+0x21 > So mod_manager is requesting a filelock on one of the lockfiles in in the MemManagerFile path. In this case it was the "manager.sessionid.sessionid.lock" file. Removing the lockfile fixed the problem. > When bisecting the mod_cluster code, I think commit "74eeb9c026380deb8d833be53b09b3d808e02d10 - Lock in insert-update" in version 1.2.2 is the culprit. This would also explain why mod_cluster 1.2.1 is the last known working version. > What we don't know, is which process is already holding the lock when all Apache threads start blocking on it. We are trying to figure that out. There are no obviously wrong lock/unlock slotmem call pairs in the mod_manager module, and no locks are requested within other locks as far as we can see. Therefor our best guess would be a deadlock on a thread already holding the globalmutex_lock in combination with the slotmem file locks, but that's just a guess without debugging it. > More context can be found here: https://bugzilla.redhat.com/show_bug.cgi?id=1080047 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira

10 years, 8 months

1
0
0 / 0

[JBoss JIRA] (MODCLUSTER-398) mod_cluster deadlock in a jboss/windows environment

by Jean-Frederic Clere (JIRA)

[ https://issues.jboss.org/browse/MODCLUSTER-398?page=com.atlassian.jira.pl... ] Jean-Frederic Clere edited comment on MODCLUSTER-398 at 4/28/14 10:17 AM: -------------------------------------------------------------------------- Actually the lock occurs in a piece of code that shouldn't be called in production (it is just there for demos). A fix that desactivates correctly the feature has been pushed (https://github.com/modcluster/mod_cluster/commit/df8af31db7468aba37c33c6e...). was (Author: jfclere): Actually the lock occurs in a piece of code that shouldn't be called in production (it is just there for demos). A fix that activates correctly the feature has been pushed (https://github.com/modcluster/mod_cluster/commit/df8af31db7468aba37c33c6e...). > mod_cluster deadlock in a jboss/windows environment > --------------------------------------------------- > > Key: MODCLUSTER-398 > URL: https://issues.jboss.org/browse/MODCLUSTER-398 > Project: mod_cluster > Issue Type: Bug > Affects Versions: 1.2.6.Final > Environment: Windows 2008, EAP6 and EWS2.0.1 > Reporter: Marc Maurer > Assignee: Jean-Frederic Clere > > Under load Apache stops serving pages, with all threads are stuck in "W : Sending reply" state. With the windows Process Explorer we then got a stacktrace from a hanging thread. We don't have debug symbols, but it's easy enough to see what's happening: > ntoskrnl.exe!KeWaitForMultipleObjects+0xc0a > ntoskrnl.exe!KeAcquireSpinLockAtDpcLevel+0x732 > ntoskrnl.exe!KeWaitForMutexObject+0x19f > ntoskrnl.exe!NtDeleteFile+0x3c4 > ntoskrnl.exe!PsDereferenceKernelStack+0x35358 > ntoskrnl.exe!KeSynchronizeExecution+0x3a23 > ntdll.dll!ZwLockFile+0xa > KERNELBASE.dll!LockFileEx+0xb2 > kernel32.dll!LockFileEx+0x1b > libapr-1.dll!apr_file_lock+0x69 <-- here > mod_slotmem.so+0x1318 <-- here > mod_manager.so+0x2a11 <-- here > mod_proxy_cluster.so+0x679e > mod_proxy.so!proxy_run_post_request+0x4e > mod_proxy.so!proxy_run_request_status+0x924 > libhttpd.dll!ap_run_handler+0x35 > libhttpd.dll!ap_invoke_handler+0x114 > libhttpd.dll!ap_die+0x2ea > libhttpd.dll!ap_psignature+0x1ae8 > libhttpd.dll!ap_run_process_connection+0x35 > libhttpd.dll!ap_process_connection+0x3b > libhttpd.dll!ap_regkey_value_remove+0x136e > msvcrt.dll!srand+0x93 > msvcrt.dll!ftime64_s+0x1dd > kernel32.dll!BaseThreadInitThunk+0xd > ntdll.dll!RtlUserThreadStart+0x21 > So mod_manager is requesting a filelock on one of the lockfiles in in the MemManagerFile path. In this case it was the "manager.sessionid.sessionid.lock" file. Removing the lockfile fixed the problem. > When bisecting the mod_cluster code, I think commit "74eeb9c026380deb8d833be53b09b3d808e02d10 - Lock in insert-update" in version 1.2.2 is the culprit. This would also explain why mod_cluster 1.2.1 is the last known working version. > What we don't know, is which process is already holding the lock when all Apache threads start blocking on it. We are trying to figure that out. There are no obviously wrong lock/unlock slotmem call pairs in the mod_manager module, and no locks are requested within other locks as far as we can see. Therefor our best guess would be a deadlock on a thread already holding the globalmutex_lock in combination with the slotmem file locks, but that's just a guess without debugging it. > More context can be found here: https://bugzilla.redhat.com/show_bug.cgi?id=1080047 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira

10 years, 8 months

1
0
0 / 0

[JBoss JIRA] (MODCLUSTER-344) No pings when using HTTP connector in default configuration

by Radoslav Husar (JIRA)

[ https://issues.jboss.org/browse/MODCLUSTER-344?page=com.atlassian.jira.pl... ] Radoslav Husar reopened MODCLUSTER-344: --------------------------------------- Assignee: Radoslav Husar (was: Jean-Frederic Clere) 24/04/2014 meeting: agreed to enable options by default for http since 1.3.x. Also noting, needs documentation update. > No pings when using HTTP connector in default configuration > ------------------------------------------------------------ > > Key: MODCLUSTER-344 > URL: https://issues.jboss.org/browse/MODCLUSTER-344 > Project: mod_cluster > Issue Type: Bug > Affects Versions: 1.2.4.Final > Reporter: Radoslav Husar > Assignee: Radoslav Husar > Attachments: access_log, error_log, mod_cluster.conf > > > When integrating Undertow, I realized the servers are never removed from m_c when I am using HTTP connector to workaroung broken cping/cpong in Undertow's AJP. > One needs to configure EnableOptions to enable liveness checking, otherwise the node is never marked as NOTOK (in the larger than ping interval) until next request is received on that node and a client gets an unnecessary error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira

10 years, 8 months

1
0
0 / 0

[JBoss JIRA] (MODCLUSTER-365) Reset MCMPs are sent to all available proxies

by RH Bugzilla Integration (JIRA)

[ https://issues.jboss.org/browse/MODCLUSTER-365?page=com.atlassian.jira.pl... ] RH Bugzilla Integration commented on MODCLUSTER-365: ---------------------------------------------------- mark yarborough <myarboro(a)redhat.com> changed the Status of [bug 1050221|https://bugzilla.redhat.com/show_bug.cgi?id=1050221] from ASSIGNED to CLOSED > Reset MCMPs are sent to all available proxies > --------------------------------------------- > > Key: MODCLUSTER-365 > URL: https://issues.jboss.org/browse/MODCLUSTER-365 > Project: mod_cluster > Issue Type: Bug > Affects Versions: 1.2.6.Final > Environment: -JBoss Enterprise Application Platform (EAP) 6.1.1 > Reporter: Aaron Ogburn > Assignee: Jean-Frederic Clere > Labels: mod_cluster > Fix For: 1.2.7.Final, 1.3.0.Final > > > Consider a JBoss server with multiple httpd servers defined in its proxy-list. If one of those httpd servers goes down and then comes back up, then reset MCMPs (CONFIG, ENABLE-APP) would be sent to the restarted httpd proxy but needlessly to all other proxies as well. This is because DefaultMCMPHandler.status just calls sendRequests, which will send to all proxies by design. > This has negative implications if auto-enable-contexts is set to false and can break desired behavior. Reproducer steps detail the issue introduced in that scenario. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira

10 years, 8 months

1
0
0 / 0

[JBoss JIRA] (MODCLUSTER-299) REGRESSION - mod_cluster issues an ENABLE-APP too early in the webapp lifecycle

by Michal Babacek (JIRA)

[ https://issues.jboss.org/browse/MODCLUSTER-299?page=com.atlassian.jira.pl... ] Michal Babacek closed MODCLUSTER-299. ------------------------------------- > REGRESSION - mod_cluster issues an ENABLE-APP too early in the webapp lifecycle > ------------------------------------------------------------------------------- > > Key: MODCLUSTER-299 > URL: https://issues.jboss.org/browse/MODCLUSTER-299 > Project: mod_cluster > Issue Type: Bug > Affects Versions: 1.0.10, MOD_CLUSTER_1_0_10_GA_CP02 > Environment: JBoss-5.1.0.GA > Reporter: Gregory Lardiere > Assignee: Jean-Frederic Clere > Fix For: MOD_CLUSTER_1_0_10_GA_CP03 > > > mod_cluster ENABLE the application too early. > The ENABLE-APP message is sent twice : at the beginning AND at the end of the application startup. > This is a regression : it worked fine after MODCLUSTER-190 was resolved (r408). > It seems that r551 (MODCLUSTER-224) has re-introduced the bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira

10 years, 8 months

1
0
0 / 0

[JBoss JIRA] (MODCLUSTER-313) Segmentation Faut when using httpd-2.2.15 with mod_cluster compiled with 2.2.17.

by Michal Babacek (JIRA)

[ https://issues.jboss.org/browse/MODCLUSTER-313?page=com.atlassian.jira.pl... ] Michal Babacek closed MODCLUSTER-313. ------------------------------------- > Segmentation Faut when using httpd-2.2.15 with mod_cluster compiled with 2.2.17. > -------------------------------------------------------------------------------- > > Key: MODCLUSTER-313 > URL: https://issues.jboss.org/browse/MODCLUSTER-313 > Project: mod_cluster > Issue Type: Bug > Affects Versions: MOD_CLUSTER_1_0_10_GA_CP02 > Reporter: Jean-Frederic Clere > Assignee: Jean-Frederic Clere > Fix For: MOD_CLUSTER_1_0_10_GA_CP03 > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira

10 years, 8 months

1
0
0 / 0

[JBoss JIRA] (MODCLUSTER-319) Backport MODCLUSTER-168 to 1.0.10.GA - mod_cluster should use hostname provided in address instead a IP address

by Michal Babacek (JIRA)

[ https://issues.jboss.org/browse/MODCLUSTER-319?page=com.atlassian.jira.pl... ] Michal Babacek closed MODCLUSTER-319. ------------------------------------- Closing. It works in EAP 5.3. > Backport MODCLUSTER-168 to 1.0.10.GA - mod_cluster should use hostname provided in address instead a IP address > --------------------------------------------------------------------------------------------------------------- > > Key: MODCLUSTER-319 > URL: https://issues.jboss.org/browse/MODCLUSTER-319 > Project: mod_cluster > Issue Type: Feature Request > Affects Versions: MOD_CLUSTER_1_0_10_GA_CP02 > Environment: -JBoss Enterprise Application Platform (EAP) 5 > -JBoss Enterprise Web Server (EWS) > -Apache httpd > -mod_cluster 1.0.10.GA_CP02 > Reporter: Aaron Ogburn > Assignee: Jean-Frederic Clere > Priority: Minor > Fix For: MOD_CLUSTER_1_0_10_GA_CP04 > > Attachments: Utils.java > > > The CONFIG message from JBoss to apache is always formatted like the following and still uses the resolved ip address instead of the bound to host name: > Port=8009, Host=127.0.0.1, Type=ajp, StickySessionForce=No, Maxattempts=1 > Even if the Connector is defined like: > <Connector port="8009" address= "localhost" protocol="AJP/1.3" redirectPort="8443" /> > This can cause issues if apache needs to proxy to JBoss through the host name and not the ip. This could be corrected by using InetAdress.toString() in org.jboss.modcluster.Utils and parsing the hostname out of this result if provided, for example: > public static String getAddress(Connector connector) > { > Object address = IntrospectionUtils.getProperty(connector.getProtocolHandler(), "address"); > > if (address == null) return "127.0.0.1"; > if (address instanceof InetAddress) > { > String addressString = ((InetAddress) address).toString(); > int index = addressString.indexOf("/"); > return (index > 0) ? addressString.substring(0, index) : addressString.substring(1); > } > else > { > return (String) address; > } > } -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira

10 years, 8 months

1
0
0 / 0

[JBoss JIRA] (MODCLUSTER-344) No pings when using HTTP connector in default configuration

by Michal Babacek (JIRA)

[ https://issues.jboss.org/browse/MODCLUSTER-344?page=com.atlassian.jira.pl... ] Michal Babacek commented on MODCLUSTER-344: ------------------------------------------- Hmm, it looks like everybody's happy... Let's either refresh the debate of close this thing. > No pings when using HTTP connector in default configuration > ------------------------------------------------------------ > > Key: MODCLUSTER-344 > URL: https://issues.jboss.org/browse/MODCLUSTER-344 > Project: mod_cluster > Issue Type: Bug > Affects Versions: 1.2.4.Final > Reporter: Radoslav Husar > Assignee: Jean-Frederic Clere > Attachments: access_log, error_log, mod_cluster.conf > > > When integrating Undertow, I realized the servers are never removed from m_c when I am using HTTP connector to workaroung broken cping/cpong in Undertow's AJP. > One needs to configure EnableOptions to enable liveness checking, otherwise the node is never marked as NOTOK (in the larger than ping interval) until next request is received on that node and a client gets an unnecessary error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira

10 years, 8 months

1
0
0 / 0

[JBoss JIRA] (MODCLUSTER-350) Optimize mod_cluster for high worker counts

by Michal Babacek (JIRA)

[ https://issues.jboss.org/browse/MODCLUSTER-350?page=com.atlassian.jira.pl... ] Michal Babacek closed MODCLUSTER-350. ------------------------------------- IMHO Fixed. I'll keep an eye on performance charts as in MODCLUSTER-372... > Optimize mod_cluster for high worker counts > ------------------------------------------- > > Key: MODCLUSTER-350 > URL: https://issues.jboss.org/browse/MODCLUSTER-350 > Project: mod_cluster > Issue Type: Enhancement > Affects Versions: 1.2.4.Final > Reporter: Aaron Ogburn > Assignee: Jean-Frederic Clere > > Performance degrades (increased load, longer responses) with increased worker counts on mod_cluster. It'd be nice if some optimizing could be done here so that workers are selected through hashes for better scaling. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira

10 years, 8 months

1
0
0 / 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

mod_cluster-issues April 2014