June 2014 - mod_cluster-issues - Jboss List Archives

[JBoss JIRA] (MODCLUSTER-398) mod_cluster deadlock in a jboss/windows environment

by RH Bugzilla Integration (JIRA)

[ https://issues.jboss.org/browse/MODCLUSTER-398?page=com.atlassian.jira.pl... ] RH Bugzilla Integration commented on MODCLUSTER-398: ---------------------------------------------------- Weinan Li <weli(a)redhat.com> changed the Status of [bug 1080047|https://bugzilla.redhat.com/show_bug.cgi?id=1080047] from MODIFIED to ON_QA > mod_cluster deadlock in a jboss/windows environment > --------------------------------------------------- > > Key: MODCLUSTER-398 > URL: https://issues.jboss.org/browse/MODCLUSTER-398 > Project: mod_cluster > Issue Type: Bug > Security Level: Public(Everyone can see) > Affects Versions: 1.2.6.Final > Environment: Windows 2008, EAP6 and EWS2.0.1 > Reporter: Marc Maurer > Assignee: Jean-Frederic Clere > Fix For: 1.3.1.Final, 1.2.9.Final > > > Under load Apache stops serving pages, with all threads are stuck in "W : Sending reply" state. With the windows Process Explorer we then got a stacktrace from a hanging thread. We don't have debug symbols, but it's easy enough to see what's happening: > ntoskrnl.exe!KeWaitForMultipleObjects+0xc0a > ntoskrnl.exe!KeAcquireSpinLockAtDpcLevel+0x732 > ntoskrnl.exe!KeWaitForMutexObject+0x19f > ntoskrnl.exe!NtDeleteFile+0x3c4 > ntoskrnl.exe!PsDereferenceKernelStack+0x35358 > ntoskrnl.exe!KeSynchronizeExecution+0x3a23 > ntdll.dll!ZwLockFile+0xa > KERNELBASE.dll!LockFileEx+0xb2 > kernel32.dll!LockFileEx+0x1b > libapr-1.dll!apr_file_lock+0x69 <-- here > mod_slotmem.so+0x1318 <-- here > mod_manager.so+0x2a11 <-- here > mod_proxy_cluster.so+0x679e > mod_proxy.so!proxy_run_post_request+0x4e > mod_proxy.so!proxy_run_request_status+0x924 > libhttpd.dll!ap_run_handler+0x35 > libhttpd.dll!ap_invoke_handler+0x114 > libhttpd.dll!ap_die+0x2ea > libhttpd.dll!ap_psignature+0x1ae8 > libhttpd.dll!ap_run_process_connection+0x35 > libhttpd.dll!ap_process_connection+0x3b > libhttpd.dll!ap_regkey_value_remove+0x136e > msvcrt.dll!srand+0x93 > msvcrt.dll!ftime64_s+0x1dd > kernel32.dll!BaseThreadInitThunk+0xd > ntdll.dll!RtlUserThreadStart+0x21 > So mod_manager is requesting a filelock on one of the lockfiles in in the MemManagerFile path. In this case it was the "manager.sessionid.sessionid.lock" file. Removing the lockfile fixed the problem. > When bisecting the mod_cluster code, I think commit "74eeb9c026380deb8d833be53b09b3d808e02d10 - Lock in insert-update" in version 1.2.2 is the culprit. This would also explain why mod_cluster 1.2.1 is the last known working version. > What we don't know, is which process is already holding the lock when all Apache threads start blocking on it. We are trying to figure that out. There are no obviously wrong lock/unlock slotmem call pairs in the mod_manager module, and no locks are requested within other locks as far as we can see. Therefor our best guess would be a deadlock on a thread already holding the globalmutex_lock in combination with the slotmem file locks, but that's just a guess without debugging it. > More context can be found here: https://bugzilla.redhat.com/show_bug.cgi?id=1080047 -- This message was sent by Atlassian JIRA (v6.2.6#6264)

9 years, 12 months

1
0
0 / 0

[JBoss JIRA] (MODCLUSTER-417) Obfuscating jvmRoute as to hide topology

by Juan Manuel CABRERA (JIRA)

[ https://issues.jboss.org/browse/MODCLUSTER-417?page=com.atlassian.jira.pl... ] Juan Manuel CABRERA commented on MODCLUSTER-417: ------------------------------------------------ Hello everyone. I'm the originator of that feature request, so I'll try to explain a little more why I think this is a good idea. Imagine an attacker that wants to create a DOS attack on a particular cluster. Its simpler for him to crash each nodes one at the time, this would require substantially less power that trying to crash the whole cluster as a whole and would be much more efficient. Each time you successfully kill a node, the load on each remaining node is mechanically increased and very likely, the last few nodes will crash on their own under the full "normal" load. An example: if killing a whole cluster of 100 nodes requires an overload of say X: - killing the first node requires 1% of X obviously - second node requires 0.96% of X so you can start killing 3 a bit. - .. - Node 12 is killed twice as fast as node 1 with the same power - Node 16 is killed 4x faster than node 1 - Node 19 and 20 die almost instantly - After the node 20 has crashed, the remaining nodes are > 100% so there is little to do to kill these if they are not dead already. As it goes with any attack possibility, there are certainly ways to prevent that with other means, but I think that the obfuscation/ciphering of the jvmroute would be a good protection. Kind regards, Juan Manuel > Obfuscating jvmRoute as to hide topology > ---------------------------------------- > > Key: MODCLUSTER-417 > URL: https://issues.jboss.org/browse/MODCLUSTER-417 > Project: mod_cluster > Issue Type: Feature Request > Security Level: Public(Everyone can see) > Components: Native (httpd modules) > Affects Versions: 1.3.0.Final, 1.2.9.Final > Reporter: Radoslav Husar > Assignee: Jean-Frederic Clere > Priority: Minor > > Feature request from https://github.com/jmcabrera > Hello guys. > First of all, this is a feature request and not a bug. > I would like to "obfuscate" the jvmRoute so that an external attacker cannot "guess" the topology of my internal infrastructure. > The "strong" way would be to have a symmetrical cipher with a configurable key. > mod_cluster could then cipher the jsessionid before exposing it to the external world, and decipher it to recover the jvmRoute and properly redirect the request. > But I guess that this would have very undesirable consequences on performance. > The "weak" way would be just obfuscate, i.e. let's say that the jsessionid is alea + '.' + jvmRoute. We could take a part of the alea to alter the jvmroute in a reversible way (XORing for instance). > Anyhow, the expected effect would be that the jvmroute would be externally different for each and every request. > Unfortunately, I have close to no C skills, hence I cannot make this myself. > (as a side note, coming from mod_jk, I'm quite impressed by the features mod_cluster offers! Thanks for the good work :) ) -- This message was sent by Atlassian JIRA (v6.2.6#6264)

9 years, 12 months

1
0
0 / 0

[JBoss JIRA] (MODCLUSTER-398) mod_cluster deadlock in a jboss/windows environment

by RH Bugzilla Integration (JIRA)

[ https://issues.jboss.org/browse/MODCLUSTER-398?page=com.atlassian.jira.pl... ] RH Bugzilla Integration commented on MODCLUSTER-398: ---------------------------------------------------- Jean-frederic Clere <jclere(a)redhat.com> changed the Status of [bug 1080047|https://bugzilla.redhat.com/show_bug.cgi?id=1080047] from NEW to MODIFIED > mod_cluster deadlock in a jboss/windows environment > --------------------------------------------------- > > Key: MODCLUSTER-398 > URL: https://issues.jboss.org/browse/MODCLUSTER-398 > Project: mod_cluster > Issue Type: Bug > Security Level: Public(Everyone can see) > Affects Versions: 1.2.6.Final > Environment: Windows 2008, EAP6 and EWS2.0.1 > Reporter: Marc Maurer > Assignee: Jean-Frederic Clere > Fix For: 1.3.1.Final, 1.2.9.Final > > > Under load Apache stops serving pages, with all threads are stuck in "W : Sending reply" state. With the windows Process Explorer we then got a stacktrace from a hanging thread. We don't have debug symbols, but it's easy enough to see what's happening: > ntoskrnl.exe!KeWaitForMultipleObjects+0xc0a > ntoskrnl.exe!KeAcquireSpinLockAtDpcLevel+0x732 > ntoskrnl.exe!KeWaitForMutexObject+0x19f > ntoskrnl.exe!NtDeleteFile+0x3c4 > ntoskrnl.exe!PsDereferenceKernelStack+0x35358 > ntoskrnl.exe!KeSynchronizeExecution+0x3a23 > ntdll.dll!ZwLockFile+0xa > KERNELBASE.dll!LockFileEx+0xb2 > kernel32.dll!LockFileEx+0x1b > libapr-1.dll!apr_file_lock+0x69 <-- here > mod_slotmem.so+0x1318 <-- here > mod_manager.so+0x2a11 <-- here > mod_proxy_cluster.so+0x679e > mod_proxy.so!proxy_run_post_request+0x4e > mod_proxy.so!proxy_run_request_status+0x924 > libhttpd.dll!ap_run_handler+0x35 > libhttpd.dll!ap_invoke_handler+0x114 > libhttpd.dll!ap_die+0x2ea > libhttpd.dll!ap_psignature+0x1ae8 > libhttpd.dll!ap_run_process_connection+0x35 > libhttpd.dll!ap_process_connection+0x3b > libhttpd.dll!ap_regkey_value_remove+0x136e > msvcrt.dll!srand+0x93 > msvcrt.dll!ftime64_s+0x1dd > kernel32.dll!BaseThreadInitThunk+0xd > ntdll.dll!RtlUserThreadStart+0x21 > So mod_manager is requesting a filelock on one of the lockfiles in in the MemManagerFile path. In this case it was the "manager.sessionid.sessionid.lock" file. Removing the lockfile fixed the problem. > When bisecting the mod_cluster code, I think commit "74eeb9c026380deb8d833be53b09b3d808e02d10 - Lock in insert-update" in version 1.2.2 is the culprit. This would also explain why mod_cluster 1.2.1 is the last known working version. > What we don't know, is which process is already holding the lock when all Apache threads start blocking on it. We are trying to figure that out. There are no obviously wrong lock/unlock slotmem call pairs in the mod_manager module, and no locks are requested within other locks as far as we can see. Therefor our best guess would be a deadlock on a thread already holding the globalmutex_lock in combination with the slotmem file locks, but that's just a guess without debugging it. > More context can be found here: https://bugzilla.redhat.com/show_bug.cgi?id=1080047 -- This message was sent by Atlassian JIRA (v6.2.6#6264)

9 years, 12 months

1
0
0 / 0

[JBoss JIRA] (MODCLUSTER-416) mod_cluster Connected count shows improper inflation

by Radoslav Husar (JIRA)

[ https://issues.jboss.org/browse/MODCLUSTER-416?page=com.atlassian.jira.pl... ] Radoslav Husar updated MODCLUSTER-416: -------------------------------------- Fix Version/s: 1.3.1.Final 1.2.10.Final > mod_cluster Connected count shows improper inflation > ---------------------------------------------------- > > Key: MODCLUSTER-416 > URL: https://issues.jboss.org/browse/MODCLUSTER-416 > Project: mod_cluster > Issue Type: Bug > Security Level: Public(Everyone can see) > Components: Native (httpd modules) > Affects Versions: 1.2.9.Final > Reporter: Aaron Ogburn > Assignee: Jean-Frederic Clere > Fix For: 1.3.1.Final, 1.2.10.Final > > > The mod_cluster connected count (a proxy_worker's busy attribute) shows improper inflation. With load stopped, non-zero values have still been seen through the mod_cluster-manager page and INFO MCMP responses. -- This message was sent by Atlassian JIRA (v6.2.6#6264)

10 years

1
0
0 / 0

[JBoss JIRA] (MODCLUSTER-417) Obfuscating jvmRoute as to hide topology

by Jean-Frederic Clere (JIRA)

[ https://issues.jboss.org/browse/MODCLUSTER-417?page=com.atlassian.jira.pl... ] Jean-Frederic Clere commented on MODCLUSTER-417: ------------------------------------------------ Obfuscating the JVMRoute is probably not enough to hide the topology of the cluster behind the proxy.... To do hide the topology you need to replace the sessiond.jvmroute by something that looks random but translate in jsessionid.jvmroute in mod_cluster logic. But I don't think that the topology of the cluster is something usable by an attacker as the nodes should be behind a firewall and the JVMRoute doesn't need to relate to the internal node name. > Obfuscating jvmRoute as to hide topology > ---------------------------------------- > > Key: MODCLUSTER-417 > URL: https://issues.jboss.org/browse/MODCLUSTER-417 > Project: mod_cluster > Issue Type: Feature Request > Security Level: Public(Everyone can see) > Components: Native (httpd modules) > Affects Versions: 1.3.0.Final, 1.2.9.Final > Reporter: Radoslav Husar > Assignee: Jean-Frederic Clere > Priority: Minor > > Feature request from https://github.com/jmcabrera > Hello guys. > First of all, this is a feature request and not a bug. > I would like to "obfuscate" the jvmRoute so that an external attacker cannot "guess" the topology of my internal infrastructure. > The "strong" way would be to have a symmetrical cipher with a configurable key. > mod_cluster could then cipher the jsessionid before exposing it to the external world, and decipher it to recover the jvmRoute and properly redirect the request. > But I guess that this would have very undesirable consequences on performance. > The "weak" way would be just obfuscate, i.e. let's say that the jsessionid is alea + '.' + jvmRoute. We could take a part of the alea to alter the jvmroute in a reversible way (XORing for instance). > Anyhow, the expected effect would be that the jvmroute would be externally different for each and every request. > Unfortunately, I have close to no C skills, hence I cannot make this myself. > (as a side note, coming from mod_jk, I'm quite impressed by the features mod_cluster offers! Thanks for the good work :) ) -- This message was sent by Atlassian JIRA (v6.2.6#6264)

10 years

1
0
0 / 0

[JBoss JIRA] (MODCLUSTER-416) mod_cluster Connected count shows improper inflation

by Jean-Frederic Clere (JIRA)

[ https://issues.jboss.org/browse/MODCLUSTER-416?page=com.atlassian.jira.pl... ] Jean-Frederic Clere updated MODCLUSTER-416: ------------------------------------------- Status: Resolved (was: Pull Request Sent) Resolution: Done merge master and 1.2.x > mod_cluster Connected count shows improper inflation > ---------------------------------------------------- > > Key: MODCLUSTER-416 > URL: https://issues.jboss.org/browse/MODCLUSTER-416 > Project: mod_cluster > Issue Type: Bug > Security Level: Public(Everyone can see) > Components: Native (httpd modules) > Affects Versions: 1.2.9.Final > Reporter: Aaron Ogburn > Assignee: Jean-Frederic Clere > > The mod_cluster connected count (a proxy_worker's busy attribute) shows improper inflation. With load stopped, non-zero values have still been seen through the mod_cluster-manager page and INFO MCMP responses. -- This message was sent by Atlassian JIRA (v6.2.6#6264)

10 years

1
0
0 / 0

[JBoss JIRA] (MODCLUSTER-416) mod_cluster Connected count shows improper inflation

by Jean-Frederic Clere (JIRA)

[ https://issues.jboss.org/browse/MODCLUSTER-416?page=com.atlassian.jira.pl... ] Jean-Frederic Clere edited comment on MODCLUSTER-416 at 6/18/14 1:55 AM: ------------------------------------------------------------------------- merged in master and 1.2.x was (Author: jfclere): merge master and 1.2.x > mod_cluster Connected count shows improper inflation > ---------------------------------------------------- > > Key: MODCLUSTER-416 > URL: https://issues.jboss.org/browse/MODCLUSTER-416 > Project: mod_cluster > Issue Type: Bug > Security Level: Public(Everyone can see) > Components: Native (httpd modules) > Affects Versions: 1.2.9.Final > Reporter: Aaron Ogburn > Assignee: Jean-Frederic Clere > > The mod_cluster connected count (a proxy_worker's busy attribute) shows improper inflation. With load stopped, non-zero values have still been seen through the mod_cluster-manager page and INFO MCMP responses. -- This message was sent by Atlassian JIRA (v6.2.6#6264)

10 years

1
0
0 / 0

[JBoss JIRA] (MODCLUSTER-416) mod_cluster Connected count shows improper inflation

by RH Bugzilla Integration (JIRA)

[ https://issues.jboss.org/browse/MODCLUSTER-416?page=com.atlassian.jira.pl... ] RH Bugzilla Integration updated MODCLUSTER-416: ----------------------------------------------- Bugzilla Update: Perform Bugzilla References: https://bugzilla.redhat.com/show_bug.cgi?id=1110515 > mod_cluster Connected count shows improper inflation > ---------------------------------------------------- > > Key: MODCLUSTER-416 > URL: https://issues.jboss.org/browse/MODCLUSTER-416 > Project: mod_cluster > Issue Type: Bug > Security Level: Public(Everyone can see) > Components: Native (httpd modules) > Affects Versions: 1.2.9.Final > Reporter: Aaron Ogburn > Assignee: Jean-Frederic Clere > > The mod_cluster connected count (a proxy_worker's busy attribute) shows improper inflation. With load stopped, non-zero values have still been seen through the mod_cluster-manager page and INFO MCMP responses. -- This message was sent by Atlassian JIRA (v6.2.6#6264)

10 years

1
0
0 / 0

[JBoss JIRA] (MODCLUSTER-416) mod_cluster Connected count shows improper inflation

by Aaron Ogburn (JIRA)

[ https://issues.jboss.org/browse/MODCLUSTER-416?page=com.atlassian.jira.pl... ] Aaron Ogburn updated MODCLUSTER-416: ------------------------------------ Git Pull Request: https://github.com/modcluster/mod_cluster/pull/91, https://github.com/modcluster/mod_cluster/pull/94 (was: https://github.com/modcluster/mod_cluster/pull/91) > mod_cluster Connected count shows improper inflation > ---------------------------------------------------- > > Key: MODCLUSTER-416 > URL: https://issues.jboss.org/browse/MODCLUSTER-416 > Project: mod_cluster > Issue Type: Bug > Security Level: Public(Everyone can see) > Components: Native (httpd modules) > Affects Versions: 1.2.9.Final > Reporter: Aaron Ogburn > Assignee: Jean-Frederic Clere > > The mod_cluster connected count (a proxy_worker's busy attribute) shows improper inflation. With load stopped, non-zero values have still been seen through the mod_cluster-manager page and INFO MCMP responses. -- This message was sent by Atlassian JIRA (v6.2.6#6264)

10 years

1
0
0 / 0

[JBoss JIRA] (MODCLUSTER-416) mod_cluster Connected count shows improper inflation

by Aaron Ogburn (JIRA)

[ https://issues.jboss.org/browse/MODCLUSTER-416?page=com.atlassian.jira.pl... ] Aaron Ogburn commented on MODCLUSTER-416: ----------------------------------------- PRs: 1.2.x: https://github.com/modcluster/mod_cluster/pull/91 master: https://github.com/modcluster/mod_cluster/pull/94 > mod_cluster Connected count shows improper inflation > ---------------------------------------------------- > > Key: MODCLUSTER-416 > URL: https://issues.jboss.org/browse/MODCLUSTER-416 > Project: mod_cluster > Issue Type: Bug > Security Level: Public(Everyone can see) > Components: Native (httpd modules) > Affects Versions: 1.2.9.Final > Reporter: Aaron Ogburn > Assignee: Jean-Frederic Clere > > The mod_cluster connected count (a proxy_worker's busy attribute) shows improper inflation. With load stopped, non-zero values have still been seen through the mod_cluster-manager page and INFO MCMP responses. -- This message was sent by Atlassian JIRA (v6.2.6#6264)

10 years

1
0
0 / 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

mod_cluster-issues June 2014