[ https://issues.jboss.org/browse/MODCLUSTER-407?page=com.atlassian.jira.pl... ] Aaron Ogburn commented on MODCLUSTER-407: ----------------------------------------- Fortunately there's only two places in mod_proxy_cluster that call apr_sleep: proxy_cluster_watchdog_func and find_best_worker. proxy_cluster_watchdog_func wouldn't be relevant since that's background/periodic functions and not request related. Checking out find_best_worker, it is indeed called by pre_request. Also it is recursive, so it seems to be the only thing that meets all criteria for a culprit that is recursive, called by pre_request, and calling apr_sleep. Also, the problem spot is going to be called only when nodes are in error state (which we have when killing nodes) so find_best_worker certainly looks like the culprit. For further clarification, I added debug messages to find_best_worker indicating when it starts and ends and also when it starts and ends a recursive loop. After stopping request load, debug logging shows threads are continuing to start new recursive loops long after incoming requests stopped. It looks like relying on the balancer->timeout alone to determine whether we recurse or not is bad logic since multiple threads can get through at once. Then once one thread finishes a recursive loop, it sets balancer->timeout back, which can cause other threads to make another recursive loop. Once a few threads get in some recursive loops like that, they can keep each other stuck by continually resetting balancer->timeout and tripping one another back into the recursive loop. -- This message was sent by Atlassian JIRA (v6.2.3#6260)