[mod_cluster-dev] Handling crashed/hung AS nodes

Brian Stansberry brian.stansberry at redhat.com
Fri Mar 27 17:23:08 EDT 2009 

Paul Ferraro wrote:
> On Fri, 2009-03-27 at 09:15 -0500, Brian Stansberry wrote:
>> jean-frederic clere wrote:
>>> Paul Ferraro wrote:
>>>> Currently, the HAModClusterService (where httpd communication is
>>>> coordinated by an HA singleton) does not react to crashed/hung members.
>>>> Specifically, when the HA singleton gets a callback that the group
>>>> membership changes, it does not send any REMOVE-APP messages to httpd on
>>>> behalf of the member that just left.  Currently, httpd will detect the
>>>> failure (via a disconnected socket) on its own and sets its internal
>>>> state accordingly, e.g. a STATUS message will return NOTOK.
>>>>
>>>> The non-handling of dropped members is actually a good thing in the
>>>> event of a network partition, where communication between nodes is lost,
>>>> but communication between httpd and the nodes is unaffected.  If we were
>>>> handling dropped members, we would have to handle the ugly scenario
>>>> described here:
>>>> https://jira.jboss.org/jira/browse/MODCLUSTER-66
>>>>
>>>> Jean-Frederic: a few questions...
>>>> 1. Is it up to the AS to drive the recovery of a NOTOK node when it
>>>> becomes functional again?
>>> Yes.
>>>
>>>>  In the case of a crashed member, fresh
>>>> CONFIG/ENABLE-APP messages will be sent upon node restart.  In the case
>>>> of a re-merged network partition, no additional messages are sent.  Is
>>>> the subsequent STATUS message (with a non-zero lbfactor) enough to
>>>> trigger the recovery of this node?
>>> Yes.
> 
> Good to know.
>  
>>>> 2. Can httpd detect hung nodes?  A hung node will not affect the
>>>> connected state of the AJP/HTTP/S connector - it could only detect this
>>>> by sending data to the connector and timing out on the response.
>>> The hung node will be detected and marked as broken but the 
>>> corresponding request(s) may be delayed or lost due to time-out.
>>>
>> How long does this take, say in a typical case where the hung node was 
>> up and running with a pool of AJP connections open? Is it the 10 secs, 
>> the default value of the "ping" property listed at 
>> https://www.jboss.org/mod_cluster/java/properties.html#proxy ?
> 
> I think he's talking about "nodeTimeout".
> 

There's also the mod_proxy ProxyTimeout directive which might??? apply 
since mod_proxy is what is being used. From the mod_proxy docs:

"This directive allows a user to specifiy a timeout on proxy requests. 
This is useful when you have a slow/buggy appserver which hangs, and you 
would rather just return a timeout and fail gracefully instead of 
waiting however long it takes the server to return."

Default there is 300 seconds!!

It would be good to beef up the docs of this quite a bit. From the 
mod_jk docs at 
http://tomcat.apache.org/connectors-doc/reference/workers.html I can get 
a pretty good idea of all the details of how mod_jk works in this area. 
Not so much from 
https://www.jboss.org/mod_cluster/java/properties.html#proxy 
particularly when I factor in that it's mod_proxy/mod_proxy_ajp that's 
actually handling requests.
>> Also, if a request is being handled by a hung node and the 
>> HAModClusterService tells httpd to stop that node, the request will 
>> fail, yes? It shouldn't just fail over, as it may have already caused 
>> the transfer of my $1,000,000 to my secret account at UBS. Failing over 
>> would cause transfer of a second $1,000,000 and sadly I don't have that 
>> much.
> 
> Not unlike those damn double-clickers...
> 

Ah, that's what happened to my second $1,000,000! Thanks!

>>>> And some questions for open discussion:
>>>> What does HAModClusterService really buy us over the normal
>>>> ModClusterService?  Do the benefits outweigh the complexity?
>>>>  * Maintains a uniform view of proxy status across each AS node
>>>>  * Can detect and send STOP-APP/REMOVE-APP messages on behalf of
>>>> hung/crashed nodes (if httpd cannot already do this) (not yet
>>>> implemented)
>>>>    + Requires special handling of network partitions
>>>>  * Potentially improve scalability by minimizing network traffic for
>>>> very large clusters.
>> Assume a near-term goal is to run a 150 node cluster with say 10 httpd 
>> servers. Assume the background thread runs every 10 seconds. That comes 
>> to 150 connections per second across the cluster being opened/closed to 
>> handle STATUS. Each httpd server handles 15 connections per second.
>>
>> With HAModClusterService the way it is now, you get the same, because 
>> besides STATUS each node also checks its ability to communicate w/ each 
>> httpd in order to validate its ability to become master. But let's 
>> assume we add some complexity to allow that health check to become much 
>> more infrequent. So ignore those ping checks. So, w/ HAModClusterService 
>> you get 1 connection/sec being opened closed across the cluster for 
>> status, 0.1 connection/sec per httpd.  But the STATUS request sent 
>> across each connection has a much bigger payload.
> 
> True, the STATUS request body is larger than the INFO request (no body),
> but the resulting STATUS-RSP is significantly smaller than the
> corresponding INFO-RSP.
> 

Ah, we're still using INFO for the connectivity checks? That's not good; 
for sure we'd want to make that happen less often.

I wasn't clear; what I was driving at was the STATUS request from 
HAModClusterService covering 150 nodes has a bigger payload than any 
single node's STATUS; i.e. there's no magic reduction in data. The 
efficiency would be in getting rid of opening/closing lots of connections.

>> How significant is the cost of opening/closing all those connections?
>>
>>>>    e.g. non-masters ping httpd less often
>>>>  * Anything else?
>> 1) Management?? You don't want to have to interact with every node to do 
>> management tasks (e.g. disable app X on domain A to drain sessions so we 
>> can shut down the domain.) A node having a complete view might allow 
>> more sophisticated management operations.  This is takes more thought 
>> though.
> 
> Good point.
> 
>> 2) The more sophisticated case discussed on 
>> https://jira.jboss.org/jira/browse/MODCLUSTER-66, where a primary 
>> partition approach is appropriate rather than letting minority 
>> subpartitions continue to live. But TBH mod_cluster might not be the 
>> right place to handle this. Probably more appropriate is to have 
>> something associated with the webapp itself determine it is in a 
>> minority partition and undeploy the webapp if so. Whether being in a 
>> minority partition is inappropriate for a particular webapp is beyond 
>> scope for mod_cluster.
>>
>> I'd originally thought the HA version would add benefit to the load 
>> balance factor calculation, but that was wrong-headed.
> 
> I would argue that there is some (albeit small) value to the load being
> requested on each node at the same time.  I would expect this to result
> in slightly less load swinging than if individual nodes calculated their
> load at different times, scattered across the status interval.
> 
>>> Well I prefer the JAVA code deciding if a node is broken that httpd. I 
>>> really want to keep the complexity in httpd to minimum and the talk I 
>>> had until now at the ApacheCon seems to show that is probably the best 
>>> way to go.
>>>
>> Agreed that httpd should be as simple as possible. But to handle the 
>> non-HAModClusterService case it will need to at least detect broken 
>> connections and basic response timeouts, right? So depending on how long 
>> it takes to detect hung nodes, httpd might be detecting them before 
>> HAModClusterService. I'm thinking of 3 scenarios:
>>
>> 1) Node completely crashes. HAModClusterService will detect this almost 
>> immediately; I'd think httpd would as well unless it just happened to 
>> not have connections open.
>>
>> 2) Some condition that causes the channel used by HAModClusterService to 
>> not process messages. This will lead to the node being suspected after 
>> 31.5 seconds with the default channel config. But httpd might detect a 
>> timeout faster than that?
>>
>> 3) Some condition that causes all the JBoss Web threads to block but 
>> doesn't impact the HAModClusterService channel. (QE's Radoslav Husar, 
>> Bela and I are trying to diagnose such a case right now.)  Only httpd 
>> will detect this; JGroups will not. We could add some logic in JBoss Web 
>> that would allow it to detect such a situation and then let 
>> (HA)ModClusterService disable the node. But non-HA ModClusterService 
>> could do that just as well as the HA version.
> 
> I imagine this is not uncommon, e.g. overloaded/deadlocked database
> causing all application threads to wait.
> 
>> Out of these 3 cases, HAModClusterService does a better job than httpd 
>> itself only in the #2 case, and there only if it takes httpd > 31.5 secs 
>> to detect a hung response.
> 
> Although, for case #2, using the plain non-HA ModClusterService avoids
> the problem entirely.
> 

I wasn't clear again. :) For #2 I meant some general problem w/ the node 
(e.g. OOM) that affects the JGroups channel and the connectors. Then 
HAModClusterService has a benefit only if it detects that faster than 
httpd.  If *only* the JGroups channel is disrupted, yeah, that's the 
same as the network partition problem you raised.

>>> Cheers
>>>
>>> Jean-Frederic
>>>
>>>> Paul
>>>>
>>>> _______________________________________________
>>>> mod_cluster-dev mailing list
>>>> mod_cluster-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/mod_cluster-dev
>>>>
>>> _______________________________________________
>>> mod_cluster-dev mailing list
>>> mod_cluster-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/mod_cluster-dev
>>
> 


-- 
Brian Stansberry
Lead, AS Clustering
JBoss, a division of Red Hat
brian.stansberry at redhat.com

More information about the mod_cluster-dev mailing list