Re: [mod_cluster-dev] Handling crashed/hung AS nodes

On Fri, 2009-03-27 at 09:15 -0500, Brian Stansberry wrote: > jean-frederic clere wrote: >> Paul Ferraro wrote: >>> Currently, the HAModClusterService (where httpd communication is >>> coordinated by an HA singleton) does not react to crashed/hung members. >>> Specifically, when the HA singleton gets a callback that the group >>> membership changes, it does not send any REMOVE-APP messages to httpd on >>> behalf of the member that just left. Currently, httpd will detect the >>> failure (via a disconnected socket) on its own and sets its internal >>> state accordingly, e.g. a STATUS message will return NOTOK. >>> >>> The non-handling of dropped members is actually a good thing in the >>> event of a network partition, where communication between nodes is lost, >>> but communication between httpd and the nodes is unaffected. If we were >>> handling dropped members, we would have to handle the ugly scenario >>> described here: >>> https://jira.jboss.org/jira/browse/MODCLUSTER-66 >>> >>> Jean-Frederic: a few questions... >>> 1. Is it up to the AS to drive the recovery of a NOTOK node when it >>> becomes functional again? >> Yes. >> >>> In the case of a crashed member, fresh >>> CONFIG/ENABLE-APP messages will be sent upon node restart. In the case >>> of a re-merged network partition, no additional messages are sent. Is >>> the subsequent STATUS message (with a non-zero lbfactor) enough to >>> trigger the recovery of this node? >> Yes. Good to know. >>> 2. Can httpd detect hung nodes? A hung node will not affect the >>> connected state of the AJP/HTTP/S connector - it could only detect this >>> by sending data to the connector and timing out on the response. >> The hung node will be detected and marked as broken but the >> corresponding request(s) may be delayed or lost due to time-out. >> > How long does this take, say in a typical case where the hung node was > up and running with a pool of AJP connections open? Is it the 10 secs, > the default value of the "ping" property listed at > https://www.jboss.org/mod_cluster/java/properties.html#proxy ? I think he's talking about "nodeTimeout".

> Also, if a request is being handled by a hung node and the > HAModClusterService tells httpd to stop that node, the request will > fail, yes? It shouldn't just fail over, as it may have already caused > the transfer of my $1,000,000 to my secret account at UBS. Failing over > would cause transfer of a second $1,000,000 and sadly I don't have that > much. Not unlike those damn double-clickers...