When I have 2 nodes (node1, node2), and a session which is created on node1 (and replicated to node2), the following can happen: * I shut down node1 (CTRL-C) * node1 terminates cleanly * I access the webapp, but get a "Session temporarily unavailable" failure message (I guess a 500) * When I check mod-cluster-manager, information about node1 is still there ! * Ca 5 seconds later, mod-cluster-manager doesn't show node1 anymore * When I now access the webapp, I get the proper failed over session on node2 and all my data is still there So my question is why doesn't node2 immediately tell httpd/mod-cluster that node1 is gone ? It seems that httpd *itself* only learns about this when it pings the socket to node1... I recall Paul once telling me we hadn't implemented that functionality, but I guess by now this is surely implemented ?

>> When I have 2 nodes (node1, node2), and a session which is created on >> node1 (and replicated to node2), the following can happen: >> >> * I shut down node1 (CTRL-C) >> * node1 terminates cleanly >> * I access the webapp, but get a "Session temporarily unavailable" >> failure message (I guess a 500) >> * When I check mod-cluster-manager, information about node1 is still >> there ! >> * Ca 5 seconds later, mod-cluster-manager doesn't show node1 anymore >> * When I now access the webapp, I get the proper failed over session >> on node2 and all my data is still there >> >> >> So my question is why doesn't node2 immediately tell httpd/mod-cluster >> that node1 is gone ? It seems that httpd *itself* only learns about >> this >> when it pings the socket to node1... Bela Ban wrote: I should be. Do you see the appropriate STOP_APP, REMOVE_APP messages in your httpd access log?

>> I recall Paul once telling me we hadn't implemented that functionality, >> but I guess by now this is surely implemented ? I never said that, but I think I know of the conversation to which you refer... If you are running mod_cluster via HAModClusterService and node1 crashed (instead of a clean shut down), node2 will not send any MCMP messages to httpd on behalf of node1 when it receives a view change.

Paul Ferraro wrote: >>> When I have 2 nodes (node1, node2), and a session which is created on >>> node1 (and replicated to node2), the following can happen: >>> >>> * I shut down node1 (CTRL-C) >>> * node1 terminates cleanly >>> * I access the webapp, but get a "Session temporarily unavailable" >>> failure message (I guess a 500) >>> * When I check mod-cluster-manager, information about node1 is still >>> there ! >>> * Ca 5 seconds later, mod-cluster-manager doesn't show node1 anymore >>> * When I now access the webapp, I get the proper failed over session >>> on node2 and all my data is still there >>> >>> >>> So my question is why doesn't node2 immediately tell httpd/mod-cluster >>> that node1 is gone ? It seems that httpd *itself* only learns about >>> this >>> when it pings the socket to node1... > Bela Ban wrote: > > I should be. Do you see the appropriate STOP_APP, REMOVE_APP messages > in your httpd access log? I do, right when I kill the node. I think the delay I've been seeing could have been caused by having both ModClusterListener *and* the mod-cluster integration bean in server.xml... I don't see this issue anymore (I've also upgraded to mod-cluster 1.0.1 in the meantime)... I'll keep trying to reproduce this. >>> I recall Paul once telling me we hadn't implemented that functionality, >>> but I guess by now this is surely implemented ? > > I never said that, but I think I know of the conversation to which you > refer... > If you are running mod_cluster via HAModClusterService and node1 crashed > (instead of a clean shut down), node2 will not send any MCMP messages to > httpd on behalf of node1 when it receives a view change. Ah, yes, I remember now ! I just killed -9 node1 and mod-cluster-manager didn't remove node1 from its worker list. Why don't we do this ? We cannot rely on a node being shutdown gracefully !

This could be very simple logic, executed by the singleton: On becoming singleton: - Get the current view and remove all nodes from httpd which are not in the current view On view change: - Same as above, with the view shipped as argument to the view change We have to think whether this would make sense for node additions, but then again every new node registers itself, so maybe this is not necessary.

WDYT ?

Paul Ferraro wrote: >>> If you are running mod_cluster via HAModClusterService and node1 crashed >>> (instead of a clean shut down), node2 will not send any MCMP messages to >>> httpd on behalf of node1 when it receives a view change. >> Ah, yes, I remember now ! I just killed -9 node1 and mod-cluster-manager >> didn't remove node1 from its worker list. >> >> Why don't we do this ? We cannot rely on a node being shutdown >> gracefully ! > Because currently a given node cannot distinguish between a crashed > member and a network partition OK, so {A,B} would remove C and D and {C,D} would remove A and B... How about we send an MCMP SUSPECT message to httpd ? This would cause httpd to double-check, e.g. via the regular AJP cping/cpong or socket connection checks, and if the doube-check fails, remove a worker ? For the regular case (a crash), this would be faster than having to wait until httpd checks the connection. So a view change would be nothing else than trigger an immediate connection check on httpd.