On 06/09/2010 02:43 PM, Bela Ban wrote: > I have the scenario where I run httpd/mod-cluster on an EC2 instance and > a few workers on different EC2 instances. > > When I "terminate" a worker instance (using the EC2 GUI), apparently the > virtual instance is terminated *ungracefully*, ie. similar to just > pulling the power plug. This means that the shutdown scripts (in > /etc/rc0.d) are not run, and the open sockets (e.g. to mod-cluster) are > not closed, so mod-cluster won't remove the worker. > > When I look at mod_cluster_manager, it continues listing the killed > worker in OK state. With CR2? Well at least for 5 seconds normally. > My questions: > > * I recall that, unlike mod-jk, mod-cluster doesn't have > cping/cpong, or any other heartbeating mechanism. Is this correct > ? It has a heartbeating logic so it should detect the dead node

> So would mod-cluster detect a worker's unreachability, e.g. when > I pull the plug on the switch connecting the worker to mod-cluster ? Yep, otherwise there is a bug somewhere. It would interesting to use debug in the httpd conf file and mail the corresponding error log. (Or open a JIRA and put the error_log file there)

On Wed, 2010-06-09 at 14:43 +0200, Bela Ban wrote: > I have the scenario where I run httpd/mod-cluster on an EC2 instance and > a few workers on different EC2 instances. > > When I "terminate" a worker instance (using the EC2 GUI), apparently the > virtual instance is terminated *ungracefully*, ie. similar to just > pulling the power plug. This means that the shutdown scripts (in > /etc/rc0.d) are not run, and the open sockets (e.g. to mod-cluster) are > not closed, so mod-cluster won't remove the worker. > > When I look at mod_cluster_manager, it continues listing the killed > worker in OK state. > > My questions: > > * I recall that, unlike mod-jk, mod-cluster doesn't have > cping/cpong, or any other heartbeating mechanism. Is this correct > ? So would mod-cluster detect a worker's unreachability, e.g. when > I pull the plug on the switch connecting the worker to mod-cluster ? > * I though that the workers detect when a member has crashed and the > cluster master then notifies the proxy. So when we have workers > {A,B,C}, and C crashes ungracefully, wouldn't A notify the proxy > of C's death, so the proxy can remove C ? > If you are indeed using HAModClusterService, you should see the following INFO message in the server.log of the master node, following the log messages regarding jgroups group membership change: Removing jvm route [...] from proxy [...] on behalf of crashed member: ...