4:48 p.m.
Hi Paul,
wanted to reiterate the importance of [1] for the next release of
mod-cluster.
I'm constantly running into this when I deploy httpd/mod-cluster and
JBoss 5.1.0 on Amazon's EC2 cloud. The easy way to stop an instance (OS
+ JBoss) on EC2 is to 'terminate' it via the AWS Console, which shuts
down the OS ("shutdown -h now").
Unfortunately, our AMIs only had an S98jboss in /etc/rc4.d for starting
JBoss, but no corresponding K98jboss for stopping it *gracefully* on
shutdown. Therefore the process was always killed via -9.
This caused very long timeouts and 5XX HTTP responses, until
httd/mod-cluster finally figured out that the worker crashed and failed
over to a different worker.
As a workaround, I created a K98jboss link so now JBoss is shut down
gracefully when the host is terminated.
However, I figure we can get into this situation in many different ways,
e.g.
* Not providing a K98jboss script on EC2
* Killing JBoss with -9 via a script (I've seen this many more than once
* Pulling a blade out of the rack. A crude way of shutting down an
instance, but that's normal in large clusters !
Can we have this feature in mod-cluster 1.1 ?
[1] https://jira.jboss.org/jira/browse/MODCLUSTER-66
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
5:23 p.m.
On Wed, 2009-08-05 at 16:48 +0200, Bela Ban wrote:
Hi Paul,
wanted to reiterate the importance of [1] for the next release of
mod-cluster.
I'm constantly running into this when I deploy httpd/mod-cluster and
JBoss 5.1.0 on Amazon's EC2 cloud. The easy way to stop an instance (OS
+ JBoss) on EC2 is to 'terminate' it via the AWS Console, which shuts
down the OS ("shutdown -h now").
Unfortunately, our AMIs only had an S98jboss in /etc/rc4.d for starting
JBoss, but no corresponding K98jboss for stopping it *gracefully* on
shutdown. Therefore the process was always killed via -9.
This caused very long timeouts and 5XX HTTP responses, until
httd/mod-cluster finally figured out that the worker crashed and failed
over to a different worker.
As a workaround, I created a K98jboss link so now JBoss is shut down
gracefully when the host is terminated.
However, I figure we can get into this situation in many different ways,
e.g.
* Not providing a K98jboss script on EC2
* Killing JBoss with -9 via a script (I've seen this many more than once
* Pulling a blade out of the rack. A crude way of shutting down an
instance, but that's normal in large clusters !
Can we have this feature in mod-cluster 1.1 ?
[1] https://jira.jboss.org/jira/browse/MODCLUSTER-66
Yes. I've set the target version accordingly.
5:55 p.m.
I don't see why a hard kill would cause 5xx responses to the client
(except perhaps for a few requests that were being handled by the killed
server when it was killed.) The httpd side should try to route future
requests to the failed node and when it can't connect fail them over.
"Very long timeouts" on subsequent requests sounds like something that
should solvable via configuration as well. (Paul, JFC please comment.)
The backend node is dead; there's no reason it should be taking the
httpd side a "very long" time to detect that when a new request comes in.
(I'm not saying MODCLUSTER-66 isn't important; just that the symptoms
you are describing seem more severe than is warranted. If people aren't
using HAModClusterService, the MODCLUSTER-66 fix won't help them, but
they still shouldn't be seeing what you describe.)
Paul Ferraro wrote:
On Wed, 2009-08-05 at 16:48 +0200, Bela Ban wrote:
> Hi Paul,
>
> wanted to reiterate the importance of [1] for the next release of
> mod-cluster.
>
> I'm constantly running into this when I deploy httpd/mod-cluster and
> JBoss 5.1.0 on Amazon's EC2 cloud. The easy way to stop an instance (OS
> + JBoss) on EC2 is to 'terminate' it via the AWS Console, which shuts
> down the OS ("shutdown -h now").
>
> Unfortunately, our AMIs only had an S98jboss in /etc/rc4.d for starting
> JBoss, but no corresponding K98jboss for stopping it *gracefully* on
> shutdown. Therefore the process was always killed via -9.
>
> This caused very long timeouts and 5XX HTTP responses, until
> httd/mod-cluster finally figured out that the worker crashed and failed
> over to a different worker.
>
> As a workaround, I created a K98jboss link so now JBoss is shut down
> gracefully when the host is terminated.
>
> However, I figure we can get into this situation in many different ways,
> e.g.
>
> * Not providing a K98jboss script on EC2
> * Killing JBoss with -9 via a script (I've seen this many more than once
> * Pulling a blade out of the rack. A crude way of shutting down an
> instance, but that's normal in large clusters !
>
> Can we have this feature in mod-cluster 1.1 ?
>
>
> [1] https://jira.jboss.org/jira/browse/MODCLUSTER-66
Yes. I've set the target version accordingly.
_______________________________________________
mod_cluster-dev mailing list
mod_cluster-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/mod_cluster-dev
--
Brian Stansberry
Lead, AS Clustering
JBoss by Red Hat
9:11 a.m.
On 08/05/2009 05:55 PM, Brian Stansberry wrote:
I don't see why a hard kill would cause 5xx responses to the
client
(except perhaps for a few requests that were being handled by the killed
server when it was killed.) The httpd side should try to route future
requests to the failed node and when it can't connect fail them over.
"Very long timeouts" on subsequent requests sounds like something that
should solvable via configuration as well. (Paul, JFC please comment.)
Long shouldn't be bigger that the ping timeout value, if it is longer
than there is a bug in the C part of mod_cluster.
Cheers
Jean-Frederic
7:58 a.m.
jean-frederic clere wrote:
On 08/05/2009 05:55 PM, Brian Stansberry wrote:
> I don't see why a hard kill would cause 5xx responses to the client
> (except perhaps for a few requests that were being handled by the killed
> server when it was killed.) The httpd side should try to route future
> requests to the failed node and when it can't connect fail them over.
>
> "Very long timeouts" on subsequent requests sounds like something that
> should solvable via configuration as well. (Paul, JFC please comment.)
Long shouldn't be bigger that the ping timeout value, if it is longer
than there is a bug in the C part of mod_cluster.
Is there a config which makes httpd remove a worker W when W closes the
AJP socket ?
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
10:24 a.m.
On 08/09/2009 07:58 AM, Bela Ban wrote:
jean-frederic clere wrote:
> On 08/05/2009 05:55 PM, Brian Stansberry wrote:
>> I don't see why a hard kill would cause 5xx responses to the client
>> (except perhaps for a few requests that were being handled by the killed
>> server when it was killed.) The httpd side should try to route future
>> requests to the failed node and when it can't connect fail them over.
>>
>> "Very long timeouts" on subsequent requests sounds like something that
>> should solvable via configuration as well. (Paul, JFC please comment.)
>
> Long shouldn't be bigger that the ping timeout value, if it is longer
> than there is a bug in the C part of mod_cluster.
Is there a config which makes httpd remove a worker W when W closes the
AJP socket ?
No not for the moment.
Cheers
Jean-Frederic
10:34 a.m.
How about a timeout setting for ping/pong that I can lower ?
jean-frederic clere wrote:
On 08/09/2009 07:58 AM, Bela Ban wrote:
>
>
> jean-frederic clere wrote:
>> On 08/05/2009 05:55 PM, Brian Stansberry wrote:
>>> I don't see why a hard kill would cause 5xx responses to the client
>>> (except perhaps for a few requests that were being handled by the
>>> killed
>>> server when it was killed.) The httpd side should try to route future
>>> requests to the failed node and when it can't connect fail them over.
>>>
>>> "Very long timeouts" on subsequent requests sounds like something
that
>>> should solvable via configuration as well. (Paul, JFC please comment.)
>>
>> Long shouldn't be bigger that the ping timeout value, if it is longer
>> than there is a bug in the C part of mod_cluster.
>
> Is there a config which makes httpd remove a worker W when W closes the
> AJP socket ?
No not for the moment.
Cheers
Jean-Frederic
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
10:38 a.m.
On 08/10/2009 10:34 AM, Bela Ban wrote:
How about a timeout setting for ping/pong that I can lower ?
See http://www.jboss.org/mod_cluster/java/properties.html#proxy
Default is 10 seconds.
Cheers
Jean-Frederic
11:28 a.m.
OK, I'll try this out. In my case, the failed worker didn't seem to be
removed after 10secs, but I'll double check and open a JIRA should this
still fail
jean-frederic clere wrote:
On 08/10/2009 10:34 AM, Bela Ban wrote:
> How about a timeout setting for ping/pong that I can lower ?
See http://www.jboss.org/mod_cluster/java/properties.html#proxy
Default is 10 seconds.
Cheers
Jean-Frederic
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
11:55 a.m.
On Aug 10, 2009, at 11:28 AM, Bela Ban wrote:
OK, I'll try this out. In my case, the failed worker didn't
seem to be
removed after 10secs, but I'll double check and open a JIRA should
this
still fail
Probably I'll talk about something different, but still connected with
this issue.
I'm assuming (hopefully correctly) if a pong message isn't received, a
node should be suspected to remove or removed immediately. I think
this is the purpose for ping, right?
I think this is working – new request are routed to working nodes, not
to dead. Mod_cluster manager should also update it's status to show
always current cluster view. Isn't this a mod_cluster manager bug?
--Marek
jean-frederic clere wrote:
> On 08/10/2009 10:34 AM, Bela Ban wrote:
>> How about a timeout setting for ping/pong that I can lower ?
>
> See http://www.jboss.org/mod_cluster/java/properties.html#proxy
> Default is 10 seconds.
--
Marek Goldmann
JBoss by Red Hat
2:24 p.m.
On 08/10/2009 11:28 AM, Bela Ban wrote:
OK, I'll try this out. In my case, the failed worker didn't
seem to be
removed after 10secs,
The httpd logic won't remove the worker it will just mark it as in error
(Which you can't see with the actual mod_cluster-manager see
MODCLUSTER-92) and you need a request that goes to the node to have the
status updated.
In the logic that checks for removed nodes it would be possible to ping
nodes that haven't been updated after a timeout (no STATUS for while)
and decide to remove them based on the result of the ping.
Cheers
Jean-Frederic
7:57 a.m.
Brian Stansberry wrote:
I don't see why a hard kill would cause 5xx responses to the
client
(except perhaps for a few requests that were being handled by the killed
server when it was killed.) The httpd side should try to route future
requests to the failed node and when it can't connect fail them over.
"Very long timeouts" on subsequent requests sounds like something that
should solvable via configuration as well. (Paul, JFC please comment.)
The backend node is dead; there's no reason it should be taking the
httpd side a "very long" time to detect that when a new request comes in.
(I'm not saying MODCLUSTER-66 isn't important; just that the symptoms
you are describing seem more severe than is warranted. If people aren't
using HAModClusterService, the MODCLUSTER-66 fix won't help them, but
they still shouldn't be seeing what you describe.)
Right. The errors I've been seeing are probably caused by a long timeout
value for ping/pong.
When the socket connection to worker W is closed, doesn't
httpd/mod-cluster automatically remove W from its worker list ? Can this
be configured ?
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
5372
days inactive
5377
days old
mod_cluster-dev@lists.jboss.org
11 comments
5 participants
participants (5)
-
Bela Ban -
Brian Stansberry -
jean-frederic clere -
Marek Goldmann -
Paul Ferraro