Configuring checking a crashed worker
by Bela Ban
*Which properties do I have to set (I guess in mod-cluster-jboss.beans
in HAModClusterConfig) to remove crashed workers from http's tables ?*
I want httpd (mod-cluster) to periodically try to connect to a worker
and remove the worker is the connection is dead.
In mod_jk, I guess (according to [1]) I'd have to set ping_mode=I, with
ping_timeout being the interval and connection_ping_interval to a value
in secs after which a conn will be checked if idle.
I don't know which props I have to use with mod_cluster: nodeTimeout ?
[2] I guess that's only for open sockets though...
[1] http://tomcat.apache.org/connectors-doc/reference/workers.html
[2] http://jboss.org/mod_cluster/native/mod_jk.html
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
16 years, 2 months
How are sessions created across domains ?
by Bela Ban
I have domains A and B. I see that all sessions are created in domain A,
and none in domain B. Only when I disable all nodes in domain A, will
sessions get created in domain B.
Is this a bug, or can I have equal creation of sessions across multiple
domains ?
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
16 years, 2 months
Disabling of workers in mod_cluster_manager
by Bela Ban
When I disable a worker (or all workers in a given domain) in
mod_cluster_manager, requests to *existing* sessions get rerouted to
other workers.
This is IMO *not* what DISABLE should do (it is what STOP used to do in
/status) ! DISABLE should *not* remove a worker from the worker tables,
but have the following behavior:
* Sticky sessions
o Requests for existing sessions: route them to the existing
workers to which they were tied (also disabled ones !)
o New sessions are *not* created on disabled workers, but only
on enabled ones
* Non-sticky sessions
o Route requests to any enabled worker
o No requests are sent to disabled workers
WDYT ?
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
16 years, 2 months
mod-cluster and kill -9
by Bela Ban
Hi Paul,
wanted to reiterate the importance of [1] for the next release of
mod-cluster.
I'm constantly running into this when I deploy httpd/mod-cluster and
JBoss 5.1.0 on Amazon's EC2 cloud. The easy way to stop an instance (OS
+ JBoss) on EC2 is to 'terminate' it via the AWS Console, which shuts
down the OS ("shutdown -h now").
Unfortunately, our AMIs only had an S98jboss in /etc/rc4.d for starting
JBoss, but no corresponding K98jboss for stopping it *gracefully* on
shutdown. Therefore the process was always killed via -9.
This caused very long timeouts and 5XX HTTP responses, until
httd/mod-cluster finally figured out that the worker crashed and failed
over to a different worker.
As a workaround, I created a K98jboss link so now JBoss is shut down
gracefully when the host is terminated.
However, I figure we can get into this situation in many different ways,
e.g.
* Not providing a K98jboss script on EC2
* Killing JBoss with -9 via a script (I've seen this many more than once
* Pulling a blade out of the rack. A crude way of shutting down an
instance, but that's normal in large clusters !
Can we have this feature in mod-cluster 1.1 ?
[1] https://jira.jboss.org/jira/browse/MODCLUSTER-66
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
16 years, 2 months
503 error when JBoss is bound to 0.0.0.0
by Marek Goldmann
I'm encountered a strange error. When I bind JBoss instance to 0.0.0.0
address instead of a fixed ethernet address, node gets registered in
mod_cluster, shows in mod_cluster-manager, but every request to
registered contexts throws 503 error.
httpd error log:
[Fri Aug 07 03:21:05 2009] [error] (111)Connection refused: proxy:
ajp: attempt to connect to 0.0.0.0:8009 (0.0.0.0) failed
[Fri Aug 07 03:21:05 2009] [error] ap_proxy_connect_backend disabling
worker for (0.0.0.0)
[Fri Aug 07 03:21:15 2009] [error] proxy: ajp: disabled connection for
(0.0.0.0)
[Fri Aug 07 03:21:25 2009] [error] proxy: ajp: disabled connection for
(0.0.0.0)
This looks like a bug for me, because many administrators are binding
JBoss to 0.0.0.0.
--
Marek Goldmann
JBoss by Red Hat
16 years, 2 months
Relocate svn repo?
by Paul Ferraro
Today, Jean-Frederic and I discussed creating an svn branch for 1.0.x
work so that work on 1.1 can proceed in trunk. However, this brings up
another issue...
Currently, we share our subversion repository with the jboss native
project. Our source code resides in an awkward subdirectory of trunk,
making branching (not to mention the maven release process) equally
awkward. Now would seem to be as good a time as any to move to our own
repository. How would this impact the build of the native side? If we
keep its current location, should this branch only copy from
trunk/mod_cluster? Thoughts?
Paul
16 years, 3 months
ClastCastException on redeployment of mod-cluster.sar
by Bela Ban
When I touched mod-cluster.sar/META-INF/mod-cluster-jboss-beans.xml, I
got the exception below.
Can someone look into this ?
2009-07-31 09:52:40,084 INFO
[org.jboss.web.tomcat.service.deployers.TomcatDeployment] (HDScanner)
undeploy, ctxPath=/invoker
2009-07-31 09:52:40,168 INFO
[org.jboss.web.tomcat.service.deployers.TomcatDeployment] (HDScanner)
undeploy, ctxPath=/juddi
2009-07-31 09:52:40,170 INFO
[org.apache.juddi.registry.RegistryServlet] (HDScanner) jUDDI Stopping:
Cleaning up existing resources.
2009-07-31 09:52:40,172 INFO
[org.jboss.web.tomcat.service.deployers.TomcatDeployment] (HDScanner)
undeploy, ctxPath=/jbossws
2009-07-31 09:52:40,176 INFO
[org.jboss.web.tomcat.service.deployers.TomcatDeployment] (HDScanner)
undeploy, ctxPath=/web-console
2009-07-31 09:52:40,182 INFO
[org.jboss.web.tomcat.service.deployers.TomcatDeployment] (HDScanner)
undeploy, ctxPath=/admin-console
2009-07-31 09:52:40,186 INFO
[org.jboss.web.tomcat.service.deployers.TomcatDeployment] (HDScanner)
undeploy, ctxPath=/jmx-console
2009-07-31 09:52:40,225 INFO [org.apache.coyote.http11.Http11Protocol]
(HDScanner) Pausing Coyote HTTP/1.1 on http-10.254.203.178-8080
2009-07-31 09:52:40,230 INFO [org.apache.coyote.http11.Http11Protocol]
(HDScanner) Stopping Coyote HTTP/1.1 on http-10.254.203.178-8080
2009-07-31 09:52:40,231 INFO [org.apache.coyote.ajp.AjpProtocol]
(HDScanner) Pausing Coyote AJP/1.3 on ajp-10.254.203.178-8009
2009-07-31 09:52:40,236 INFO [org.apache.coyote.ajp.AjpProtocol]
(HDScanner) Stopping Coyote AJP/1.3 on ajp-10.254.203.178-8009
2009-07-31 09:52:40,292 INFO [org.apache.catalina.core.StandardService]
(HDScanner) Stopping service jboss.web
2009-07-31 09:52:41,506 ERROR
[org.jboss.kernel.plugins.dependency.AbstractKernelController]
(HDScanner) Error installing to Start: name=HAModClusterService
state=Create mode=On Demand requiredState=Installed
java.lang.ClassCastException:
org.jboss.modcluster.ha.ModClusterServiceDRMEntry cannot be cast to
org.jboss.modcluster.ha.ModClusterServiceDRMEntry
at
org.jboss.modcluster.ha.HAModClusterService.narrowCandidateList(HAModClusterService.java:496)
at
org.jboss.modcluster.ha.HAModClusterService.getElectionCandidates(HAModClusterService.java:477)
at
org.jboss.ha.framework.server.HASingletonImpl.election(HASingletonImpl.java:206)
at
org.jboss.ha.framework.server.HASingletonImpl.elected(HASingletonImpl.java:199)
at
org.jboss.ha.framework.server.HASingletonImpl.partitionTopologyChanged(HASingletonImpl.java:162)
at
org.jboss.ha.framework.server.HASingletonImpl.registerDRMListener(HASingletonImpl.java:124)
at
org.jboss.ha.framework.server.HAServiceImpl.start(HAServiceImpl.java:180)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:59)
at
org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:150)
at
org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66)
at
org.jboss.kernel.plugins.dependency.KernelControllerContextAction$JoinpointDispatchWrapper.execute(KernelControllerContextAction.java:241)
at
org.jboss.kernel.plugins.dependency.ExecutionWrapper.execute(ExecutionWrapper.java:47)
at
org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchExecutionWrapper(KernelControllerContextAction.java:109)
at
org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchJoinPoint(KernelControllerContextAction.java:70)
at
org.jboss.kernel.plugins.dependency.LifecycleAction.installActionInternal(LifecycleAction.java:221)
at
org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:54)
at
org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:42)
at
org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62)
at
org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71)
at
org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51)
at
org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)
at
org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1631)
at
org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:934)
at
org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)
at
org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:984)
at
org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:822)
at
org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:553)
at
org.jboss.system.ServiceController.doChange(ServiceController.java:688)
at
org.jboss.system.ServiceController.start(ServiceController.java:460)
at
org.jboss.system.microcontainer.jmx.ServiceControllerStartStopLifecycleCallback.install(ServiceControllerStartStopLifecycleCallback.java:44)
at sun.reflect.GeneratedMethodAccessor252.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:59)
at
org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:150)
at
org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66)
at
org.jboss.beans.info.plugins.AbstractBeanInfo.invoke(AbstractBeanInfo.java:300)
at
org.jboss.kernel.plugins.dependency.AbstractKernelControllerContext.invoke(AbstractKernelControllerContext.java:286)
at
org.jboss.dependency.plugins.AbstractLifecycleCallbackItem.install(AbstractLifecycleCallbackItem.java:87)
at
org.jboss.dependency.plugins.AbstractController.handleLifecycleCallbacks(AbstractController.java:1568)
at
org.jboss.dependency.plugins.AbstractController.handleInstallLifecycleCallbacks(AbstractController.java:1533)
at
org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:943)
at
org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)
at
org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:984)
at
org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:822)
at
org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:553)
at
org.jboss.system.ServiceController.doChange(ServiceController.java:688)
at
org.jboss.system.ServiceController.start(ServiceController.java:460)
at
org.jboss.system.microcontainer.jmx.ServiceControllerStartStopLifecycleCallback.install(ServiceControllerStartStopLifecycleCallback.java:44)
at sun.reflect.GeneratedMethodAccessor252.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:59)
at
org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:150)
at
org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66)
at
org.jboss.beans.info.plugins.AbstractBeanInfo.invoke(AbstractBeanInfo.java:300)
at
org.jboss.kernel.plugins.dependency.AbstractKernelControllerContext.invoke(AbstractKernelControllerContext.java:286)
at
org.jboss.dependency.plugins.AbstractLifecycleCallbackItem.install(AbstractLifecycleCallbackItem.java:87)
at
org.jboss.dependency.plugins.AbstractController.handleLifecycleCallbacks(AbstractController.java:1568)
at
org.jboss.dependency.plugins.AbstractController.handleInstallLifecycleCallbacks(AbstractController.java:1533)
at
org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:943)
at
org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)
at
org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:984)
at
org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:774)
at
org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:540)
at
org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer.deploy(BeanMetaDataDeployer.java:121)
at
org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer.deploy(BeanMetaDataDeployer.java:51)
at
org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalDeploy(AbstractSimpleRealDeployer.java:62)
at
org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:50)
at
org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:171)
at
org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(DeployersImpl.java:1439)
at
org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1157)
at
org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1178)
at
org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:1098)
at
org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)
at
org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1631)
at
org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:934)
at
org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)
at
org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:984)
at
org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:822)
at
org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:553)
at
org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:781)
at
org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:702)
at
org.jboss.system.server.profileservice.repository.MainDeployerAdapter.process(MainDeployerAdapter.java:117)
at
org.jboss.system.server.profileservice.hotdeploy.HDScanner.scan(HDScanner.java:362)
at
org.jboss.system.server.profileservice.hotdeploy.HDScanner.run(HDScanner.java:255)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
2009-07-31 09:52:41,530 WARN
[org.jboss.system.server.profileservice.hotdeploy.HDScanner] (HDScanner)
Failed to process changes
org.jboss.deployers.client.spi.IncompleteDeploymentException: Summary of
incomplete deployments (SEE PREVIOUS ERRORS FOR DETAILS):
DEPLOYMENTS IN ERROR:
Deployment "HAModClusterService" is in error due to the following
reason(s): java.lang.ClassCastException:
org.jboss.modcluster.ha.ModClusterServiceDRMEntry cannot be cast to
org.jboss.modcluster.ha.ModClusterServiceDRMEntry
at
org.jboss.deployers.plugins.deployers.DeployersImpl.checkComplete(DeployersImpl.java:993)
at
org.jboss.deployers.plugins.deployers.DeployersImpl.checkComplete(DeployersImpl.java:939)
at
org.jboss.deployers.plugins.main.MainDeployerImpl.checkComplete(MainDeployerImpl.java:873)
at
org.jboss.system.server.profileservice.repository.MainDeployerAdapter.checkComplete(MainDeployerAdapter.java:128)
at
org.jboss.system.server.profileservice.hotdeploy.HDScanner.scan(HDScanner.java:369)
at
org.jboss.system.server.profileservice.hotdeploy.HDScanner.run(HDScanner.java:255)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
16 years, 3 months
