5:16 a.m.
[
https://issues.jboss.org/browse/WFLY-5011?page=com.atlassian.jira.plugin....
]
Michal Karm Babacek commented on WFLY-5011:
-------------------------------------------
[~pferraro], [~mvinkler] Apart from {{OutdatedTopologyException}}, I experience also the
infamous {{ISPN000329: Unable to read rebalancing status from coordinator}}. I have a
hunch these are closely related; hence this comment and not a new JIRA.
The test is dead simple, with no load whatsoever -- merely several requests to establish
sessions.
* balancer: Apache HTTP Server with mod_cluster
* worker: {{jboss-eap-7.0}}
[jboss-eap-7.0.log|https://gist.github.com/Karm/7fe27a2c9aaac40e1b0d#file-...]
* worker: {{jboss-eap-7.0-2}}
[jboss-eap-7.0-2.log|https://gist.github.com/Karm/7fe27a2c9aaac40e1b0d#fil...]
* worker: {{jboss-eap-7.0-3}}
[jboss-eap-7.0-3.log|https://gist.github.com/Karm/7fe27a2c9aaac40e1b0d#fil...]
All workers run; {{jboss-eap-7.0}} worker is killed, i.e. its JVM is killed; then
{{jboss-eap-7.0-2}} worker, to which the failover occurred, stars returning HTTP 500
Errors due to:
{noformat}2015-10-20 14:03:47,293 ERROR [io.undertow.request] (default task-2) UT005023:
Exception handling request to /clusterbench/requestinfo/:
org.infinispan.statetransfer.OutdatedTopologyException: Cache not running on node
jboss-eap-7.0, or the node is missing{noformat} Simultaneously, {{jboss-eap-7.0-3}} worker
starts complaining about the dead coordinator:{noformat}2015-10-20 14:03:47,306 WARN
[org.infinispan.topology.ClusterTopologyManagerImpl] (ServerService Thread Pool -- 64)
ISPN000329: Unable to read rebalancing status from coordinator jboss-eap-7.0:
org.infinispan.remoting.transport.jgroups.SuspectException: Cache not running on node
jboss-eap-7.0
at
org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:46)
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:753)
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$80(JGroupsTransport.java:589)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
at
org.infinispan.remoting.transport.jgroups.SingleResponseFuture.futureDone(SingleResponseFuture.java:30)
at org.jgroups.blocks.Request.checkCompletion(Request.java:169)
at org.jgroups.blocks.UnicastRequest.viewChange(UnicastRequest.java:164)
at org.jgroups.blocks.RequestCorrelator.receiveView(RequestCorrelator.java:331)
at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:242)
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:684)
at org.jgroups.JChannel.up(JChannel.java:738)
at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:123)
at org.jgroups.stack.Protocol.up(Protocol.java:374)
at org.jgroups.protocols.FORK.up(FORK.java:118)
at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:394)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:394)
at org.jgroups.protocols.pbcast.GMS.installView(GMS.java:735)
at
org.jgroups.protocols.pbcast.ParticipantGmsImpl.handleViewChange(ParticipantGmsImpl.java:140)
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:925)
at org.jgroups.stack.Protocol.up(Protocol.java:412)
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:294)
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:474)
at org.jgroups.protocols.pbcast.NAKACK2.deliverBatch(NAKACK2.java:982)
at org.jgroups.protocols.pbcast.NAKACK2.removeAndPassUp(NAKACK2.java:912)
at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:846)
at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:618)
at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:200)
at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
at org.jgroups.protocols.Discovery.up(Discovery.java:295)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1577)
at org.jgroups.protocols.TP$MyHandler.run(TP.java:1796)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}
h3. Question
Could I white-list the *ISPN000329* exception in my tests or is it actually related to the
grave problem of {{OutdatedTopologyExceptions}} on the other node? Last but not least,
lemme stress one more time that there is virtually no load whatsoever; this is a
functional test with a handful of requests running on a one physical Solaris machine.
OutdatedTopologyException: Cache not running on node
<node_name>
----------------------------------------------------------------
Key: WFLY-5011
URL: https://issues.jboss.org/browse/WFLY-5011
Project: WildFly
Issue Type: Bug
Components: Clustering
Affects Versions: 10.0.0.Alpha6, 10.0.0.Beta1, 10.0.0.CR2
Reporter: Michal Vinkler
Assignee: Paul Ferraro
Priority: Critical
Seen in our HTTP-based failover tests (no matter what failover type:
jvmkill/shutdown/undeploy), with *distributed* cache used.
Doe not occur when replicated cache is used.
Setup: 4 node cluster, one node at time is shutdown, while standalone clients keep
calling the application.
After failing one node(failover type: jvmkill/shutdown/undeploy) - perf18 for example,
perf19,20,21 log the following error message many times (seems like one error per each
session)
{code}
[JBossINF] [0m[31m05:48:40,646 ERROR [io.undertow.request] (default task-110) UT005023:
Exception handling request to /clusterbench/session:
org.infinispan.statetransfer.OutdatedTopologyException: Cache not running on node perf18
[JBossINF] at
org.infinispan.interceptors.distribution.TxDistributionInterceptor.checkTxCommandResponses(TxDistributionInterceptor.java:274)
[JBossINF] at
org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:186)
[JBossINF] at
org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
[JBossINF] at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
[JBossINF] at
org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
[JBossINF] at
org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:174)
[JBossINF] at
org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
[JBossINF] at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
[JBossINF] at
org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
[JBossINF] at
org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:174)
[JBossINF] at
org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
[JBossINF] at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
[JBossINF] at
org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
[JBossINF] at
org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:174)
[JBossINF] at
org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
[JBossINF] at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
[JBossINF] at
org.infinispan.interceptors.locking.PessimisticLockingInterceptor.acquireRemoteIfNeeded(PessimisticLockingInterceptor.java:238)
[JBossINF] at
org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:66)
[JBossINF] at
org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:70)
[JBossINF] at
org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
[JBossINF] at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
[JBossINF] at
org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
[JBossINF] at
org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:86)
[JBossINF] at
org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
[JBossINF] at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
[JBossINF] at
org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:346)
[JBossINF] at
org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:318)
[JBossINF] at
org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
[JBossINF] at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
[JBossINF] at
org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
[JBossINF] at
org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:86)
[JBossINF] at
org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
[JBossINF] at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
[JBossINF] at
org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:364)
[JBossINF] at
org.infinispan.statetransfer.StateTransferInterceptor.handleDefault(StateTransferInterceptor.java:349)
[JBossINF] at
org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:86)
[JBossINF] at
org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
[JBossINF] at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
[JBossINF] at
org.infinispan.interceptors.CacheMgmtInterceptor.visitDataReadCommand(CacheMgmtInterceptor.java:103)
[JBossINF] at
org.infinispan.interceptors.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:91)
[JBossINF] at
org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
[JBossINF] at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
[JBossINF] at
org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
[JBossINF] at
org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
[JBossINF] at
org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:86)
[JBossINF] at
org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
[JBossINF] at
org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:430)
[JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:427)
[JBossINF] at
org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:287)
[JBossINF] at
org.wildfly.clustering.web.infinispan.session.coarse.CoarseSessionFactory.findValue(CoarseSessionFactory.java:120)
[JBossINF] at
org.wildfly.clustering.web.infinispan.session.coarse.CoarseSessionFactory.findValue(CoarseSessionFactory.java:56)
[JBossINF] at
org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:232)
[JBossINF] at
org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:116)
[JBossINF] at
io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:725)
[JBossINF] at
io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:367)
[JBossINF] at
org.jboss.weld.servlet.SessionHolder.requestInitialized(SessionHolder.java:47)
[JBossINF] at
org.jboss.weld.servlet.HttpContextLifecycle.requestInitialized(HttpContextLifecycle.java:231)
[JBossINF] at
org.jboss.weld.servlet.WeldInitialListener.requestInitialized(WeldInitialListener.java:152)
[JBossINF] at
io.undertow.servlet.core.ApplicationListeners.requestInitialized(ApplicationListeners.java:216)
[JBossINF] at
io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:281)
[JBossINF] at
io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:261)
[JBossINF] at
io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:80)
[JBossINF] at
io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:172)
[JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:199)
[JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:774)
[JBossINF] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[JBossINF] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[JBossINF] at java.lang.Thread.run(Thread.java:745)
{code}
Server log:
http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-http-...
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
3894
days inactive
3894
days old
0 comments
1 participants
participants (1)
-
Michal Karm Babacek (JIRA)