IllegalLifecycleStateException due to Cache marshaller has been stopped ----------------------------------------------------------------------- Key: ISPN-7558 URL: https://issues.jboss.org/browse/ISPN-7558 Project: Infinispan Issue Type: Bug Components: Core Affects Versions: 8.2.6.Final Reporter: Radoslav Husar Assignee: Paul Ferraro Priority: Minor We saw this error in *failover* scenarios: {code} 14:28:41,783 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 85) WFLYCLINF0003: Stopped client-mappings cache from ejb container [JBossINF] 14:28:41,783 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 77) WFLYCLINF0003: Stopped routing cache from web container [JBossINF] 14:28:41,790 INFO [org.infinispan.CLUSTER] (remote-thread--p8-t3) [Context=client-mappings][Scope=perf20]ISPN100003: Finished local rebalance [JBossINF] 14:28:41,791 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000080: Disconnecting JGroups channel ejb [JBossINF] 14:28:41,791 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000082: Stopping the RpcDispatcher for channel ejb [JBossINF] 14:28:41,792 INFO [org.infinispan.CLUSTER] (remote-thread--p7-t6) [Context=routing][Scope=perf20]ISPN100003: Finished local rebalance [JBossINF] 14:28:41,797 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000080: Disconnecting JGroups channel web [JBossINF] 14:28:41,799 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000082: Stopping the RpcDispatcher for channel web [JBossINF] 14:28:41,818 ERROR [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (thread-19) ISPN000065: Exception while marshalling object: CacheNotFoundResponse: org.infinispan.IllegalLifecycleStateException: Cache marshaller has been stopped [JBossINF] at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.getExternalizerTable(JBossMarshaller.java:155) [JBossINF] at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.getObjectWriter(JBossMarshaller.java:144) [JBossINF] at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:132) [JBossINF] at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58) [JBossINF] at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111) [JBossINF] at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectToObjectStream(AbstractJBossMarshaller.java:72) [JBossINF] at org.infinispan.marshall.core.VersionAwareMarshaller.objectToBuffer(VersionAwareMarshaller.java:77) [JBossINF] at org.infinispan.commons.marshall.AbstractMarshaller.objectToBuffer(AbstractMarshaller.java:41) [JBossINF] at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectToBuffer(AbstractDelegatingMarshaller.java:85) [JBossINF] at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectToBuffer(MarshallerAdapter.java:23) [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.reply(CommandAwareRpcDispatcher.java:248) [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.lambda$executeCommandFromLocalCluster$1(CommandAwareRpcDispatcher.java:199) [JBossINF] at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:121) [JBossINF] at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75) [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:200) [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:169) [JBossINF] at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455) [JBossINF] at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406) [JBossINF] at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357) [JBossINF] at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245) [JBossINF] at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664) [JBossINF] at org.jgroups.JChannel.up(JChannel.java:738) [JBossINF] at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120) [JBossINF] at org.jgroups.stack.Protocol.up(Protocol.java:380) [JBossINF] at org.jgroups.protocols.FORK.up(FORK.java:114) [JBossINF] at org.jgroups.protocols.FRAG2.up(FRAG2.java:165) [JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:374) [JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:390) [JBossINF] at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1037) [JBossINF] at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234) [JBossINF] at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:442) [JBossINF] at org.jgroups.protocols.pbcast.NAKACK2.deliver(NAKACK2.java:968) [JBossINF] at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:848) [JBossINF] at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:611) [JBossINF] at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155) [JBossINF] at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:200) [JBossINF] at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:325) [JBossINF] at org.jgroups.protocols.MERGE3.up(MERGE3.java:292) [JBossINF] at org.jgroups.protocols.Discovery.up(Discovery.java:296) [JBossINF] at org.jgroups.protocols.TP.passMessageUp(TP.java:1657) [JBossINF] at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1872) [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [JBossINF] at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52) [JBossINF] at java.lang.Thread.run(Thread.java:745) [JBossINF] Caused by: an exception which occurred: [JBossINF] in object org.infinispan.IllegalLifecycleStateException@244b87df [JBossINF] -> toString = org.infinispan.IllegalLifecycleStateException: Cache marshaller has been stopped [JBossINF] [JBossINF] 14:28:41,912 INFO [org.jboss.as.server] (management-handler-thread - 1) WFLYSRV0009: Undeployed "clusterbench-ee7.ear" (runtime-name: "clusterbench-ee7.ear") {code} Error occurs on *server* in three *async* scenarios during server stopping. Client is not affected. Scenarios affected: ejb-ejbremote-shutdown-repl-async (2 occurences) ejb-ejbremote-undeploy-dist-async (1 occurence) ejb-ejbservlet-undeploy-repl-sync (2 occurences) http-granular-shutdown-dist-sync (1 occurence) http-session-shutdown-repl-async (1 occurence) http-session-shutdown-concurrent-async (1 occurence) Link to server log: http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_eap-7x-fa...