[infinispan-issues] [JBoss JIRA] (ISPN-7559) Exception while marshalling object: Cache manager is shutting down

Exception while marshalling object: Cache manager is shutting down ------------------------------------------------------------------ Key: ISPN-7559 URL: https://issues.jboss.org/browse/ISPN-7559 Project: Infinispan Issue Type: Bug Components: Core Affects Versions: 8.2.6.Final Reporter: Radoslav Husar Assignee: Paul Ferraro Labels: infinispan_clean_shutdown In failover tests we saw an error on *server*. It occured during server stopping and application undeployment. Server log stacktrace: {code} 12:54:10,382 ERROR [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (thread-19,ee,perf18) ISPN000065: Exception while marshalling object: ExceptionResponse: java.io.IOException: ISPN000239: Cache manager is shutting down, so type write externalizer for type=java.io.IOException cannot be resolved [JBossINF] at org.infinispan.marshall.core.ExternalizerTable.getObjectWriter(ExternalizerTable.java:197) [JBossINF] at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:132) [JBossINF] at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58) [JBossINF] at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111) [JBossINF] at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectToObjectStream(AbstractJBossMarshaller.java:72) [JBossINF] at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectToBuffer(AbstractJBossMarshaller.java:80) [JBossINF] at org.infinispan.commons.marshall.AbstractMarshaller.objectToBuffer(AbstractMarshaller.java:41) [JBossINF] at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectToBuffer(MarshallerAdapter.java:23) [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.reply(CommandAwareRpcDispatcher.java:248) [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:183) [JBossINF] at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455) [JBossINF] at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406) [JBossINF] at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357) [JBossINF] at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245) [JBossINF] at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664) [JBossINF] at org.jgroups.JChannel.up(JChannel.java:738) [JBossINF] at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120) [JBossINF] at org.jgroups.stack.Protocol.up(Protocol.java:380) [JBossINF] at org.jgroups.protocols.FORK.up(FORK.java:114) [JBossINF] at org.jgroups.protocols.FRAG2.up(FRAG2.java:165) [JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:374) [JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:390) [JBossINF] at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1037) [JBossINF] at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234) [JBossINF] at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:442) [JBossINF] at org.jgroups.protocols.pbcast.NAKACK2.deliver(NAKACK2.java:968) [JBossINF] at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:848) [JBossINF] at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:611) [JBossINF] at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155) [JBossINF] at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:200) [JBossINF] at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:325) [JBossINF] at org.jgroups.protocols.MERGE3.up(MERGE3.java:292) [JBossINF] at org.jgroups.protocols.Discovery.up(Discovery.java:296) [JBossINF] at org.jgroups.protocols.TP.passMessageUp(TP.java:1657) [JBossINF] at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1873) [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [JBossINF] at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52) [JBossINF] at java.lang.Thread.run(Thread.java:745) [JBossINF] Caused by: an exception which occurred: [JBossINF] in object java.io.IOException@314c010 [JBossINF] -> toString = java.io.IOException: ISPN000239: Cache manager is shutting down, so type write externalizer for type=org.infinispan.remoting.responses.ExceptionResponse cannot be resolved [JBossINF] [JBossINF] 12:54:10,585 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 79) ISPN000030: Passivated 4000 entries in 364 milliseconds 12:54:10,594 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 79) WFLYCLINF0003: Stopped clusterbench-ee7.ear/clusterbench-ee7-ejb.jar cache from ejb container {code} Occurs after {code} 12:54:10,371 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000082: Stopping the RpcDispatcher for channel web {code} There are NoSuchEJBException error messages on client at the same time in scenario eap-7x-failover-ejb-ejbremote-shutdown-repl-async: {code} 12:54:10,468 INFO [org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler] (Remoting "config-based-ejb-client-endpoint" task-1) Retrying invocation which failed on node perf18 with exception: javax.ejb.NoSuchEJBException: No such EJB[appname=clusterbench-ee7,modulename=clusterbench-ee7-ejb,distinctname=,beanname=RemoteStatefulSBImpl,viewclassname=org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSB] at org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler.processMessage(NoSuchEJBExceptionResponseHandler.java:64) at org.jboss.ejb.client.remoting.ChannelAssociation.processResponse(ChannelAssociation.java:395) at org.jboss.ejb.client.remoting.ChannelAssociation$ResponseReceiver.handleMessage(ChannelAssociation.java:507) at org.jboss.remoting3.remote.RemoteConnectionChannel.lambda$handleMessageData$3(RemoteConnectionChannel.java:434) at org.jboss.remoting3.EndpointImpl$TrackingExecutor.lambda$execute$0(EndpointImpl.java:789) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) {code} and WARN messages in scenario eap-7x-failover-ejb-ejbservlet-undeploy-repl-sync {code} 14:50:29:275 EST [WARN ][Runner - 159] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Error sampling data: <org.jboss.smartfrog.loaddriver.RequestProcessingException: Invalid response code: 500 Content: <html><head><title>Error</title></head><body>Internal Server Error</body></html>> org.jboss.smartfrog.loaddriver.RequestProcessingException: Invalid response code: 500 Content: <html><head><title>Error</title></head><body>Internal Server Error</body></html> at org.jboss.smartfrog.loaddriver.http.HttpRequestProcessorFactoryImpl$HttpRequestProcessor.processRequest(HttpRequestProcessorFactoryImpl.java:163) at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:52) at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:103) at java.lang.Thread.run(Thread.java:745) {code} Scenarios affected: eap-7x-failover-ejb-ejbremote-shutdown-repl-async (1 occurence) eap-7x-failover-ejb-ejbservlet-undeploy-repl-sync (1 occurence) eap-7x-failover-jvmkill-ha-ss (1 occurence) Link to server log: http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/perflab_eap-7x-failov...