[infinispan-issues] [JBoss JIRA] (ISPN-5573) Node shutting down can throw IllegalLifecycleStateException wrapped multiple times

Node shutting down can throw IllegalLifecycleStateException wrapped multiple times ---------------------------------------------------------------------------------- Key: ISPN-5573 URL: https://issues.jboss.org/browse/ISPN-5573 Project: Infinispan Issue Type: Bug Components: Core Affects Versions: 8.0.0.Alpha2 Reporter: William Burns Assignee: William Burns Fix For: 8.0.0.Beta1 When working on some new work I have a stress test that constantly stops and starts a cache to test the resiliency of my code. I found that every once in a while if I have something that takes a little longer to serialize it could throw back a non SuspectException to the originator. It looks like the underlying issue is that IllegalLifecycleStateException is wrapped by multiple exceptions and thus it comes back as a simple CacheException instead which treats the exception properly. {code} 15:31:48,053 TRACE (OOB-2,main-NodeD-61136:) [ClusterStreamManagerImpl] Encounted exception for cf133b23-9bbf-49ee-823d-f2f650f39d9c from main-NodeH-22716 java.util.concurrent.CompletionException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from main-NodeH-22716, see cause for remote stack trace at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273) at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280) at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:824) at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1954) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.futureDone(SingleResponseFuture.java:27) at org.jgroups.blocks.Request.checkCompletion(Request.java:169) at org.jgroups.blocks.UnicastRequest.receiveResponse(UnicastRequest.java:83) at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:398) at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:250) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:675) at org.jgroups.JChannel.up(JChannel.java:739) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1029) at org.jgroups.protocols.RSVP.up(RSVP.java:201) at org.jgroups.protocols.FRAG2.up(FRAG2.java:165) at org.jgroups.protocols.FlowControl.up(FlowControl.java:394) at org.jgroups.protocols.tom.TOA.up(TOA.java:121) at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1042) at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234) at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1064) at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:779) at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426) at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:652) at org.jgroups.protocols.Discovery.up(Discovery.java:291) at org.jgroups.protocols.TP.passMessageUp(TP.java:1577) at org.jgroups.protocols.TP$MyHandler.run(TP.java:1796) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from main-NodeH-22716, see cause for remote stack trace at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:747) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$76(JGroupsTransport.java:586) at org.infinispan.remoting.transport.jgroups.JGroupsTransport$$Lambda$5/1861463967.apply(Unknown Source) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) ... 26 more Caused by: org.infinispan.commons.CacheException: java.lang.RuntimeException: Failure to marshal argument(s) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.rethrowAsCacheException(CommandAwareRpcDispatcher.java:127) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:119) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:565) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:180) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:222) at org.infinispan.stream.impl.LocalStreamManagerImpl.streamOperationRehashAware(LocalStreamManagerImpl.java:240) at org.infinispan.stream.impl.StreamRequestCommand.perform(StreamRequestCommand.java:93) at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokePerform(BasePerCacheInboundInvocationHandler.java:85) at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:34) ... 3 more Caused by: java.lang.RuntimeException: Failure to marshal argument(s) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.marshallCall(CommandAwareRpcDispatcher.java:238) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:252) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:116) ... 10 more Caused by: org.infinispan.IllegalLifecycleStateException: Cache marshaller has been stopped at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.getObjectWriter(JBossMarshaller.java:146) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:132) at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58) at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111) at org.infinispan.commons.marshall.MarshallUtil.marshallMap(MarshallUtil.java:36) at org.infinispan.marshall.exts.MapExternalizer.writeObject(MapExternalizer.java:60) at org.infinispan.marshall.exts.MapExternalizer.writeObject(MapExternalizer.java:28) at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:442) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:138) at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58) at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111) at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:58) at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:128) at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:112) at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:68) at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:442) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:138) at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58) at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111) at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectToObjectStream(AbstractJBossMarshaller.java:72) at org.infinispan.marshall.core.VersionAwareMarshaller.objectToBuffer(VersionAwareMarshaller.java:77) at org.infinispan.commons.marshall.AbstractMarshaller.objectToBuffer(AbstractMarshaller.java:41) at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectToBuffer(AbstractDelegatingMarshaller.java:85) at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectToBuffer(MarshallerAdapter.java:23) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.marshallCall(CommandAwareRpcDispatcher.java:236) ... 12 more Caused by: an exception which occurred: in object java.lang.Integer@20d5 in object java.util.HashMap@0 in object org.infinispan.stream.impl.StreamResponseCommand@66d77895 {code}