SingleKeyBackupWriteCommands run out of memory ---------------------------------------------- Key: ISPN-12435 URL: https://issues.redhat.com/browse/ISPN-12435 Project: Infinispan Issue Type: Bug Components: Cloud Affects Versions: 10.1.8.Final Reporter: mac peng Priority: Blocker Attachments: The fragile in TriangleOrderManager.png SingleKeyBackupWriteCommands run out of memory if sender does send out single SingleKeyBackupWriteCommand for some reason(Exception or deadlock during org.infinispan.interceptors.distribution.TriangleDistributionInterceptor.sendToBackups {color:#FF0000} long sequence = triangleOrderManager.next(segmentId, topologyId);{color} //{color:#FF0000}Sequence is increased but the cmd has no chance to send out for there is exception/deadlock later.{color} BackupWriteCommand backupCommand = backupBuilder.build(commandsFactory, command, entry.getValue()); backupCommand.setSequence(sequence); backupCommand.setSegmentId(segmentId); if (trace) { log.tracef("Command %s got sequence %s for segment %s", command.getCommandInvocationId(), segmentId, sequence); } {color:#FF0000}rpcManager.sendToMany(backups, backupCommand, DeliverOrder.NONE);{color} 16-Oct-2020 01:09:38.403 ERROR [org.infinispan.remoting.rpc.RpcManagerImpl.rethrowAsCacheException] (dp-27-thread-1) ISPN000073: Unexpected error while re plicating {color:#FF0000}java.lang.RuntimeException: What's happen if Externalizer has exception?{color} {color:#FF0000} at com.ericsson.container.sip.fsm.InviteSessionFsm$InviteSessionExternalizer.writeObject(InviteSessionFsm.java:397){color} at com.ericsson.container.sip.fsm.InviteSessionFsm$InviteSessionExternalizer.writeObject(InviteSessionFsm.java:385) at com.ericsson.sessiondataservices.serialization.AdvancedExternalizerBridge.writeObject(AdvancedExternalizerBridge.java:42) at org.infinispan.marshall.core.impl.ExternalExternalizers$ForeignAdvancedExternalizer.writeObject(ExternalExternalizers.java:86) at org.infinispan.marshall.core.GlobalMarshaller.writeExternal(GlobalMarshaller.java:670) at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:375) at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:352) at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) at com.ericsson.ssa.sip.UA$UAExternalizer.writeObject(UA.java:930) at com.ericsson.ssa.sip.UA$UAExternalizer.writeObject(UA.java:919) at com.ericsson.sessiondataservices.serialization.AdvancedExternalizerBridge.writeObject(AdvancedExternalizerBridge.java:42) at org.infinispan.marshall.core.impl.ExternalExternalizers$ForeignAdvancedExternalizer.writeObject(ExternalExternalizers.java:86) at org.infinispan.marshall.core.GlobalMarshaller.writeExternal(GlobalMarshaller.java:670) at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:375) at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:352) at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) at com.ericsson.container.sip.session.dialog.DialogFragmentData$DialogFragmentDataExternalizer.writeObject(DialogFragmentData.java:780) at com.ericsson.container.sip.session.dialog.DialogFragmentData$DialogFragmentDataExternalizer.writeObject(DialogFragmentData.java:761) at com.ericsson.sessiondataservices.serialization.AdvancedExternalizerBridge.writeObject(AdvancedExternalizerBridge.java:42) at org.infinispan.marshall.core.impl.ExternalExternalizers$ForeignAdvancedExternalizer.writeObject(ExternalExternalizers.java:86) at org.infinispan.marshall.core.GlobalMarshaller.writeExternal(GlobalMarshaller.java:670) at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:375) at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:352) at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) at com.ericsson.container.sip.session.dialog.DialogFragmentData$DialogFragmentDataExternalizer.writeObject(DialogFragmentData.java:780) at com.ericsson.container.sip.session.dialog.DialogFragmentData$DialogFragmentDataExternalizer.writeObject(DialogFragmentData.java:761) at com.ericsson.sessiondataservices.serialization.AdvancedExternalizerBridge.writeObject(AdvancedExternalizerBridge.java:42) at org.infinispan.marshall.core.impl.ExternalExternalizers$ForeignAdvancedExternalizer.writeObject(ExternalExternalizers.java:86) at org.infinispan.marshall.core.GlobalMarshaller.writeExternal(GlobalMarshaller.java:670) at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:375) at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:352) at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) at org.infinispan.commands.triangle.SingleKeyBackupWriteCommand.writeTo(SingleKeyBackupWriteCommand.java:108) at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:70) at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:122) at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:118) at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:69) at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:656) at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:371) at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:352) at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:181) at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:174) at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:302) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1075) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1272) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendToMany(JGroupsTransport.java:277) at org.infinispan.remoting.rpc.RpcManagerImpl.sendToMany(RpcManagerImpl.java:396) {color:#FF0000}at org.infinispan.interceptors.distribution.TriangleDistributionInterceptor.sendToBackups(TriangleDistributionInterceptor.java:490){color} at org.infinispan.interceptors.distribution.TriangleDistributionInterceptor.lambda$localPrimaryOwnerWrite$4(TriangleDistributionInterceptor.java:446) The receiver won't handle the subsquent SingleKeyBackupWriteCommands for current SingleKeyBackupWriteCommand is missing. Then the  subsquent SingleKeyBackupWriteCommands will be putted into blockedTasks.offer(runnable); by BlockingTaskAwareExecutorServiceImpl instead of execute it.