[Red Hat JIRA] (ISPN-12758) Lack of LIFECYCLE permission errors during startup
by Georg F (Jira)
Georg F created ISPN-12758:
------------------------------
Summary: Lack of LIFECYCLE permission errors during startup
Key: ISPN-12758
URL: https://issues.redhat.com/browse/ISPN-12758
Project: Infinispan
Issue Type: Bug
Components: Security
Affects Versions: 12.0.1.Final
Reporter: Georg F
I'm running 3 Infinispan nodes with distributed caches (2 cache owners). After enabling security with roles/permissions I get a lot of stack traces when an Infinispan node is starting up. The errors can only be observed in the beginning then caches are rebalancing. Once the startup is completed no more errors are shown and everything seems to be working as expected.
{noformat}
Feb 18 17:10:52 ispn-3 bash[261064]: 17:10:52,364 WARN (jgroups-26,ispn-3) [org.infinispan.CLUSTER] ISPN000071: Caught exception when handling command org.infinispan.manager.impl.ReplicableManagerFunctionCommand@3f7ffaaa
java.lang.SecurityException: ISPN000287: Unauthorized access: subject 'null' lacks 'LIFECYCLE' permission
Feb 18 17:10:52 ispn-3 bash[261064]: at org.infinispan.security.impl.AuthorizationHelper.checkPermission(AuthorizationHelper.java:98)
Feb 18 17:10:52 ispn-3 bash[261064]: at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:660)
Feb 18 17:10:52 ispn-3 bash[261064]: at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:643)
Feb 18 17:10:52 ispn-3 bash[261064]: at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:532)
Feb 18 17:10:52 ispn-3 bash[261064]: at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:510)
Feb 18 17:10:52 ispn-3 bash[261064]: at org.infinispan.manager.impl.AbstractDelegatingEmbeddedCacheManager.getCache(AbstractDelegatingEmbeddedCacheManager.java:193)
Feb 18 17:10:52 ispn-3 bash[261064]: at org.infinispan.manager.impl.UnwrappingEmbeddedCacheManager.getCache(UnwrappingEmbeddedCacheManager.java:25)
Feb 18 17:10:52 ispn-3 bash[261064]: at org.infinispan.notifications.cachelistener.cluster.ClusterListenerReplicateCallable.apply(ClusterListenerReplicateCallable.java:75)
Feb 18 17:10:52 ispn-3 bash[261064]: at org.infinispan.notifications.cachelistener.cluster.ClusterListenerReplicateCallable.apply(ClusterListenerReplicateCallable.java:39)
Feb 18 17:10:52 ispn-3 bash[261064]: at org.infinispan.manager.impl.ReplicableManagerFunctionCommand.invokeAsync(ReplicableManagerFunctionCommand.java:39)
Feb 18 17:10:52 ispn-3 bash[261064]: at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler$ReplicableCommandRunner.run(GlobalInboundInvocationHandler.java:255)
Feb 18 17:10:52 ispn-3 bash[261064]: at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleReplicableCommand(GlobalInboundInvocationHandler.java:174)
Feb 18 17:10:52 ispn-3 bash[261064]: at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:113)
Feb 18 17:10:52 ispn-3 bash[261064]: at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processRequest(JGroupsTransport.java:1383)
Feb 18 17:10:52 ispn-3 bash[261064]: at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1307)
Feb 18 17:10:52 ispn-3 bash[261064]: at org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$300(JGroupsTransport.java:131)
Feb 18 17:10:52 ispn-3 bash[261064]: at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1450)
Feb 18 17:10:52 ispn-3 bash[261064]: at org.jgroups.JChannel.up(JChannel.java:784)
Feb 18 17:10:52 ispn-3 bash[261064]: at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:913)
Feb 18 17:10:52 ispn-3 bash[261064]: at org.jgroups.protocols.FRAG3.up(FRAG3.java:165)
Feb 18 17:10:52 ispn-3 bash[261064]: at org.jgroups.protocols.FlowControl.up(FlowControl.java:351)
Feb 18 17:10:52 ispn-3 bash[261064]: at org.jgroups.protocols.FlowControl.up(FlowControl.java:359)
Feb 18 17:10:52 ispn-3 bash[261064]: at org.jgroups.protocols.pbcast.GMS.up(GMS.java:876)
Feb 18 17:10:52 ispn-3 bash[261064]: at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:243)
Feb 18 17:10:52 ispn-3 bash[261064]: at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1049)
Feb 18 17:10:52 ispn-3 bash[261064]: at org.jgroups.protocols.UNICAST3.addMessage(UNICAST3.java:772)
Feb 18 17:10:52 ispn-3 bash[261064]: at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:753)
Feb 18 17:10:52 ispn-3 bash[261064]: at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:405)
Feb 18 17:10:52 ispn-3 bash[261064]: at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:592)
Feb 18 17:10:52 ispn-3 bash[261064]: at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:132)
Feb 18 17:10:52 ispn-3 bash[261064]: at org.jgroups.protocols.FailureDetection.up(FailureDetection.java:186)
Feb 18 17:10:52 ispn-3 bash[261064]: at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:254)
Feb 18 17:10:52 ispn-3 bash[261064]: at org.jgroups.protocols.MERGE3.up(MERGE3.java:281)
Feb 18 17:10:52 ispn-3 bash[261064]: at org.jgroups.protocols.Discovery.up(Discovery.java:300)
Feb 18 17:10:52 ispn-3 bash[261064]: at org.jgroups.protocols.TP.passMessageUp(TP.java:1396)
Feb 18 17:10:52 ispn-3 bash[261064]: at org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87)
Feb 18 17:10:52 ispn-3 bash[261064]: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
Feb 18 17:10:52 ispn-3 bash[261064]: at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Feb 18 17:10:52 ispn-3 bash[261064]: at java.base/java.lang.Thread.run(Thread.java:834)
Feb 18 17:10:52 ispn-3 bash[261064]:
{noformat}
Similar errors occur when the /metrics endpoint is called during startup. Again, no issues once startup is completed:
{noformat}
Feb 18 16:28:49 ispn-2 bash[255577]: 16:28:49,126 ERROR (SINGLE_PORT-ServerIO-4-1) [org.infinispan.stats.impl.ClusterCacheStatsImpl] Could not execute cluster wide cache stats operation java.util.concurrent.CompletionExce[685/1918]
.infinispan.commons.CacheException: java.lang.SecurityException: ISPN000287: Unauthorized access: subject 'null' lacks 'LIFECYCLE' permission
Feb 18 16:28:49 ispn-2 bash[255577]: at java.base/java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:412)
Feb 18 16:28:49 ispn-2 bash[255577]: at java.base/java.util.concurrent.CompletableFuture.join(CompletableFuture.java:2044)
Feb 18 16:28:49 ispn-2 bash[255577]: at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:118)
Feb 18 16:28:49 ispn-2 bash[255577]: at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:117)
Feb 18 16:28:49 ispn-2 bash[255577]: at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:210)
Feb 18 16:28:49 ispn-2 bash[255577]: at org.infinispan.stats.impl.AbstractClusterStats.getStatAsLong(AbstractClusterStats.java:201)
Feb 18 16:28:49 ispn-2 bash[255577]: at org.infinispan.stats.impl.ClusterCacheStatsImpl.getAverageReadTime(ClusterCacheStatsImpl.java:155)
Feb 18 16:28:49 ispn-2 bash[255577]: at org.infinispan.factories.impl.MBeanMetadata$AttributeMetadata.lambda$getter$0(MBeanMetadata.java:137)
Feb 18 16:28:49 ispn-2 bash[255577]: at org.infinispan.metrics.impl.MetricsCollector.lambda$registerMetrics$0(MetricsCollector.java:139)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.smallrye.metrics.exporters.OpenMetricsExporter.createSimpleValueLine(OpenMetricsExporter.java:492)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.smallrye.metrics.exporters.OpenMetricsExporter.exposeEntries(OpenMetricsExporter.java:192)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.smallrye.metrics.exporters.OpenMetricsExporter.getEntriesForScope(OpenMetricsExporter.java:158)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.smallrye.metrics.exporters.OpenMetricsExporter.exportAllScopes(OpenMetricsExporter.java:109)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.smallrye.metrics.MetricsRequestHandler.handleRequest(MetricsRequestHandler.java:116)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.smallrye.metrics.MetricsRequestHandler.handleRequest(MetricsRequestHandler.java:73)
Feb 18 16:28:49 ispn-2 bash[255577]: at org.infinispan.rest.resources.MetricsResource.metrics(MetricsResource.java:92)
Feb 18 16:28:49 ispn-2 bash[255577]: at org.infinispan.rest.framework.impl.RestDispatcherImpl.dispatch(RestDispatcherImpl.java:75)
Feb 18 16:28:49 ispn-2 bash[255577]: at org.infinispan.rest.RestRequestHandler.handleRestRequest(RestRequestHandler.java:139)
Feb 18 16:28:49 ispn-2 bash[255577]: at org.infinispan.rest.RestRequestHandler.channelRead0(RestRequestHandler.java:90)
Feb 18 16:28:49 ispn-2 bash[255577]: at org.infinispan.rest.RestRequestHandler.channelRead0(RestRequestHandler.java:37)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.handler.codec.http.HttpServerKeepAliveHandler.channelRead(HttpServerKeepAliveHandler.java:64)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
Feb 18 16:28:49 ispn-2 bash[255577]: at org.infinispan.rest.CorsHandler.channelRead(CorsHandler.java:99)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
Feb 18 16:28:49 ispn-2 bash[255577]: at org.infinispan.rest.StreamCorrelatorHandler.channelRead(StreamCorrelatorHandler.java:27)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
Feb 18 16:28:49 ispn-2 bash[255577]: at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
Feb 18 16:28:49 ispn-2 bash[255577]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Feb 18 16:28:49 ispn-2 bash[255577]: at java.base/java.lang.Thread.run(Thread.java:834)
Feb 18 16:28:49 ispn-2 bash[255577]: Caused by: org.infinispan.commons.CacheException: java.lang.SecurityException: ISPN000287: Unauthorized access: subject 'null' lacks 'LIFECYCLE' permission
Feb 18 16:28:49 ispn-2 bash[255577]: at org.infinispan.stats.impl.ClusterCacheStatsImpl.lambda$updateStats$0(ClusterCacheStatsImpl.java:109)
Feb 18 16:28:49 ispn-2 bash[255577]: at org.infinispan.manager.impl.AllClusterExecutor.lambda$submitConsumer$6(AllClusterExecutor.java:192)
Feb 18 16:28:49 ispn-2 bash[255577]: at org.infinispan.manager.impl.AbstractClusterExecutor.consumeResponse(AbstractClusterExecutor.java:65)
Feb 18 16:28:49 ispn-2 bash[255577]: at org.infinispan.manager.impl.AllClusterExecutor.lambda$submitConsumer$7(AllClusterExecutor.java:191){noformat}
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
3 years, 2 months
[Red Hat JIRA] (ISPN-12435) SingleKeyBackupWriteCommands run out of memory
by Dan Berindei (Jira)
[ https://issues.redhat.com/browse/ISPN-12435?page=com.atlassian.jira.plugi... ]
Dan Berindei reassigned ISPN-12435:
-----------------------------------
Assignee: Dan Berindei
> SingleKeyBackupWriteCommands run out of memory
> ----------------------------------------------
>
> Key: ISPN-12435
> URL: https://issues.redhat.com/browse/ISPN-12435
> Project: Infinispan
> Issue Type: Bug
> Components: Cloud
> Affects Versions: 10.1.8.Final
> Reporter: mac peng
> Assignee: Dan Berindei
> Priority: Blocker
> Attachments: The fragile in TriangleOrderManager.png
>
>
> SingleKeyBackupWriteCommands run out of memory if sender does send out single SingleKeyBackupWriteCommand for some reason(Exception or deadlock during org.infinispan.interceptors.distribution.TriangleDistributionInterceptor.sendToBackups
> {color:#FF0000} long sequence = triangleOrderManager.next(segmentId, topologyId);{color}
> //{color:#FF0000}Sequence is increased but the cmd has no chance to send out for there is exception/deadlock later.{color}
> BackupWriteCommand backupCommand = backupBuilder.build(commandsFactory, command, entry.getValue());
> backupCommand.setSequence(sequence);
> backupCommand.setSegmentId(segmentId);
> if (trace) {
> log.tracef("Command %s got sequence %s for segment %s", command.getCommandInvocationId(), segmentId,
> sequence);
> }
> {color:#FF0000}rpcManager.sendToMany(backups, backupCommand, DeliverOrder.NONE);{color}
> 16-Oct-2020 01:09:38.403 ERROR [org.infinispan.remoting.rpc.RpcManagerImpl.rethrowAsCacheException] (dp-27-thread-1) ISPN000073: Unexpected error while re
> plicating
> {color:#FF0000}java.lang.RuntimeException: What's happen if Externalizer has exception?{color}
> {color:#FF0000} at com.ericsson.container.sip.fsm.InviteSessionFsm$InviteSessionExternalizer.writeObject(InviteSessionFsm.java:397){color}
> at com.example.container.sip.fsm.InviteSessionFsm$InviteSessionExternalizer.writeObject(InviteSessionFsm.java:385)
> at com.example.sessiondataservices.serialization.AdvancedExternalizerBridge.writeObject(AdvancedExternalizerBridge.java:42)
> at org.infinispan.marshall.core.impl.ExternalExternalizers$ForeignAdvancedExternalizer.writeObject(ExternalExternalizers.java:86)
> at org.infinispan.marshall.core.GlobalMarshaller.writeExternal(GlobalMarshaller.java:670)
> at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:375)
> at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:352)
> at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
> at com.example.ssa.sip.UA$UAExternalizer.writeObject(UA.java:930)
> at com.example.ssa.sip.UA$UAExternalizer.writeObject(UA.java:919)
> at com.example.sessiondataservices.serialization.AdvancedExternalizerBridge.writeObject(AdvancedExternalizerBridge.java:42)
> at org.infinispan.marshall.core.impl.ExternalExternalizers$ForeignAdvancedExternalizer.writeObject(ExternalExternalizers.java:86)
> at org.infinispan.marshall.core.GlobalMarshaller.writeExternal(GlobalMarshaller.java:670)
> at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:375)
> at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:352)
> at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
> at com.example.container.sip.session.dialog.DialogFragmentData$DialogFragmentDataExternalizer.writeObject(DialogFragmentData.java:780)
> at com.example.container.sip.session.dialog.DialogFragmentData$DialogFragmentDataExternalizer.writeObject(DialogFragmentData.java:761)
> at com.example.sessiondataservices.serialization.AdvancedExternalizerBridge.writeObject(AdvancedExternalizerBridge.java:42)
> at org.infinispan.marshall.core.impl.ExternalExternalizers$ForeignAdvancedExternalizer.writeObject(ExternalExternalizers.java:86)
> at org.infinispan.marshall.core.GlobalMarshaller.writeExternal(GlobalMarshaller.java:670)
> at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:375)
> at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:352)
> at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
> at com.example.container.sip.session.dialog.DialogFragmentData$DialogFragmentDataExternalizer.writeObject(DialogFragmentData.java:780)
> at com.example.container.sip.session.dialog.DialogFragmentData$DialogFragmentDataExternalizer.writeObject(DialogFragmentData.java:761)
> at com.example.sessiondataservices.serialization.AdvancedExternalizerBridge.writeObject(AdvancedExternalizerBridge.java:42)
> at org.infinispan.marshall.core.impl.ExternalExternalizers$ForeignAdvancedExternalizer.writeObject(ExternalExternalizers.java:86)
> at org.infinispan.marshall.core.GlobalMarshaller.writeExternal(GlobalMarshaller.java:670)
> at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:375)
> at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:352)
> at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
> at org.infinispan.commands.triangle.SingleKeyBackupWriteCommand.writeTo(SingleKeyBackupWriteCommand.java:108)
> at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:70)
> at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:122)
> at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:118)
> at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:69)
> at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:656)
> at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:371)
> at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:352)
> at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:181)
> at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:174)
> at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:302)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1075)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1272)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendToMany(JGroupsTransport.java:277)
> at org.infinispan.remoting.rpc.RpcManagerImpl.sendToMany(RpcManagerImpl.java:396)
> {color:#FF0000}at org.infinispan.interceptors.distribution.TriangleDistributionInterceptor.sendToBackups(TriangleDistributionInterceptor.java:490){color}
> at org.infinispan.interceptors.distribution.TriangleDistributionInterceptor.lambda$localPrimaryOwnerWrite$4(TriangleDistributionInterceptor.java:446)
> The receiver won't handle the subsquent SingleKeyBackupWriteCommands for current SingleKeyBackupWriteCommand is missing. Then the subsquent SingleKeyBackupWriteCommands will be putted into blockedTasks.offer(runnable); by BlockingTaskAwareExecutorServiceImpl instead of execute it.
>
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
3 years, 2 months
[Red Hat JIRA] (ISPN-12435) SingleKeyBackupWriteCommands run out of memory
by Dan Berindei (Jira)
[ https://issues.redhat.com/browse/ISPN-12435?page=com.atlassian.jira.plugi... ]
Dan Berindei updated ISPN-12435:
--------------------------------
Description:
SingleKeyBackupWriteCommands run out of memory if sender does send out single SingleKeyBackupWriteCommand for some reason(Exception or deadlock during org.infinispan.interceptors.distribution.TriangleDistributionInterceptor.sendToBackups
{color:#FF0000} long sequence = triangleOrderManager.next(segmentId, topologyId);{color}
//{color:#FF0000}Sequence is increased but the cmd has no chance to send out for there is exception/deadlock later.{color}
BackupWriteCommand backupCommand = backupBuilder.build(commandsFactory, command, entry.getValue());
backupCommand.setSequence(sequence);
backupCommand.setSegmentId(segmentId);
if (trace) {
log.tracef("Command %s got sequence %s for segment %s", command.getCommandInvocationId(), segmentId,
sequence);
}
{color:#FF0000}rpcManager.sendToMany(backups, backupCommand, DeliverOrder.NONE);{color}
16-Oct-2020 01:09:38.403 ERROR [org.infinispan.remoting.rpc.RpcManagerImpl.rethrowAsCacheException] (dp-27-thread-1) ISPN000073: Unexpected error while re
plicating
{color:#FF0000}java.lang.RuntimeException: What's happen if Externalizer has exception?{color}
{color:#FF0000} at com.ericsson.container.sip.fsm.InviteSessionFsm$InviteSessionExternalizer.writeObject(InviteSessionFsm.java:397){color}
at com.example.container.sip.fsm.InviteSessionFsm$InviteSessionExternalizer.writeObject(InviteSessionFsm.java:385)
at com.example.sessiondataservices.serialization.AdvancedExternalizerBridge.writeObject(AdvancedExternalizerBridge.java:42)
at org.infinispan.marshall.core.impl.ExternalExternalizers$ForeignAdvancedExternalizer.writeObject(ExternalExternalizers.java:86)
at org.infinispan.marshall.core.GlobalMarshaller.writeExternal(GlobalMarshaller.java:670)
at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:375)
at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:352)
at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
at com.example.ssa.sip.UA$UAExternalizer.writeObject(UA.java:930)
at com.example.ssa.sip.UA$UAExternalizer.writeObject(UA.java:919)
at com.example.sessiondataservices.serialization.AdvancedExternalizerBridge.writeObject(AdvancedExternalizerBridge.java:42)
at org.infinispan.marshall.core.impl.ExternalExternalizers$ForeignAdvancedExternalizer.writeObject(ExternalExternalizers.java:86)
at org.infinispan.marshall.core.GlobalMarshaller.writeExternal(GlobalMarshaller.java:670)
at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:375)
at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:352)
at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
at com.example.container.sip.session.dialog.DialogFragmentData$DialogFragmentDataExternalizer.writeObject(DialogFragmentData.java:780)
at com.example.container.sip.session.dialog.DialogFragmentData$DialogFragmentDataExternalizer.writeObject(DialogFragmentData.java:761)
at com.example.sessiondataservices.serialization.AdvancedExternalizerBridge.writeObject(AdvancedExternalizerBridge.java:42)
at org.infinispan.marshall.core.impl.ExternalExternalizers$ForeignAdvancedExternalizer.writeObject(ExternalExternalizers.java:86)
at org.infinispan.marshall.core.GlobalMarshaller.writeExternal(GlobalMarshaller.java:670)
at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:375)
at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:352)
at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
at com.example.container.sip.session.dialog.DialogFragmentData$DialogFragmentDataExternalizer.writeObject(DialogFragmentData.java:780)
at com.example.container.sip.session.dialog.DialogFragmentData$DialogFragmentDataExternalizer.writeObject(DialogFragmentData.java:761)
at com.example.sessiondataservices.serialization.AdvancedExternalizerBridge.writeObject(AdvancedExternalizerBridge.java:42)
at org.infinispan.marshall.core.impl.ExternalExternalizers$ForeignAdvancedExternalizer.writeObject(ExternalExternalizers.java:86)
at org.infinispan.marshall.core.GlobalMarshaller.writeExternal(GlobalMarshaller.java:670)
at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:375)
at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:352)
at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
at org.infinispan.commands.triangle.SingleKeyBackupWriteCommand.writeTo(SingleKeyBackupWriteCommand.java:108)
at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:70)
at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:122)
at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:118)
at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:69)
at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:656)
at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:371)
at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:352)
at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:181)
at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:174)
at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:302)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1075)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1272)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendToMany(JGroupsTransport.java:277)
at org.infinispan.remoting.rpc.RpcManagerImpl.sendToMany(RpcManagerImpl.java:396)
{color:#FF0000}at org.infinispan.interceptors.distribution.TriangleDistributionInterceptor.sendToBackups(TriangleDistributionInterceptor.java:490){color}
at org.infinispan.interceptors.distribution.TriangleDistributionInterceptor.lambda$localPrimaryOwnerWrite$4(TriangleDistributionInterceptor.java:446)
The receiver won't handle the subsquent SingleKeyBackupWriteCommands for current SingleKeyBackupWriteCommand is missing. Then the subsquent SingleKeyBackupWriteCommands will be putted into blockedTasks.offer(runnable); by BlockingTaskAwareExecutorServiceImpl instead of execute it.
was:
SingleKeyBackupWriteCommands run out of memory if sender does send out single SingleKeyBackupWriteCommand for some reason(Exception or deadlock during org.infinispan.interceptors.distribution.TriangleDistributionInterceptor.sendToBackups
{color:#FF0000} long sequence = triangleOrderManager.next(segmentId, topologyId);{color}
//{color:#FF0000}Sequence is increased but the cmd has no chance to send out for there is exception/deadlock later.{color}
BackupWriteCommand backupCommand = backupBuilder.build(commandsFactory, command, entry.getValue());
backupCommand.setSequence(sequence);
backupCommand.setSegmentId(segmentId);
if (trace) {
log.tracef("Command %s got sequence %s for segment %s", command.getCommandInvocationId(), segmentId,
sequence);
}
{color:#FF0000}rpcManager.sendToMany(backups, backupCommand, DeliverOrder.NONE);{color}
16-Oct-2020 01:09:38.403 ERROR [org.infinispan.remoting.rpc.RpcManagerImpl.rethrowAsCacheException] (dp-27-thread-1) ISPN000073: Unexpected error while re
plicating
{color:#FF0000}java.lang.RuntimeException: What's happen if Externalizer has exception?{color}
{color:#FF0000} at com.ericsson.container.sip.fsm.InviteSessionFsm$InviteSessionExternalizer.writeObject(InviteSessionFsm.java:397){color}
at com.ericsson.container.sip.fsm.InviteSessionFsm$InviteSessionExternalizer.writeObject(InviteSessionFsm.java:385)
at com.ericsson.sessiondataservices.serialization.AdvancedExternalizerBridge.writeObject(AdvancedExternalizerBridge.java:42)
at org.infinispan.marshall.core.impl.ExternalExternalizers$ForeignAdvancedExternalizer.writeObject(ExternalExternalizers.java:86)
at org.infinispan.marshall.core.GlobalMarshaller.writeExternal(GlobalMarshaller.java:670)
at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:375)
at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:352)
at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
at com.ericsson.ssa.sip.UA$UAExternalizer.writeObject(UA.java:930)
at com.ericsson.ssa.sip.UA$UAExternalizer.writeObject(UA.java:919)
at com.ericsson.sessiondataservices.serialization.AdvancedExternalizerBridge.writeObject(AdvancedExternalizerBridge.java:42)
at org.infinispan.marshall.core.impl.ExternalExternalizers$ForeignAdvancedExternalizer.writeObject(ExternalExternalizers.java:86)
at org.infinispan.marshall.core.GlobalMarshaller.writeExternal(GlobalMarshaller.java:670)
at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:375)
at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:352)
at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
at com.ericsson.container.sip.session.dialog.DialogFragmentData$DialogFragmentDataExternalizer.writeObject(DialogFragmentData.java:780)
at com.ericsson.container.sip.session.dialog.DialogFragmentData$DialogFragmentDataExternalizer.writeObject(DialogFragmentData.java:761)
at com.ericsson.sessiondataservices.serialization.AdvancedExternalizerBridge.writeObject(AdvancedExternalizerBridge.java:42)
at org.infinispan.marshall.core.impl.ExternalExternalizers$ForeignAdvancedExternalizer.writeObject(ExternalExternalizers.java:86)
at org.infinispan.marshall.core.GlobalMarshaller.writeExternal(GlobalMarshaller.java:670)
at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:375)
at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:352)
at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
at com.ericsson.container.sip.session.dialog.DialogFragmentData$DialogFragmentDataExternalizer.writeObject(DialogFragmentData.java:780)
at com.ericsson.container.sip.session.dialog.DialogFragmentData$DialogFragmentDataExternalizer.writeObject(DialogFragmentData.java:761)
at com.ericsson.sessiondataservices.serialization.AdvancedExternalizerBridge.writeObject(AdvancedExternalizerBridge.java:42)
at org.infinispan.marshall.core.impl.ExternalExternalizers$ForeignAdvancedExternalizer.writeObject(ExternalExternalizers.java:86)
at org.infinispan.marshall.core.GlobalMarshaller.writeExternal(GlobalMarshaller.java:670)
at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:375)
at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:352)
at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
at org.infinispan.commands.triangle.SingleKeyBackupWriteCommand.writeTo(SingleKeyBackupWriteCommand.java:108)
at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:70)
at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:122)
at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:118)
at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:69)
at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:656)
at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:371)
at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:352)
at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:181)
at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:174)
at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:302)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1075)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1272)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendToMany(JGroupsTransport.java:277)
at org.infinispan.remoting.rpc.RpcManagerImpl.sendToMany(RpcManagerImpl.java:396)
{color:#FF0000}at org.infinispan.interceptors.distribution.TriangleDistributionInterceptor.sendToBackups(TriangleDistributionInterceptor.java:490){color}
at org.infinispan.interceptors.distribution.TriangleDistributionInterceptor.lambda$localPrimaryOwnerWrite$4(TriangleDistributionInterceptor.java:446)
The receiver won't handle the subsquent SingleKeyBackupWriteCommands for current SingleKeyBackupWriteCommand is missing. Then the subsquent SingleKeyBackupWriteCommands will be putted into blockedTasks.offer(runnable); by BlockingTaskAwareExecutorServiceImpl instead of execute it.
> SingleKeyBackupWriteCommands run out of memory
> ----------------------------------------------
>
> Key: ISPN-12435
> URL: https://issues.redhat.com/browse/ISPN-12435
> Project: Infinispan
> Issue Type: Bug
> Components: Cloud
> Affects Versions: 10.1.8.Final
> Reporter: mac peng
> Priority: Blocker
> Attachments: The fragile in TriangleOrderManager.png
>
>
> SingleKeyBackupWriteCommands run out of memory if sender does send out single SingleKeyBackupWriteCommand for some reason(Exception or deadlock during org.infinispan.interceptors.distribution.TriangleDistributionInterceptor.sendToBackups
> {color:#FF0000} long sequence = triangleOrderManager.next(segmentId, topologyId);{color}
> //{color:#FF0000}Sequence is increased but the cmd has no chance to send out for there is exception/deadlock later.{color}
> BackupWriteCommand backupCommand = backupBuilder.build(commandsFactory, command, entry.getValue());
> backupCommand.setSequence(sequence);
> backupCommand.setSegmentId(segmentId);
> if (trace) {
> log.tracef("Command %s got sequence %s for segment %s", command.getCommandInvocationId(), segmentId,
> sequence);
> }
> {color:#FF0000}rpcManager.sendToMany(backups, backupCommand, DeliverOrder.NONE);{color}
> 16-Oct-2020 01:09:38.403 ERROR [org.infinispan.remoting.rpc.RpcManagerImpl.rethrowAsCacheException] (dp-27-thread-1) ISPN000073: Unexpected error while re
> plicating
> {color:#FF0000}java.lang.RuntimeException: What's happen if Externalizer has exception?{color}
> {color:#FF0000} at com.ericsson.container.sip.fsm.InviteSessionFsm$InviteSessionExternalizer.writeObject(InviteSessionFsm.java:397){color}
> at com.example.container.sip.fsm.InviteSessionFsm$InviteSessionExternalizer.writeObject(InviteSessionFsm.java:385)
> at com.example.sessiondataservices.serialization.AdvancedExternalizerBridge.writeObject(AdvancedExternalizerBridge.java:42)
> at org.infinispan.marshall.core.impl.ExternalExternalizers$ForeignAdvancedExternalizer.writeObject(ExternalExternalizers.java:86)
> at org.infinispan.marshall.core.GlobalMarshaller.writeExternal(GlobalMarshaller.java:670)
> at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:375)
> at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:352)
> at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
> at com.example.ssa.sip.UA$UAExternalizer.writeObject(UA.java:930)
> at com.example.ssa.sip.UA$UAExternalizer.writeObject(UA.java:919)
> at com.example.sessiondataservices.serialization.AdvancedExternalizerBridge.writeObject(AdvancedExternalizerBridge.java:42)
> at org.infinispan.marshall.core.impl.ExternalExternalizers$ForeignAdvancedExternalizer.writeObject(ExternalExternalizers.java:86)
> at org.infinispan.marshall.core.GlobalMarshaller.writeExternal(GlobalMarshaller.java:670)
> at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:375)
> at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:352)
> at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
> at com.example.container.sip.session.dialog.DialogFragmentData$DialogFragmentDataExternalizer.writeObject(DialogFragmentData.java:780)
> at com.example.container.sip.session.dialog.DialogFragmentData$DialogFragmentDataExternalizer.writeObject(DialogFragmentData.java:761)
> at com.example.sessiondataservices.serialization.AdvancedExternalizerBridge.writeObject(AdvancedExternalizerBridge.java:42)
> at org.infinispan.marshall.core.impl.ExternalExternalizers$ForeignAdvancedExternalizer.writeObject(ExternalExternalizers.java:86)
> at org.infinispan.marshall.core.GlobalMarshaller.writeExternal(GlobalMarshaller.java:670)
> at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:375)
> at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:352)
> at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
> at com.example.container.sip.session.dialog.DialogFragmentData$DialogFragmentDataExternalizer.writeObject(DialogFragmentData.java:780)
> at com.example.container.sip.session.dialog.DialogFragmentData$DialogFragmentDataExternalizer.writeObject(DialogFragmentData.java:761)
> at com.example.sessiondataservices.serialization.AdvancedExternalizerBridge.writeObject(AdvancedExternalizerBridge.java:42)
> at org.infinispan.marshall.core.impl.ExternalExternalizers$ForeignAdvancedExternalizer.writeObject(ExternalExternalizers.java:86)
> at org.infinispan.marshall.core.GlobalMarshaller.writeExternal(GlobalMarshaller.java:670)
> at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:375)
> at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:352)
> at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
> at org.infinispan.commands.triangle.SingleKeyBackupWriteCommand.writeTo(SingleKeyBackupWriteCommand.java:108)
> at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:70)
> at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:122)
> at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:118)
> at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:69)
> at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:656)
> at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:371)
> at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:352)
> at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:181)
> at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:174)
> at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:302)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1075)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1272)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendToMany(JGroupsTransport.java:277)
> at org.infinispan.remoting.rpc.RpcManagerImpl.sendToMany(RpcManagerImpl.java:396)
> {color:#FF0000}at org.infinispan.interceptors.distribution.TriangleDistributionInterceptor.sendToBackups(TriangleDistributionInterceptor.java:490){color}
> at org.infinispan.interceptors.distribution.TriangleDistributionInterceptor.lambda$localPrimaryOwnerWrite$4(TriangleDistributionInterceptor.java:446)
> The receiver won't handle the subsquent SingleKeyBackupWriteCommands for current SingleKeyBackupWriteCommand is missing. Then the subsquent SingleKeyBackupWriteCommands will be putted into blockedTasks.offer(runnable); by BlockingTaskAwareExecutorServiceImpl instead of execute it.
>
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
3 years, 2 months