[infinispan-issues] [JBoss JIRA] (ISPN-6857) OutdatedTopologyException in clustered invalidation cache because StateTransferInterceptor not in the chain

[Marek Posolda (JIRA)]

    [ https://issues.jboss.org/browse/ISPN-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266072#comment-13266072 ] 

Marek Posolda edited comment on ISPN-6857 at 7/15/16 4:09 AM:
--------------------------------------------------------------

The stacktrace thrown on node1 ( looks like it's propagated from node2 where remote invalidation command failed):
{code}
org.infinispan.statetransfer.OutdatedTopologyException: Cache topology changed while the command was executing: expected 1, got 2
	at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:522)
	at org.infinispan.interceptors.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:560)
	at org.infinispan.interceptors.EntryWrappingInterceptor.visitInvalidateCommand(EntryWrappingInterceptor.java:175)
	at org.infinispan.commands.write.InvalidateCommand.acceptVisitor(InvalidateCommand.java:125)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
	at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitInvalidateCommand(AbstractLockingInterceptor.java:109)
	at org.infinispan.commands.write.InvalidateCommand.acceptVisitor(InvalidateCommand.java:125)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
	at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:127)
	at org.infinispan.commands.write.InvalidateCommand.acceptVisitor(InvalidateCommand.java:125)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:107)
	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:76)
	at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:127)
	at org.infinispan.commands.write.InvalidateCommand.acceptVisitor(InvalidateCommand.java:125)
	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
	at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommand(BaseRpcInvokingCommand.java:43)
	at org.infinispan.commands.remote.SingleRpcCommand.perform(SingleRpcCommand.java:48)
	at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokePerform(BasePerCacheInboundInvocationHandler.java:92)
	at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:34)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
{code}


was (Author: mposolda):
I am attaching the Java unit test for reproducing issue. On my laptop when I run it, I can almost always simulate the issue.

The stacktrace thrown on node1 ( looks like it's propagated from node2 where remote invalidation command failed):
{code}
org.infinispan.statetransfer.OutdatedTopologyException: Cache topology changed while the command was executing: expected 1, got 2
	at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:522)
	at org.infinispan.interceptors.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:560)
	at org.infinispan.interceptors.EntryWrappingInterceptor.visitInvalidateCommand(EntryWrappingInterceptor.java:175)
	at org.infinispan.commands.write.InvalidateCommand.acceptVisitor(InvalidateCommand.java:125)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
	at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitInvalidateCommand(AbstractLockingInterceptor.java:109)
	at org.infinispan.commands.write.InvalidateCommand.acceptVisitor(InvalidateCommand.java:125)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
	at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:127)
	at org.infinispan.commands.write.InvalidateCommand.acceptVisitor(InvalidateCommand.java:125)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:107)
	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:76)
	at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:127)
	at org.infinispan.commands.write.InvalidateCommand.acceptVisitor(InvalidateCommand.java:125)
	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
	at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommand(BaseRpcInvokingCommand.java:43)
	at org.infinispan.commands.remote.SingleRpcCommand.perform(SingleRpcCommand.java:48)
	at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokePerform(BasePerCacheInboundInvocationHandler.java:92)
	at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:34)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
{code}

> OutdatedTopologyException in clustered invalidation cache because StateTransferInterceptor not in the chain
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: ISPN-6857
>                 URL: https://issues.jboss.org/browse/ISPN-6857
>             Project: Infinispan
>          Issue Type: Bug
>    Affects Versions: 8.1.0.Final
>            Reporter: Marek Posolda
>         Attachments: OutdatedTopologyExceptionReproducerTest.java
>
>
> I have the following setup:
> - 2 nodes in cluster with mode INVALIDATION_SYNC. No-transaction cache.
> - Node1 is started
> - Called "cache.remove" on some key on node1. At the same time, node2 is starting, which is causing  topology change.
> - The "cache.remove" call on node1 is throwing OutdatedTopologyException.
> I found the cause is that StateTransferInterceptor is not added in InterceptorChain during INVALIDATION mode. It's just available during REPLICATION or DISTRIBUTED modes - https://github.com/infinispan/infinispan/blob/master/core/src/main/java/org/infinispan/factories/InterceptorChainFactory.java#L158
> Indeed when I manually added StateTransferInterceptor to my invalidation cache:
> {code}
> invalidationConfigBuilder.customInterceptors()
>     .addInterceptor()
>     .before(NonTransactionalLockingInterceptor.class)
>     .interceptorClass(StateTransferInterceptor.class);
> {code}
>  
> I can see that issue is gone as OutdatedTopologyException is catched and command is retried with new topology.



--
This message was sent by Atlassian JIRA
(v6.4.11#64026)