[JBoss JIRA] (ISPN-5046) PartitionHandling: split during commit can leave the cache inconsistent after merge
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-5046?page=com.atlassian.jira.plugin.... ]
Pedro Ruivo reassigned ISPN-5046:
---------------------------------
Assignee: Pedro Ruivo (was: Dan Berindei)
> PartitionHandling: split during commit can leave the cache inconsistent after merge
> -----------------------------------------------------------------------------------
>
> Key: ISPN-5046
> URL: https://issues.jboss.org/browse/ISPN-5046
> Project: Infinispan
> Issue Type: Bug
> Components: Core, State Transfer
> Affects Versions: 7.0.2.Final, 7.1.0.Alpha1
> Reporter: Dan Berindei
> Assignee: Pedro Ruivo
> Priority: Critical
> Fix For: 7.2.0.CR1
>
>
> Say we have a cluster ABCD; a transaction T was started on A, with B as the primary owner and C the backup owner. B and C both acknowledge the prepare, and the network splits into AB and CD right before A sends the commit command. Eventually A suspects C and D, but the commit still succeeds on B before C and D are suspected. And SuspectExceptions are ignored for commit commands, so the user won't see any error.
> However, C will eventually suspect A and B. When the CD cache topology is installed, it will roll back transaction T. After the merge, both partitions are in degraded mode, so we assume that they both have the latest data and the key is never updated on C.
> From C's point of view, this is very similar to ISPN-3421. The fix should also be similar, we could delay the transaction rollback on C until we get a confirmation from B that T was not committed there. Since B is inaccessible, it will eventually get a SuspectException and the CD cache topology, at which point the cache is in degraded mode and it can wait for a merge. On merge, it should check the status of the transaction on B again, and either commit or rollback based on what B did.
> We also need to suspend the cleanup of completed transactions while the cache is in degraded mode, otherwise C might not find T on B after the merge.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
11 years
[JBoss JIRA] (ISPN-4778) PessimisticLockingInterceptor throws when handling remote clear command
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-4778?page=com.atlassian.jira.plugin.... ]
Pedro Ruivo closed ISPN-4778.
-----------------------------
Resolution: Won't Fix
Clear consistency was relaxed. No locks are involved now and it does not interact with transactions
> PessimisticLockingInterceptor throws when handling remote clear command
> -----------------------------------------------------------------------
>
> Key: ISPN-4778
> URL: https://issues.jboss.org/browse/ISPN-4778
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 6.0.2.Final
> Environment: JBoss WildFly 8.1.0.FINAL
> Reporter: Arjan t
> Assignee: Pedro Ruivo
> Labels: remote
>
> Using InfiniSpan as it's shipped with Jboss WildFly 8.1.0.Final as distributed cache for Hibernate, it appears that the ClearCommand does not work in a cluster when *pessimistic locking* is used. Pessimistic locking seems to be the default in WildFly, even when theoretically it shouldn't be.
> This will result in the following exception:
> {noformat}
> java.lang.ClassCastException: org.infinispan.context.impl.NonTxInvocationContext cannot be cast to org.infinispan.context.impl.TxInvocationContext
> at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitClearCommand(PessimisticLockingInterceptor.java:194)
> at org.infinispan.commands.write.ClearCommand.acceptVisitor(ClearCommand.java:38)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitClearCommand(AbstractVisitor.java:47)
> at org.infinispan.commands.write.ClearCommand.acceptVisitor(ClearCommand.java:38)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:255)
> at org.infinispan.interceptors.TxInterceptor.visitClearCommand(TxInterceptor.java:206)
> at org.infinispan.commands.write.ClearCommand.acceptVisitor(ClearCommand.java:38)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitClearCommand(AbstractVisitor.java:47)
> at org.infinispan.commands.write.ClearCommand.acceptVisitor(ClearCommand.java:38)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73)
> at org.infinispan.commands.AbstractVisitor.visitClearCommand(AbstractVisitor.java:47)
> at org.infinispan.commands.write.ClearCommand.acceptVisitor(ClearCommand.java:38)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)
> at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommand(BaseRpcInvokingCommand.java:39)
> at org.infinispan.commands.remote.SingleRpcCommand.perform(SingleRpcCommand.java:48)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:95)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:50)
> at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:172)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> {noformat}
> The incoming command looks as follows:
> {noformat}
> CacheRpcCommand cmd:
> command:
> ClearCommand{flags=null}
> icf:
> org.infinispan.context.TransactionalInvocationContextFactory@3ef1861e
> Interceptor chain:
>
> >> org.infinispan.interceptors.InvocationContextInterceptor -- checks if stopping, otherwise continues
> >> org.infinispan.interceptors.CacheMgmtInterceptor -- does nothing
> >> org.infinispan.interceptors.TxInterceptor -- checks "shouldEnlist", if false does nothing
> >> org.infinispan.interceptors.NotificationInterceptor -- does nothing
> >> org.infinispan.interceptors.locking.PessimisticLockingInterceptor -- Throws exception if something in cache
> >> org.infinispan.interceptors.EntryWrappingInterceptor
> >> org.infinispan.interceptors.InvalidationInterceptor
> >> org.infinispan.interceptors.CallInterceptor
> {noformat}
> The problem seems to be that {{org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommand}} always creates a {{NonTxInvocationContext}}. As per the following line of code:
> {code}
> final InvocationContext ctx = icf.createRemoteInvocationContextForCommand(vc, getOrigin());
> {code}
> When handling the ClearCommand, the PessimisticLockInterceptor always casts this to a {{TxInvocationContext}} whenever {{dataContainer}} is not empty, e.g. when there is cached data on the node where the clear command arrives. This happens in the following code:
> {code}
> public Object visitClearCommand(InvocationContext ctx, ClearCommand command) throws Throwable {
> try {
> boolean skipLocking = hasSkipLocking(command);
> long lockTimeout = getLockAcquisitionTimeout(command, skipLocking);
> for (InternalCacheEntry entry : dataContainer.entrySet())
> lockAndRegisterBackupLock((TxInvocationContext) ctx, entry.getKey(), lockTimeout, skipLocking);
> return invokeNextInterceptor(ctx, command);
> } catch (Throwable te) {
> releaseLocksOnFailureBeforePrepare(ctx);
> throw te;
> }
> }
> {code}
> So seemingly this can't ever work.
> Either the {{PessimisticLockingInterceptor}} can't be in a the interceptor chain when handling commands from a remote destination, or something has to be done about about the {{InvocationContext}} when handling remote commands?
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
11 years
[JBoss JIRA] (ISPN-4778) PessimisticLockingInterceptor throws when handling remote clear command
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-4778?page=com.atlassian.jira.plugin.... ]
Pedro Ruivo reassigned ISPN-4778:
---------------------------------
Assignee: Pedro Ruivo
> PessimisticLockingInterceptor throws when handling remote clear command
> -----------------------------------------------------------------------
>
> Key: ISPN-4778
> URL: https://issues.jboss.org/browse/ISPN-4778
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 6.0.2.Final
> Environment: JBoss WildFly 8.1.0.FINAL
> Reporter: Arjan t
> Assignee: Pedro Ruivo
> Labels: remote
>
> Using InfiniSpan as it's shipped with Jboss WildFly 8.1.0.Final as distributed cache for Hibernate, it appears that the ClearCommand does not work in a cluster when *pessimistic locking* is used. Pessimistic locking seems to be the default in WildFly, even when theoretically it shouldn't be.
> This will result in the following exception:
> {noformat}
> java.lang.ClassCastException: org.infinispan.context.impl.NonTxInvocationContext cannot be cast to org.infinispan.context.impl.TxInvocationContext
> at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitClearCommand(PessimisticLockingInterceptor.java:194)
> at org.infinispan.commands.write.ClearCommand.acceptVisitor(ClearCommand.java:38)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitClearCommand(AbstractVisitor.java:47)
> at org.infinispan.commands.write.ClearCommand.acceptVisitor(ClearCommand.java:38)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:255)
> at org.infinispan.interceptors.TxInterceptor.visitClearCommand(TxInterceptor.java:206)
> at org.infinispan.commands.write.ClearCommand.acceptVisitor(ClearCommand.java:38)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitClearCommand(AbstractVisitor.java:47)
> at org.infinispan.commands.write.ClearCommand.acceptVisitor(ClearCommand.java:38)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73)
> at org.infinispan.commands.AbstractVisitor.visitClearCommand(AbstractVisitor.java:47)
> at org.infinispan.commands.write.ClearCommand.acceptVisitor(ClearCommand.java:38)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)
> at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommand(BaseRpcInvokingCommand.java:39)
> at org.infinispan.commands.remote.SingleRpcCommand.perform(SingleRpcCommand.java:48)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:95)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:50)
> at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:172)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> {noformat}
> The incoming command looks as follows:
> {noformat}
> CacheRpcCommand cmd:
> command:
> ClearCommand{flags=null}
> icf:
> org.infinispan.context.TransactionalInvocationContextFactory@3ef1861e
> Interceptor chain:
>
> >> org.infinispan.interceptors.InvocationContextInterceptor -- checks if stopping, otherwise continues
> >> org.infinispan.interceptors.CacheMgmtInterceptor -- does nothing
> >> org.infinispan.interceptors.TxInterceptor -- checks "shouldEnlist", if false does nothing
> >> org.infinispan.interceptors.NotificationInterceptor -- does nothing
> >> org.infinispan.interceptors.locking.PessimisticLockingInterceptor -- Throws exception if something in cache
> >> org.infinispan.interceptors.EntryWrappingInterceptor
> >> org.infinispan.interceptors.InvalidationInterceptor
> >> org.infinispan.interceptors.CallInterceptor
> {noformat}
> The problem seems to be that {{org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommand}} always creates a {{NonTxInvocationContext}}. As per the following line of code:
> {code}
> final InvocationContext ctx = icf.createRemoteInvocationContextForCommand(vc, getOrigin());
> {code}
> When handling the ClearCommand, the PessimisticLockInterceptor always casts this to a {{TxInvocationContext}} whenever {{dataContainer}} is not empty, e.g. when there is cached data on the node where the clear command arrives. This happens in the following code:
> {code}
> public Object visitClearCommand(InvocationContext ctx, ClearCommand command) throws Throwable {
> try {
> boolean skipLocking = hasSkipLocking(command);
> long lockTimeout = getLockAcquisitionTimeout(command, skipLocking);
> for (InternalCacheEntry entry : dataContainer.entrySet())
> lockAndRegisterBackupLock((TxInvocationContext) ctx, entry.getKey(), lockTimeout, skipLocking);
> return invokeNextInterceptor(ctx, command);
> } catch (Throwable te) {
> releaseLocksOnFailureBeforePrepare(ctx);
> throw te;
> }
> }
> {code}
> So seemingly this can't ever work.
> Either the {{PessimisticLockingInterceptor}} can't be in a the interceptor chain when handling commands from a remote destination, or something has to be done about about the {{InvocationContext}} when handling remote commands?
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
11 years
[JBoss JIRA] (ISPN-4073) clear() can create inconsistencies
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-4073?page=com.atlassian.jira.plugin.... ]
Pedro Ruivo commented on ISPN-4073:
-----------------------------------
Clear consistency was relaxed. Closing this JIRA
> clear() can create inconsistencies
> ----------------------------------
>
> Key: ISPN-4073
> URL: https://issues.jboss.org/browse/ISPN-4073
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 7.0.0.Alpha1
> Reporter: Pedro Ruivo
>
> Since the clear operation only acquires locks on the existing keys, it is possible for a clear operation to execute concurrently with another operation that is adding a new key. Since the lock on the new key is not acquired by the clear, some nodes can perform first the clear then the operation and others vice-versa.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
11 years
[JBoss JIRA] (ISPN-4073) clear() can create inconsistencies
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-4073?page=com.atlassian.jira.plugin.... ]
Pedro Ruivo closed ISPN-4073.
-----------------------------
Assignee: Pedro Ruivo
Resolution: Won't Fix
> clear() can create inconsistencies
> ----------------------------------
>
> Key: ISPN-4073
> URL: https://issues.jboss.org/browse/ISPN-4073
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 7.0.0.Alpha1
> Reporter: Pedro Ruivo
> Assignee: Pedro Ruivo
>
> Since the clear operation only acquires locks on the existing keys, it is possible for a clear operation to execute concurrently with another operation that is adding a new key. Since the lock on the new key is not acquired by the clear, some nodes can perform first the clear then the operation and others vice-versa.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
11 years