[JBoss JIRA] (ISPN-7562) Error executing command LockControlCommand in L1 scenario
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/ISPN-7562?page=com.atlassian.jira.plugin.... ]
Radoslav Husar updated ISPN-7562:
---------------------------------
Priority: Major (was: Minor)
> Error executing command LockControlCommand in L1 scenario
> ---------------------------------------------------------
>
> Key: ISPN-7562
> URL: https://issues.jboss.org/browse/ISPN-7562
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 8.2.6.Final
> Reporter: Radoslav Husar
>
> We saw this error in failover scenario *http-session-undeploy-dist-async-l1*.
> Server log stacktrace:
> {code}
> 09:57:08,234 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (remote-thread--p6-t13) ISPN000136: Error executing command LockControlCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: Could not acquire lock on SessionCreationMetaDataKey(yvcoGxpq8mCqt8kDOkpTjZSNeYoLdKma5ehwB3TI) in behalf of transaction GlobalTransaction:<perf20>:22151:remote. Current owner GlobalTransaction:<perf20>:19986:remote.
> [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.timeout(DefaultPendingLockManager.java:261)
> [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.awaitOn(DefaultPendingLockManager.java:344)
> [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.awaitPendingTransactionsForAllKeys(DefaultPendingLockManager.java:147)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockAllKeys(AbstractTxLockingInterceptor.java:198)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockAllOrRegisterBackupLock(AbstractTxLockingInterceptor.java:166)
> [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitLockControlCommand(PessimisticLockingInterceptor.java:202)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:113)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:158)
> [JBossINF] at org.infinispan.interceptors.TxInterceptor.visitLockControlCommand(TxInterceptor.java:216)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:113)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163)
> [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitLockControlCommand(TransactionSynchronizerInterceptor.java:75)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:113)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:229)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.visitLockControlCommand(StateTransferInterceptor.java:102)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:113)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)
> [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.visitLockControlCommand(InvocationContextInterceptor.java:84)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:113)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:113)
> [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:335)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.perform(LockControlCommand.java:125)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokePerform(BasePerCacheInboundInvocationHandler.java:92)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:34)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [JBossINF] at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> [JBossINF] at java.lang.Thread.run(Thread.java:745)
> {code}
> WARN messages are logged after new cluster view was received and before the error message:
> {code}
> 09:56:49,265 WARN [org.infinispan.statetransfer.StateConsumerImpl] (stateTransferExecutor-thread--p22-t32) Discarding received cache entries for segment 63 of cache clusterbench-ee7.ear.clusterbench-ee7-web-default.war because they do not belong to this node.
> {code}
> Link to server log:
> http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_eap-7x-fa...
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
7 years, 10 months
[JBoss JIRA] (ISPN-5021) Nodes that finish the rebalance later can see outdated values
by Radim Vansa (JIRA)
[ https://issues.jboss.org/browse/ISPN-5021?page=com.atlassian.jira.plugin.... ]
Radim Vansa commented on ISPN-5021:
-----------------------------------
[~pferraro] Ah, seems you're hitting the 'no owners' part in JBEAP-8958 rather than this bug (reading outdated values).
> Nodes that finish the rebalance later can see outdated values
> -------------------------------------------------------------
>
> Key: ISPN-5021
> URL: https://issues.jboss.org/browse/ISPN-5021
> Project: Infinispan
> Issue Type: Bug
> Components: Core, State Transfer
> Affects Versions: 7.0.2.Final
> Reporter: Dan Berindei
> Assignee: Pedro Ruivo
> Priority: Critical
> Labels: consistency, testsuite_stability
> Fix For: 9.0.0.Final
>
>
> Copied from [ISPN-4444|https://issues.jboss.org/browse/ISPN-4444?focusedCommentId=1302...]
> If the CH_UPDATE command is delayed on the old owner, the new owners might update the key without the old owner knowing, and a locality check on the old owner won't help.
> I remember one thing that struck me when reading the Raft algorithm was that they install configuration changes symmetrically, in 3 phases. We might need to do the same for our rebalance:
> 1. T0: read_ch=old, write_ch=old
> 2. start a rebalance
> 3. T1: read_ch=old, write_ch=old+new
> 4. new owners have all the data
> 5. T2: read_ch=new, write_ch=old+new
> 6. remove old cache entries and ignore further writes
> 7. T3: read_ch=new, write_ch=new
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
7 years, 10 months
[JBoss JIRA] (ISPN-5021) Nodes that finish the rebalance later can see outdated values
by Radim Vansa (JIRA)
[ https://issues.jboss.org/browse/ISPN-5021?page=com.atlassian.jira.plugin.... ]
Radim Vansa commented on ISPN-5021:
-----------------------------------
[~pferraro] Could you elaborate? How have you confirmed that it's this issue? (I am wondering because the bug should be rare, and was reported more than 2 years ago).
> Nodes that finish the rebalance later can see outdated values
> -------------------------------------------------------------
>
> Key: ISPN-5021
> URL: https://issues.jboss.org/browse/ISPN-5021
> Project: Infinispan
> Issue Type: Bug
> Components: Core, State Transfer
> Affects Versions: 7.0.2.Final
> Reporter: Dan Berindei
> Assignee: Pedro Ruivo
> Priority: Critical
> Labels: consistency, testsuite_stability
> Fix For: 9.0.0.Final
>
>
> Copied from [ISPN-4444|https://issues.jboss.org/browse/ISPN-4444?focusedCommentId=1302...]
> If the CH_UPDATE command is delayed on the old owner, the new owners might update the key without the old owner knowing, and a locality check on the old owner won't help.
> I remember one thing that struck me when reading the Raft algorithm was that they install configuration changes symmetrically, in 3 phases. We might need to do the same for our rebalance:
> 1. T0: read_ch=old, write_ch=old
> 2. start a rebalance
> 3. T1: read_ch=old, write_ch=old+new
> 4. new owners have all the data
> 5. T2: read_ch=new, write_ch=old+new
> 6. remove old cache entries and ignore further writes
> 7. T3: read_ch=new, write_ch=new
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
7 years, 10 months
[JBoss JIRA] (ISPN-5728) Implement all the default methods in Java 8's Map interface
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-5728?page=com.atlassian.jira.plugin.... ]
William Burns reassigned ISPN-5728:
-----------------------------------
Assignee: Katia Aresti (was: William Burns)
> Implement all the default methods in Java 8's Map interface
> -----------------------------------------------------------
>
> Key: ISPN-5728
> URL: https://issues.jboss.org/browse/ISPN-5728
> Project: Infinispan
> Issue Type: Feature Request
> Components: Core
> Affects Versions: 8.0.0.Final
> Reporter: Dan Berindei
> Assignee: Katia Aresti
>
> Java 8 added many new methods to the {{Map}} interface. Some of them were already present in {{ConcurrentMap}}, but others are new: {{getOrDefault()}}, {{forEach()}}, {{replaceAll()}}, {{computeIfAbsent()}}, {{computeIfPresent()}}, {{compute()}}, {{merge()}}.
> We should try to write Infinispan-specific implementations wherever that would improve correctness and/or performance.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
7 years, 10 months
[JBoss JIRA] (ISPN-7526) Write skew check throws even if the previous value was not read
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-7526?page=com.atlassian.jira.plugin.... ]
Pedro Ruivo updated ISPN-7526:
------------------------------
Git Pull Request: https://github.com/infinispan/infinispan/pull/4908 (was: https://github.com/infinispan/infinispan/pull/4905)
> Write skew check throws even if the previous value was not read
> ---------------------------------------------------------------
>
> Key: ISPN-7526
> URL: https://issues.jboss.org/browse/ISPN-7526
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Transactions
> Affects Versions: 9.0.0.CR2, 8.2.6.Final
> Reporter: Radim Vansa
> Assignee: Radim Vansa
> Fix For: 9.0.0.Final
>
>
> When the entry was overwritten without reading previous value and then we read it (this read is handled from the current context), write skew check is still applied and can fail, despite that it's unnecessary.
> {code:java}
> cache.put("k", "init");
> tm.begin();
> cache.getAdvancedCache().withFlags(Flag.IGNORE_RETURN_VALUES).put("k", "v1");
> assertEquals("v1", cache.put("k", "v2"));
> Transaction tx = tm.suspend();
> assertEquals("init", cache.put("k", "other"));
> tm.resume(tx);
> tm.commit(); // fails with WriteSkewCheckException
> {code}
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
7 years, 10 months
[JBoss JIRA] (ISPN-7526) Write skew check throws even if the previous value was not read
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-7526?page=com.atlassian.jira.plugin.... ]
Pedro Ruivo updated ISPN-7526:
------------------------------
Status: Resolved (was: Pull Request Sent)
Fix Version/s: 9.0.0.Final
Resolution: Done
> Write skew check throws even if the previous value was not read
> ---------------------------------------------------------------
>
> Key: ISPN-7526
> URL: https://issues.jboss.org/browse/ISPN-7526
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Transactions
> Affects Versions: 9.0.0.CR2, 8.2.6.Final
> Reporter: Radim Vansa
> Assignee: Radim Vansa
> Fix For: 9.0.0.Final
>
>
> When the entry was overwritten without reading previous value and then we read it (this read is handled from the current context), write skew check is still applied and can fail, despite that it's unnecessary.
> {code:java}
> cache.put("k", "init");
> tm.begin();
> cache.getAdvancedCache().withFlags(Flag.IGNORE_RETURN_VALUES).put("k", "v1");
> assertEquals("v1", cache.put("k", "v2"));
> Transaction tx = tm.suspend();
> assertEquals("init", cache.put("k", "other"));
> tm.resume(tx);
> tm.commit(); // fails with WriteSkewCheckException
> {code}
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
7 years, 10 months
[JBoss JIRA] (ISPN-7556) Decoder2x.readOptionalParams should not call markReaderIndex()
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-7556?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-7556:
-------------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan/pull/4913
> Decoder2x.readOptionalParams should not call markReaderIndex()
> --------------------------------------------------------------
>
> Key: ISPN-7556
> URL: https://issues.jboss.org/browse/ISPN-7556
> Project: Infinispan
> Issue Type: Bug
> Components: Server
> Affects Versions: 9.0.0.CR2
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Fix For: 9.0.0.Final
>
>
> {{Decoder2x.readOptionalParams()}} calls {{buffer.markReaderIndex()}} after reading each parameter, but doesn't save the parameters anywhere. That means if the buffer doesn't contain all the parameters, the decode pass ends unsuccessfully, and the next decode pass doesn't see the parameters read by the previous pass.
> Higher-level methods like {{Decoder2x.readMaybeNamedFactory()}} also call {{readOptionalParams()}} without saving their own state first, meaning their state can be lost as well.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
7 years, 10 months
[JBoss JIRA] (ISPN-7556) Decoder2x.readOptionalParams should not call markReaderIndex()
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-7556?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-7556:
-------------------------------
Status: Open (was: New)
> Decoder2x.readOptionalParams should not call markReaderIndex()
> --------------------------------------------------------------
>
> Key: ISPN-7556
> URL: https://issues.jboss.org/browse/ISPN-7556
> Project: Infinispan
> Issue Type: Bug
> Components: Server
> Affects Versions: 9.0.0.CR2
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Fix For: 9.0.0.Final
>
>
> {{Decoder2x.readOptionalParams()}} calls {{buffer.markReaderIndex()}} after reading each parameter, but doesn't save the parameters anywhere. That means if the buffer doesn't contain all the parameters, the decode pass ends unsuccessfully, and the next decode pass doesn't see the parameters read by the previous pass.
> Higher-level methods like {{Decoder2x.readMaybeNamedFactory()}} also call {{readOptionalParams()}} without saving their own state first, meaning their state can be lost as well.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
7 years, 10 months