[JBoss JIRA] (ISPN-4426) Transaction replayed but not committed
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-4426?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-4426:
------------------------------------
I found a little more information in the log: the CommitCommand is actually executed 3 times, because both the originator and and the old owners forward the command to the new owners in the write CH (including the old owners):
{noformat}
10:07:31,964 TRACE [org.infinispan.remoting.InboundInvocationHandlerImpl] (remote-thread-5) Calling perform() on CommitCommand {gtx=GlobalTransaction:<edg-perf03-58446>:23655:remote, cacheName='testCache', topologyId=41}
10:07:31,968 TRACE [org.infinispan.remoting.InboundInvocationHandlerImpl] (remote-thread-23) Calling perform() on CommitCommand {gtx=GlobalTransaction:<edg-perf03-58446>:23655:remote, cacheName='testCache', topologyId=41}
10:07:32,408 TRACE [org.infinispan.remoting.InboundInvocationHandlerImpl] (remote-thread-23) Calling perform() on CommitCommand {gtx=GlobalTransaction:<edg-perf03-58446>:23655:remote, cacheName='testCache', topologyId=41}
{noformat}
The problem seems to be caused by the ISPN-4137 fix, as the transaction is now only marked as completed in TxInterceptor. The first thread (remote-thread-5) decides to replay the PrepareCommand and updates the {{lookedUpEntriesTopology}} to the new topology. Then the second thread (remote-thread-23) sees that it doesn't need to replay the PrepareCommand and tries to execute the CommitCommand directly. The modifications list is empty, however, so nothing is committed, but the transaction is marked as completed, which then prevents remote-thread-5 (and again remote-thread-23) from doing anything else:
{noformat}
10:07:31,974 DEBUG [org.infinispan.statetransfer.StateTransferInterceptor] (remote-thread-5) Replaying the transactions received as a result of state transfer PrepareCommand {modifications=[PutKeyValueCommand{key=key_0000000000001531, value=[21 #21: 246, 320, 846, 1083, 1256, 1398, 1526, 1667, 1816, 2533, 2557, 2724, 2804, 2840, 2921, 3492, 3613, 3659, 4413, 4677, 5760, ], flags=[SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP], putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedMetadata{version=null}, successful=true}, ... onePhaseCommit=false, gtx=GlobalTransaction:<edg-perf03-58446>:23655:remote, cacheName='testCache', topologyId=-1}
10:07:31,974 TRACE [org.infinispan.commands.tx.PrepareCommand] (remote-thread-5) Invoking remotely originated prepare: PrepareCommand {modifications=[PutKeyValueCommand{key=key_0000000000001531, value=[21 #21: 246, 320, 846, 1083, 1256, 1398, 1526, 1667, 1816, 2533, 2557, 2724, 2804, 2840, 2921, 3492, 3613, 3659, 4413, 4677, 5760, ], flags=[SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP], putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedMetadata{version=null}, successful=true}, ... ], onePhaseCommit=false, gtx=GlobalTransaction:<edg-perf03-58446>:23655:remote, cacheName='testCache', topologyId=-1} with invocation context: org.infinispan.context.impl.RemoteTxInvocationContext@7a5688e0
10:07:31,976 DEBUG [org.infinispan.statetransfer.StateTransferInterceptor] (remote-thread-23) handleTopologyAffectedCommand for command CommitCommand {gtx=GlobalTransaction:<edg-perf03-58446>:23655:remote, cacheName='testCache', topologyId=41}, origin edg-perf03-58446
10:07:31,980 DEBUG [org.infinispan.interceptors.EntryWrappingInterceptor] (remote-thread-23) Commiting CommitCommand
10:07:31,980 DEBUG [org.infinispan.statetransfer.StateTransferInterceptor] (remote-thread-23) handled command CommitCommand {gtx=GlobalTransaction:<edg-perf03-58446>:23655:remote, cacheName='testCache', topologyId=41}, origin edg-perf03-58446
10:07:32,013 DEBUG [org.infinispan.statetransfer.StateTransferInterceptor] (remote-thread-5) Prepare command replayed
10:07:32,013 DEBUG [org.infinispan.statetransfer.StateTransferInterceptor] (remote-thread-5) handleTopologyAffectedCommand for command CommitCommand {gtx=GlobalTransaction:<edg-perf03-58446>:23655:remote, cacheName='testCache', topologyId=41}, origin edg-perf03-58446
10:07:32,014 DEBUG [org.infinispan.statetransfer.StateTransferInterceptor] (remote-thread-5) handled command CommitCommand {gtx=GlobalTransaction:<edg-perf03-58446>:23655:remote, cacheName='testCache', topologyId=41}, origin edg-perf03-58446
10:07:32,408 TRACE [org.infinispan.commands.tx.AbstractTransactionBoundaryCommand] (remote-thread-23) Did not find a RemoteTransaction for GlobalTransaction:<edg-perf03-58446>:23655:remote
{noformat}
> Transaction replayed but not committed
> --------------------------------------
>
> Key: ISPN-4426
> URL: https://issues.jboss.org/browse/ISPN-4426
> Project: Infinispan
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Components: State Transfer
> Affects Versions: 7.0.0.Alpha4
> Reporter: Radim Vansa
> Assignee: Dan Berindei
> Priority: Critical
> Labels: 63gablocker
>
> Dist TX cache, node C is joining. In previous topology, entry is owned by A (primary) and B (backup). In new topology, primary ownership is transferred to C, B stays backup.
> 1. TX is prepared in old topology and is being committed, when topology changes
> 2. on C (the new owner), the TX info is received and later even the old entry
> 3. C receives the CommitCommand, therefore, it correctly replays the PrepareCommand.
> 4. When the entries are about to be committed, in TxInterceptor the transaction is found to be already completed as it has lower TxID.
> Result: the transaction is not being executed and stale data stay on the node (with my algortihm it eventually led to complete entry loss).
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
11 years, 9 months
[JBoss JIRA] (ISPN-4247) JPA cachestore fails with Oracle12c and PostgreadPlus 9
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-4247?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant commented on ISPN-4247:
---------------------------------------
I'm thinking we shouldn't really fix this since the dialect can be specified in the persistence.xml and it's a perfectly acceptable workaround.
> JPA cachestore fails with Oracle12c and PostgreadPlus 9
> -------------------------------------------------------
>
> Key: ISPN-4247
> URL: https://issues.jboss.org/browse/ISPN-4247
> Project: Infinispan
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Components: Core
> Reporter: Vojtech Juranek
> Assignee: Dan Berindei
> Labels: 63gablocker
> Fix For: 7.0.0.Alpha4, 7.0.0.Final
>
>
> JPA cache store fails on Oracle12c and PostgreadPlus 9 as Hibernate is not able to determine the dialect to be used:
> {noformat}
> [java] Caused by: org.hibernate.HibernateException: Unable to determine Dialect to use [name=Oracle, majorVersion=12]; user must register resolver or explicitly set 'hibernate.dialect'
> {noformat}
> resp.
> {noformat}
> [java] Caused by: org.hibernate.HibernateException: Unable to determine Dialect to use [name=EnterpriseDB, majorVersion=9]; user must register resolver or explicitly set 'hibernate.dialect'
> {noformat}
> To fix it, HIbernate upgrade is needed. PotgresPlus is fixed in 4.3.0.Beta4 (tracking JIRA is [HHH-8349|https://hibernate.atlassian.net/browse/HHH-8349]), Oracle12c is fixed in Hibernate 5.0.0 (tracking JIRA is [HHH-9044|https://hibernate.atlassian.net/browse/HHH-9044])
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
11 years, 9 months
[JBoss JIRA] (ISPN-3441) Silence OutdatedTopologyExceptions
by Radim Vansa (JIRA)
[ https://issues.jboss.org/browse/ISPN-3441?page=com.atlassian.jira.plugin.... ]
Radim Vansa updated ISPN-3441:
------------------------------
Affects Version/s: 7.0.0.Alpha4
(was: 6.0.0.Alpha3)
> Silence OutdatedTopologyExceptions
> ----------------------------------
>
> Key: ISPN-3441
> URL: https://issues.jboss.org/browse/ISPN-3441
> Project: Infinispan
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Components: State Transfer
> Affects Versions: 6.0.0.Final, 7.0.0.Alpha4
> Reporter: Michal Linhard
> Assignee: Dan Berindei
> Priority: Minor
>
> from:
> https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/JDG/view/RESILIENCE...
> when crashing one node of four in dist sync (owners=2) config resilience test I get:
> {code}
> 06:47:16,805 ERROR [org.infinispan.remoting.InboundInvocationHandlerImpl] (remote-thread-20) Exception executing command: org.infinispan.statetransfer.OutdatedTopologyException: Cache topology changed while the command was executing: expected 5, got 6
> at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:126) [infinispan-core-6.0.0.Alpha3-redhat-1.jar:6.0.0.Alpha3-redhat-1]
> at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:72) [infinispan-core-6.0.0.Alpha3-redhat-1.jar:6.0.0.Alpha3-redhat-1]
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:62) [infinispan-core-6.0.0.Alpha3-redhat-1.jar:6.0.0.Alpha3-redhat-1]
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.0.0.Alpha3-redhat-1.jar:6.0.0.Alpha3-redhat-1]
> at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:292) [infinispan-core-6.0.0.Alpha3-redhat-1.jar:6.0.0.Alpha3-redhat-1]
> at org.infinispan.interceptors.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:374) [infinispan-core-6.0.0.Alpha3-redhat-1.jar:6.0.0.Alpha3-redhat-1]
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:157) [infinispan-core-6.0.0.Alpha3-redhat-1.jar:6.0.0.Alpha3-redhat-1]
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:62) [infinispan-core-6.0.0.Alpha3-redhat-1.jar:6.0.0.Alpha3-redhat-1]
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.0.0.Alpha3-redhat-1.jar:6.0.0.Alpha3-redhat-1]
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:47) [infinispan-core-6.0.0.Alpha3-redhat-1.jar:6.0.0.Alpha3-redhat-1]
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:62) [infinispan-core-6.0.0.Alpha3-redhat-1.jar:6.0.0.Alpha3-redhat-1]
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.0.0.Alpha3-redhat-1.jar:6.0.0.Alpha3-redhat-1]
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112) [infinispan-core-6.0.0.Alpha3-redhat-1.jar:6.0.0.Alpha3-redhat-1]
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:32) [infinispan-core-6.0.0.Alpha3-redhat-1.jar:6.0.0.Alpha3-redhat-1]
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:62) [infinispan-core-6.0.0.Alpha3-redhat-1.jar:6.0.0.Alpha3-redhat-1]
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.0.0.Alpha3-redhat-1.jar:6.0.0.Alpha3-redhat-1]
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112) [infinispan-core-6.0.0.Alpha3-redhat-1.jar:6.0.0.Alpha3-redhat-1]
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:32) [infinispan-core-6.0.0.Alpha3-redhat-1.jar:6.0.0.Alpha3-redhat-1]
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:62) [infinispan-core-6.0.0.Alpha3-redhat-1.jar:6.0.0.Alpha3-redhat-1]
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.0.0.Alpha3-redhat-1.jar:6.0.0.Alpha3-redhat-1]
> at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:200) [infinispan-core-6.0.0.Alpha3-redhat-1.jar:6.0.0.Alpha3-redhat-1]
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:128) [infinispan-core-6.0.0.Alpha3-redhat-1.jar:6.0.0.Alpha3-redhat-1]
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:62) [infinispan-core-6.0.0.Alpha3-redhat-1.jar:6.0.0.Alpha3-redhat-1]
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.0.0.Alpha3-redhat-1.jar:6.0.0.Alpha3-redhat-1]
> at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:148) [infinispan-core-6.0.0.Alpha3-redhat-1.jar:6.0.0.Alpha3-redhat-1]
> at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:134) [infinispan-core-6.0.0.Alpha3-redhat-1.jar:6.0.0.Alpha3-redhat-1]
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:62) [infinispan-core-6.0.0.Alpha3-redhat-1.jar:6.0.0.Alpha3-redhat-1]
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.0.0.Alpha3-redhat-1.jar:6.0.0.Alpha3-redhat-1]
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:106) [infinispan-core-6.0.0.Alpha3-redhat-1.jar:6.0.0.Alpha3-redhat-1]
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:70) [infinispan-core-6.0.0.Alpha3-redhat-1.jar:6.0.0.Alpha3-redhat-1]
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:32) [infinispan-core-6.0.0.Alpha3-redhat-1.jar:6.0.0.Alpha3-redhat-1]
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:62) [infinispan-core-6.0.0.Alpha3-redhat-1.jar:6.0.0.Alpha3-redhat-1]
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:321) [infinispan-core-6.0.0.Alpha3-redhat-1.jar:6.0.0.Alpha3-redhat-1]
> at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommand(BaseRpcInvokingCommand.java:39) [infinispan-core-6.0.0.Alpha3-redhat-1.jar:6.0.0.Alpha3-redhat-1]
> at org.infinispan.commands.remote.SingleRpcCommand.perform(SingleRpcCommand.java:48) [infinispan-core-6.0.0.Alpha3-redhat-1.jar:6.0.0.Alpha3-redhat-1]
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:97) [infinispan-core-6.0.0.Alpha3-redhat-1.jar:6.0.0.Alpha3-redhat-1]
> at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:46) [infinispan-core-6.0.0.Alpha3-redhat-1.jar:6.0.0.Alpha3-redhat-1]
> at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:169) [infinispan-core-6.0.0.Alpha3-redhat-1.jar:6.0.0.Alpha3-redhat-1]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25]
> at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]
> {code}
> Isn't this similar candidate for silencing as SuspectException (ISPN-2577) ?
> The difference here is that it doesn't bubble up to hot rod user.
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
11 years, 9 months
[JBoss JIRA] (ISPN-4424) ReplaceIfUnmodified is broken under concurrent execution
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-4424?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-4424:
-----------------------------------------------
Tristan Tarrant <ttarrant(a)redhat.com> changed the Status of [bug 1110647|https://bugzilla.redhat.com/show_bug.cgi?id=1110647] from NEW to ASSIGNED
> ReplaceIfUnmodified is broken under concurrent execution
> --------------------------------------------------------
>
> Key: ISPN-4424
> URL: https://issues.jboss.org/browse/ISPN-4424
> Project: Infinispan
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Components: Remote Protocols
> Affects Versions: 6.0.2.Final, 7.0.0.Alpha4
> Reporter: Galder Zamarreño
> Assignee: Galder Zamarreño
> Fix For: 7.0.0.Final
>
>
> Versioned update with a multi threaded Hot Rod client results in inconsistency. Some replaceWithVersion return true ignoring a version update executed in another thread. Here's a log excerpt of a concurrency stress test:
> ```
> 2014-06-20 16:16:56,798 INFO [PutFromNull] (pool-7-thread-10) count=462,prev=462,new=463
> 2014-06-20 16:16:56,820 INFO [PutFromNull] (pool-7-thread-9) count=463,prev=463,new=464
> 2014-06-20 16:16:56,831 INFO [PutFromNull] (pool-7-thread-2) count=464,prev=463,new=464
> 2014-06-20 16:16:56,845 INFO [PutFromNull] (pool-7-thread-9) count=465,prev=464,new=465
> ```
> Here you see two threads applying the same replacement, from 463 to 464.
> The issue appears a result of a race condition in Hot Rod server's protocol decoder. When replaceIfUmodified is received, the cache entry is retrieved to verify whether the version in the server and the version sent in the command match. However, the cache entry retrieved is mutable, and the value could change midway through this operation as a result of another thread updating the value. Please find below some log snippets showing this.
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
11 years, 9 months
[JBoss JIRA] (ISPN-4431) CLI codecs do set entry version in metadata
by Tristan Tarrant (JIRA)
Tristan Tarrant created ISPN-4431:
-------------------------------------
Summary: CLI codecs do set entry version in metadata
Key: ISPN-4431
URL: https://issues.jboss.org/browse/ISPN-4431
Project: Infinispan
Issue Type: Bug
Security Level: Public (Everyone can see)
Reporter: Tristan Tarrant
Assignee: Mircea Markus
When the CLI writes to the cache using one of the server codecs it must set a version, because the server protocol handlers expect to find it
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
11 years, 9 months