[JBoss JIRA] (ISPN-2891) Gap in time between commit of transaction and actual value update
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2891?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2891:
--------------------------------
Fix Version/s: 5.3.0.Beta2
(was: 5.3.0.Beta1)
> Gap in time between commit of transaction and actual value update
> -----------------------------------------------------------------
>
> Key: ISPN-2891
> URL: https://issues.jboss.org/browse/ISPN-2891
> Project: Infinispan
> Issue Type: Bug
> Components: Transactions
> Affects Versions: 5.2.1.Final
> Reporter: Jim Crossley
> Assignee: Galder Zamarreño
> Labels: 5.2.x
> Fix For: 5.3.0.Beta2, 5.3.0.Final
>
> Attachments: bad.log, good.log, pessimistic.log, ugly.log
>
>
> Since upgrading our AS7.2 dependency in Immutant (transitively pulling in 5.2.1.Final), one of our integration tests has begun failing intermittently on our CI server. We've yet to see the failure in local runs, only on CI, so I suspect there's something racist going on.
> The two tests (one for optimistic locking, the other for pessimistic) integrate an Infinispan cache (on which the Immutant cache is built) with HornetQ and XA transactions. A number of queue listeners respond to messages by attempting to increment a value in the cache. The failure occurs with both locking schemes, but much more often with optimistic.
> We've confirmed the failure on 5.2.2 as well.
> Attached you'll find three traces of the optimistic test: the good, the bad, and the ugly. All three correspond to this test: https://github.com/immutant/immutant/blob/31a2ef6222088ccb828898e9e3e4531...
> So you can correlate the log messages prefixed with "JC:" in the traces to the code. Note in particular the last two lines in locking.clj: a logged message containing the count, and then an assertion of the count. Note that the "bad" trace was an actual failing test, but the "ugly" trace was a successful test, even though the trace clearly shows the count logged as 2, not 3. The Infinispan TRACE output clearly shows the value as 3, hence the ugliness of this test.
> It's important to understand that the "work" function occurs within an XA transaction. This means, as I understand it, that if three messages are published to "/queue/done", the cached count should equal 3. Line #44 in locking.clj will block until it receives 3 messages, after which the cached count should be 3.
> These tests always pass locally. They only ever fail on CI, which runs *very* slowly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years
[JBoss JIRA] (ISPN-2863) org.infinispan.distexec.mapreduce.TopologyAwareTwoNodesMapReduceTest.testInvokeMapWithReduceExceptionPhaseInRemoteExecution
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2863?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2863:
--------------------------------
Fix Version/s: 5.3.0.Beta2
(was: 5.3.0.Beta1)
> org.infinispan.distexec.mapreduce.TopologyAwareTwoNodesMapReduceTest.testInvokeMapWithReduceExceptionPhaseInRemoteExecution
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-2863
> URL: https://issues.jboss.org/browse/ISPN-2863
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Execution and Map/Reduce
> Affects Versions: 5.2.2.Final
> Environment: OracleJDK1.7, Solaris10Sparc x64
> Reporter: Vitalii Chepeliuk
> Assignee: Vladimir Blagojevic
> Labels: testsuite_stability
> Fix For: 5.3.0.Beta2
>
>
> Error Message
> Method SimpleTwoNodesMapReduceTest.testInvokeMapWithReduceExceptionPhaseInRemoteExecution()[pri:0, instance:org.infinispan.distexec.mapreduce.TopologyAwareTwoNodesMapReduceTest@794d1a90] should have thrown an exception of class org.infinispan.CacheException
> Stacktrace
> org.testng.TestException:
> Method SimpleTwoNodesMapReduceTest.testInvokeMapWithReduceExceptionPhaseInRemoteExecution()[pri:0, instance:org.infinispan.distexec.mapreduce.TopologyAwareTwoNodesMapReduceTest@794d1a90] should have thrown an exception of class org.infinispan.CacheException
> at org.testng.internal.Invoker.handleInvocationResults(Invoker.java:1518)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:764)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:907)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1237)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> Standard Output
> [testng-TopologyAwareTwoNodesMapReduceTest] Test testInvokeMapWithReduceExceptionPhaseInRemoteExecution(org.infinispan.distexec.mapreduce.TopologyAwareTwoNodesMapReduceTest) failed.
> 2013-02-25 15:13:34,026 ERROR [UnitTestTestNGListener] (testng-TopologyAwareTwoNodesMapReduceTest) Test testInvokeMapWithReduceExceptionPhaseInRemoteExecution(org.infinispan.distexec.mapreduce.TopologyAwareTwoNodesMapReduceTest) failed.
> org.testng.TestException:
> Method SimpleTwoNodesMapReduceTest.testInvokeMapWithReduceExceptionPhaseInRemoteExecution()[pri:0, instance:org.infinispan.distexec.mapreduce.TopologyAwareTwoNodesMapReduceTest@794d1a90] should have thrown an exception of class org.infinispan.CacheException
> at org.testng.internal.Invoker.handleInvocationResults(Invoker.java:1518)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:764)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:907)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1237)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> Test suite progress: tests succeeded: 1680, failed: 1, skipped: 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years
[JBoss JIRA] (ISPN-2976) Log4J dependencies in codebase to be cleaned up
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2976?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2976:
--------------------------------
Fix Version/s: 5.3.0.Beta2
(was: 5.3.0.Beta1)
> Log4J dependencies in codebase to be cleaned up
> -----------------------------------------------
>
> Key: ISPN-2976
> URL: https://issues.jboss.org/browse/ISPN-2976
> Project: Infinispan
> Issue Type: Task
> Affects Versions: 5.2.5.Final
> Reporter: Manik Surtani
> Assignee: Mircea Markus
> Fix For: 5.3.0.Beta2, 5.3.0.Final
>
>
> When attempting to move to Log4J 2.0, I've noticed a number of hard deps on log4j classes.
> {{SampleConfigFilesCorrectnessTest}} - this class makes use of a custom appender to analyse what a user is being warned of when a config file is parsed. Why are we using Log4J for this? Our own logging interface should be mocked and messages captured directly.
> {{RehashStressTest}} and {{NucleotideCache}} - seems like a bug, I presume the author intended to use {{org.infinispan.logging.Log}}.
> {{CompressedFileAppender}} and {{ThreadNameFilter}}- can this be written in a way that works with Log4J 1.x as well as 2.x? Or have the SPIs changed that much?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years
[JBoss JIRA] (ISPN-2965) L1 and early invalidation leaves inconsistent state
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2965?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2965:
--------------------------------
Fix Version/s: 5.3.0.Beta2
(was: 5.3.0.Beta1)
> L1 and early invalidation leaves inconsistent state
> ---------------------------------------------------
>
> Key: ISPN-2965
> URL: https://issues.jboss.org/browse/ISPN-2965
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache, Transactions
> Affects Versions: 5.2.1.Final
> Reporter: Sebastian Tusk
> Assignee: Adrian Nistor
> Labels: 5.2.x
> Fix For: 5.3.0.Beta2, 5.3.0.Final
>
>
> In a distributed transactional cache with L1 enabled I can observe the following.
> Prepare cache by adding an entry with Cache.put( k, v1 ).
> 1. Node B starts with adding a changed value. Cache.put( k, v2 )
> 2. Node B TxDistributionInterceptor.visitPrepareCommand flushL1Caches sends invalidations.
> 3. Node A calls Cache.get( k ) retrieves v1 and stores this value in L1.
> 4. Node B proceeds with transaction.
> The result is that Node A answers subsequent Cache.get(k) with v1 and Node B answers with v2.
> It seems the invalidation is either send to early or must be synchronized in some way with the transaction.
> Cache config:
> <namedCache name="entity">
> <jmxStatistics enabled="true" />
> <clustering mode="dist">
> <stateTransfer fetchInMemoryState="false" timeout="20000" />
> <async />
> <l1 enabled="true" />
> <hash numOwners="1"/>
> </clustering>
> <locking isolationLevel="READ_COMMITTED"
> lockAcquisitionTimeout="15000" useLockStriping="false" />
> <eviction maxEntries="10000" strategy="LRU" />
> <expiration maxIdle="100000" wakeUpInterval="5000"/>
> <storeAsBinary storeKeysAsBinary="true" storeValuesAsBinary="false" enabled="false" />
> <transaction transactionMode="TRANSACTIONAL" autoCommit="false" lockingMode="OPTIMISTIC"/>
> </namedCache>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years
[JBoss JIRA] (ISPN-2955) Async marshalling executor retry when queue fills
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2955?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2955:
--------------------------------
Fix Version/s: 5.3.0.Beta2
(was: 5.3.0.Beta1)
> Async marshalling executor retry when queue fills
> -------------------------------------------------
>
> Key: ISPN-2955
> URL: https://issues.jboss.org/browse/ISPN-2955
> Project: Infinispan
> Issue Type: Enhancement
> Components: Marshalling
> Affects Versions: 5.2.5.Final
> Reporter: Manik Surtani
> Assignee: Manik Surtani
> Fix For: 5.3.0.Beta2, 5.3.0.Final
>
>
> When using an async transport and async marshalling, an executor is used to process the marshalling task in a separate thread and the caller's thread is allowed to return immediately.
> When the executor's queue fills and the queue cannot accept any more tasks, it throws a {{RejectedExecutionException}}, causing a very bad user/developer experience. A more correct approach to this is to catch the {{RejectedExecutionException}}, block and retry the task submission.
> The end result is that, in the degenerate case (when the executor queue is full) instead of throwing exceptions, those invocations will perform slightly slower.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years
[JBoss JIRA] (ISPN-2964) putForExternalRead to L1 not invalidated
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2964?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2964:
--------------------------------
Fix Version/s: 5.3.0.Beta2
(was: 5.3.0.Beta1)
> putForExternalRead to L1 not invalidated
> ----------------------------------------
>
> Key: ISPN-2964
> URL: https://issues.jboss.org/browse/ISPN-2964
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 5.2.1.Final
> Reporter: Sebastian Tusk
> Assignee: Mircea Markus
> Fix For: 5.3.0.Beta2, 5.3.0.Final
>
> Attachments: SebastianTusk_ISPN-2964_fix.patch
>
>
> With transactional distributed caches it happens that Cache.putForExternalRead(k,v) places an entry into L1 that never gets invalidated. It seems to happen when the the owner of k doesn't have the entry. In this case the non owner puts k into his cache without having the owner registering this. Usually the owner stores all requesters in L1ManagerImpl.addRequester and sends out invalidations to the requesters. What should happen is that the entry is replicated to the owner.
> Cache config:
> <namedCache name="entity">
> <jmxStatistics enabled="true" />
> <clustering mode="dist">
> <stateTransfer fetchInMemoryState="false" timeout="20000" />
> <async />
> <l1 enabled="true" />
> <hash numOwners="1"/>
> </clustering>
> <locking isolationLevel="READ_COMMITTED"
> lockAcquisitionTimeout="15000" useLockStriping="false" />
> <eviction maxEntries="10000" strategy="LRU" />
> <expiration maxIdle="100000" wakeUpInterval="5000"/>
> <storeAsBinary storeKeysAsBinary="true" storeValuesAsBinary="false" enabled="false" />
> <transaction transactionMode="TRANSACTIONAL" autoCommit="false" lockingMode="OPTIMISTIC"/>
> </namedCache>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years
[JBoss JIRA] (ISPN-2971) AsyncStoreStressTest is broken
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2971?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2971:
--------------------------------
Fix Version/s: 5.3.0.Beta2
(was: 5.3.0.Beta1)
> AsyncStoreStressTest is broken
> ------------------------------
>
> Key: ISPN-2971
> URL: https://issues.jboss.org/browse/ISPN-2971
> Project: Infinispan
> Issue Type: Bug
> Components: Loaders and Stores
> Affects Versions: 5.2.5.Final
> Reporter: Adrian Nistor
> Assignee: Adrian Nistor
> Labels: testsuite
> Fix For: 5.3.0.Beta2
>
>
> This test is not run during normal test suite, but we still need to fix this failure and the leaked threads (see System.exit(0) in main method):
> {code}
> testReadWriteRemove(org.infinispan.stress.AsyncStoreStressTest) Time elapsed: 72.377 sec <<< FAILURE!
> java.lang.UnsupportedOperationException
> at java.util.AbstractMap$SimpleImmutableEntry.setValue(AbstractMap.java:726)
> at org.infinispan.stress.AsyncStoreStressTest.testReadWriteRemove(AsyncStoreStressTest.java:140)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years
[JBoss JIRA] (ISPN-2956) putIfAbsent on Hot Rod Java client doesn't reliably fulfil contract
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2956?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2956:
--------------------------------
Fix Version/s: 5.3.0.Beta2
(was: 5.3.0.Beta1)
> putIfAbsent on Hot Rod Java client doesn't reliably fulfil contract
> -------------------------------------------------------------------
>
> Key: ISPN-2956
> URL: https://issues.jboss.org/browse/ISPN-2956
> Project: Infinispan
> Issue Type: Bug
> Reporter: Galder Zamarreño
> Assignee: Galder Zamarreño
> Labels: hotrod-java-client, remote-clients
> Fix For: 5.3.0.Beta2, 5.3.0.Final
>
>
> Hot Rod's putIfAbsent might have issues on some edge cases:
> {quote}I want to know whether the putting entry already exists in the remote
> cache cluster, or not.
> I thought that RemoteCache.putIfAbsent() would be useful for that
> purpose, i.e.,
> {code}
> if (remoteCache.putIfAbsent(k,v) == null) {
> // new entry.
> } else {
> // k already exists.
> }
> {code}
> But no.
> The putIfAbsent() for new entry may return non-null value, if one of the
> server crushed while putting.
> The behavior is like the following:
> 1. Client do putIfAbsent(k,v).
> 2. The server receives the request and sends replication requests to
> other servers. If the server crushed before completing replication, some
> servers own that (k,v), but others not.
> 3. Client receives the error. The putIfAbsent() internally retries the
> same request to the next server in the cluster server list.
> 4. If the next server owns the (k,v), the putIfAbsent() returns the
> replicated (k,v) at the step 2, without any error.
> So, putIfAbsent() is not reliable for knowing whether the putting entry
> is *exactly* new or not.
> Does anyone have any idea/workaround for this purpose?{quote}
> A workaround is to do this:
> {quote}We got a simple solution, which can be applied to our customer's application.
> If each value part of putting (k,v) is unique or contains unique value,
> the client can do *double check* wether the entry is new.
> {code}
> val = System.nanoTime(); // or uuid is also useful.
> if ((ret = cache.putIfAbsent(key, val)) == null
> || ret.equals(val)) {
> // new entry, if the return value is just the same.
> } else {
> // key already exists.
> }
> {code}
> We are proposing this workaround which almost works fine.{quote}
> However, this is a bit of a cludge.
> Hot Rod should be improved with an operation that allows a version to be passed when entry is created, instead of relying on the client generating it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years
[JBoss JIRA] (ISPN-3048) Eviction needs to be transactional
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-3048?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-3048:
--------------------------------
Fix Version/s: 5.3.0.Beta2
(was: 5.3.0.Beta1)
> Eviction needs to be transactional
> ----------------------------------
>
> Key: ISPN-3048
> URL: https://issues.jboss.org/browse/ISPN-3048
> Project: Infinispan
> Issue Type: Bug
> Components: Eviction
> Affects Versions: 5.3.0.Alpha1
> Reporter: Paul Ferraro
> Assignee: Dan Berindei
> Priority: Critical
> Fix For: 5.3.0.Beta2, 5.3.0.Final
>
>
> Currently, Infinispan eviction is non-transactional. This makes Infinispan's eviction manager virtually unusable, since non-transactional eviction can cause phantom reads and data loss because it violates the isolation of concurrent transactions. This is especially problematic when using a passivation-enabled cache store. In this case, a cache eviction/passivation can cause a concurrently executed cache retrieval to return null - even though the act of passivation does not change the data - it only changes where it is stored.
> We work around this in the AS by performing eviction manually, using pessimistic locking in combination with eager lock acquisition prior to eviction. This is unfortunate, since it prevents me from leveraging Infinispan's build-in eviction strategies.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years