[JBoss JIRA] (ISPN-2608) VersionedDistStateTransferTest intermittent failure
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-2608?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-2608:
------------------------------------
Pedro, your test failure seems to be caused by ISPN-3315: the test tries to check the results of state transfer before the state transfer is actually over, and node B sends a remote get command to A for key "hello" just after A installed the new CH in which it is no longer an owner.
{noformat}
07:25:24.257 TRACE [CacheTopology] (remote-thread-1,NodeB) Current consistent hash's routing table: 0: 0 1, 1: 0 1, 2: 0 1, 3: 0 1, 4: 0 1, 5: 0 1, 6: 0 1, 7: 0 1, 8: 0 1, 9: 0 3, 10: 0, 11: 0, 12: 3 0, 13: 3 0, 14: 3 0, 15: 2 0, 16: 2 0, 17: 2 0, 18: 2 0, 19: 2 0, 20: 0, 21: 0, 22: 0, *23: 0*, 24: 3, 25: 3, 26: 3, 27: 3, 28: 3, 29: 3, 30: 1 2, 31: 1 2, 32: 1 2, 33: 1 2, 34: 1 2, 35: 1 2, 36: 1 2, 37: 1 2, 38: 1 2, 39: 1 2, 40: 1, 41: 1, 42: 3 1, 43: 3 1, 44: 3 1, 45: 2 3, 46: 2 3, 47: 2 3, 48: 2 3, 49: 2, 50: 3, 51: 2, 52: 3, 53: 3, 54: 3, 55: 2, 56: 2, 57: 3, 58: 3 2, 59: 3
07:25:25.046 DEBUG [VersionedDistStateTransferTest] (testng) Leaver stopped, checking transferred data
07:25:25.048 TRACE [CacheTopology] (asyncTransportThread-3,NodeA) Current consistent hash's routing table: 0: 0 1, 1: 0 1, 2: 0 1, 3: 0 1, 4: 0 1, 5: 0 1, 6: 0 1, 7: 0 1, 8: 0 1, 9: 0 3, 10: 0 1, 11: 0 3, 12: 3 0, 13: 3 0, 14: 3 0, 15: 2 0, 16: 2 0, 17: 2 0, 18: 2 0, 19: 2 0, 20: 0 1, 21: 0 2, 22: 0 3, *23: 2 3*, 24: 3 0, 25: 3 0, 26: 3 0, 27: 3 1, 28: 3 2, 29: 3 0, 30: 1 2, 31: 1 2, 32: 1 2, 33: 1 2, 34: 1 2, 35: 1 2, 36: 1 2, 37: 1 2, 38: 1 2, 39: 1 2, 40: 1 3, 41: 1 2, 42: 1 3, 43: 1 3, 44: 1 3, 45: 2 3, 46: 2 3, 47: 2 3, 48: 2 3, 49: 2 1, 50: 3 0, 51: 2 3, 52: 3 1, 53: 3 2, 54: 3 0, 55: 2 3, 56: 2 1, 57: 3 2, 58: 2 3, 59: 3 0
07:25:25.067 TRACE [RpcManagerImpl] (testng) NodeB-45000 invoking ClusteredGetCommand{key=MagicKey#hello{7919f024@NodeC-31313/23}, flags=null} to recipient list [NodeA-1669] with options RpcOptions{timeout=15000, unit=MILLISECONDS, fifoOrder=false, totalOrder=false, responseFilter=org.infinispan.remoting.responses.ClusteredGetResponseValidityFilter@2b51ed99, responseMode=WAIT_FOR_VALID_RESPONSE, skipReplicationQueue=false}
07:25:25.067 TRACE [CacheTopology] (remote-thread-1,NodeB) Current consistent hash's routing table: 0: 0 1, 1: 0 1, 2: 0 1, 3: 0 1, 4: 0 1, 5: 0 1, 6: 0 1, 7: 0 1, 8: 0 1, 9: 0 3, 10: 0 1, 11: 0 3, 12: 3 0, 13: 3 0, 14: 3 0, 15: 2 0, 16: 2 0, 17: 2 0, 18: 2 0, 19: 2 0, 20: 0 1, 21: 0 2, 22: 0 3, *23: 2 3*, 24: 3 0, 25: 3 0, 26: 3 0, 27: 3 1, 28: 3 2, 29: 3 0, 30: 1 2, 31: 1 2, 32: 1 2, 33: 1 2, 34: 1 2, 35: 1 2, 36: 1 2, 37: 1 2, 38: 1 2, 39: 1 2, 40: 1 3, 41: 1 2, 42: 1 3, 43: 1 3, 44: 1 3, 45: 2 3, 46: 2 3, 47: 2 3, 48: 2 3, 49: 2 1, 50: 3 0, 51: 2 3, 52: 3 1, 53: 3 2, 54: 3 0, 55: 2 3, 56: 2 1, 57: 3 2, 58: 2 3, 59: 3 0
07:25:25.076 DEBUG [StateConsumerImpl] (asyncTransportThread-3,NodeA) Removing state for segments not in [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 16, 19, 18, 21, 20, 22, 25, 24, 26, 29, 50, 54, 59] or [23] for cache ___defaultcache
07:25:25.080 TRACE [GetKeyValueCommand] (OOB-4,ISPN,NodeA-1669) Entry not found
07:25:25.080 TRACE [RpcManagerImpl] (testng) Response(s) to ClusteredGetCommand{key=MagicKey#hello{7919f024@NodeC-31313/23}, flags=null} is {}
{noformat}
I'll change this test to wait for the state transfer to end before checking the transferred state, and I'll add a separate test for ISPN-3315 to check the behaviour of get during the state transfer.
> VersionedDistStateTransferTest intermittent failure
> ---------------------------------------------------
>
> Key: ISPN-2608
> URL: https://issues.jboss.org/browse/ISPN-2608
> Project: Infinispan
> Issue Type: Bug
> Components: State transfer, Test Suite
> Affects Versions: 5.2.0.Beta5
> Reporter: Mircea Markus
> Assignee: Dan Berindei
> Labels: testsuite_stability
> Fix For: 6.0.0.Alpha1
>
> Attachments: vdstt.log.gz, VersionedDistStateTransferTest.testStateTransfer.tar.gz
>
>
> Failure:
> {quote}
> org.infinispan.CacheException: Could not commit implicit transaction
> at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1168)
> at org.infinispan.CacheImpl.putInternal(CacheImpl.java:741)
> at org.infinispan.CacheImpl.put(CacheImpl.java:735)
> at org.infinispan.CacheImpl.put(CacheImpl.java:729)
> at org.infinispan.CacheSupport.put(CacheSupport.java:53)
> at org.infinispan.container.versioning.VersionedDistStateTransferTest.testStateTransfer(VersionedDistStateTransferTest.java:101)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:715)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:907)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1237)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:680)
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 9 months
[JBoss JIRA] (ISPN-2618) org.infinispan.distribution.rehash.L1OnRehashDisabledTest.testInvalidationBehaviorOnRehash randomly fails
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-2618?page=com.atlassian.jira.plugin.... ]
Dan Berindei resolved ISPN-2618.
--------------------------------
Resolution: Out of Date
The test has never failed in CI since we switched to TeamCity.
> org.infinispan.distribution.rehash.L1OnRehashDisabledTest.testInvalidationBehaviorOnRehash randomly fails
> ----------------------------------------------------------------------------------------------------------
>
> Key: ISPN-2618
> URL: https://issues.jboss.org/browse/ISPN-2618
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 5.2.0.Beta5
> Reporter: Anna Manukyan
> Assignee: Dan Berindei
> Labels: testsuite_stability
> Fix For: 6.0.0.Alpha1
>
>
> The test is randomly failing on different environments.
> The error is:
> {code}
> Error Message
> Fail on non-owner cache L1OnRehashDisabledTest-NodeA-8972: dc.get(MagicKey#k2{18b213c7@L1OnRehashDisabledTest-NodeC-18200}) returned MortalCacheEntry{key=MagicKey#k2{18b213c7@L1OnRehashDisabledTest-NodeC-18200}, value=MortalCacheValue{value=v2, lifespan=600000, created=1355149157148}}!
> Stacktrace
> java.lang.AssertionError: Fail on non-owner cache L1OnRehashDisabledTest-NodeA-8972: dc.get(MagicKey#k2{18b213c7@L1OnRehashDisabledTest-NodeC-18200}) returned MortalCacheEntry{key=MagicKey#k2{18b213c7@L1OnRehashDisabledTest-NodeC-18200}, value=MortalCacheValue{value=v2, lifespan=600000, created=1355149157148}}!
> at org.infinispan.distribution.BaseDistFunctionalTest.assertOwnershipAndNonOwnership(BaseDistFunctionalTest.java:212)
> at org.infinispan.distribution.rehash.L1OnRehashTest.testInvalidationBehaviorOnRehash(L1OnRehashTest.java:117)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:715)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:907)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1237)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 9 months
[JBoss JIRA] (ISPN-3321) NPE in MapReduceTask reduce phase
by Alan Field (JIRA)
[ https://issues.jboss.org/browse/ISPN-3321?page=com.atlassian.jira.plugin.... ]
Alan Field commented on ISPN-3321:
----------------------------------
Here's another job with three instances of this exception:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/jdg-radargun-mapredu...
These happened when running a job that fills the cache with data equal to 33% of the Java heap and then executes a word count Map?Reduce task with 6, 7, and 8 nodes. (The exception doesn't occur when running with 1 to 5 nodes)
> NPE in MapReduceTask reduce phase
> ---------------------------------
>
> Key: ISPN-3321
> URL: https://issues.jboss.org/browse/ISPN-3321
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Execution and Map/Reduce
> Affects Versions: 5.3.0.Final
> Reporter: Alan Field
> Assignee: Mircea Markus
>
> During the execution of a MapReduce word count job with 6 nodes, the following NPE is thrown:
> 11:19:37,870 ERROR [org.infinispan.remoting.InboundInvocationHandlerImpl] (remote-thread-2) Exception executing command
> java.lang.NullPointerException
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.reduce(MapReduceManagerImpl.java:153)
> at org.infinispan.commands.read.ReduceCommand.perform(ReduceCommand.java:88)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:122)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:68)
> at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:194)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> The full log is here - https://jenkins.mw.lab.eng.bos.redhat.com/hudson/user/afield@REDHAT.COM/m...'s%20jobs/job/jdg-radargun-mapreduce-test/181/console-edg-perf06/
> Looking at the code to see if I can figure out what happened.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 9 months
[JBoss JIRA] (ISPN-3321) NPE in MapReduceTask reduce phase
by Alan Field (JIRA)
Alan Field created ISPN-3321:
--------------------------------
Summary: NPE in MapReduceTask reduce phase
Key: ISPN-3321
URL: https://issues.jboss.org/browse/ISPN-3321
Project: Infinispan
Issue Type: Bug
Components: Distributed Execution and Map/Reduce
Affects Versions: 5.3.0.Final
Reporter: Alan Field
Assignee: Mircea Markus
During the execution of a MapReduce word count job with 6 nodes, the following NPE is thrown:
11:19:37,870 ERROR [org.infinispan.remoting.InboundInvocationHandlerImpl] (remote-thread-2) Exception executing command
java.lang.NullPointerException
at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.reduce(MapReduceManagerImpl.java:153)
at org.infinispan.commands.read.ReduceCommand.perform(ReduceCommand.java:88)
at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:122)
at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:68)
at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:194)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
The full log is here - https://jenkins.mw.lab.eng.bos.redhat.com/hudson/user/afield@REDHAT.COM/m...'s%20jobs/job/jdg-radargun-mapreduce-test/181/console-edg-perf06/
Looking at the code to see if I can figure out what happened.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 9 months
[JBoss JIRA] (ISPN-3236) Repeatable Read: Transactions are not isolated when read non-existing keys
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-3236?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-3236:
-------------------------------
Status: Resolved (was: Pull Request Sent)
Resolution: Done
> Repeatable Read: Transactions are not isolated when read non-existing keys
> --------------------------------------------------------------------------
>
> Key: ISPN-3236
> URL: https://issues.jboss.org/browse/ISPN-3236
> Project: Infinispan
> Issue Type: Bug
> Components: Transactions
> Affects Versions: 5.3.0.CR1
> Reporter: Pedro Ruivo
> Assignee: Pedro Ruivo
> Labels: isolation
> Fix For: 6.0.0.Alpha1, 6.0.0.Final
>
>
> Repeatable read transaction is not isolated when they read a non-existing key.
> tm.begin()
> cache.get(k) //returns null
> //in the meanwhile a transaction writes on k and commits
> cache.get(k) //return the new value. IMO, this is not valid for
> REPEATABLE_READ isolation level!
> //TODO add a test case for all the write operations (except put map because it does not read the old value)
> tm.begin()
> cache.get(k) //returns null (op#1)
> //in the meanwhile a transaction writes on k and commits
> write operation performed:
> * put: must return the same value as op#1
> * conditional put //if op#1 returns null the operation should be always successful (i.e. the key is updated, return true). Otherwise, the key remains unchanged (return false)
> * replace: must return the same value as op#1
> * conditional replace: replace should be successful if checked with the op#1 return value (return true). Otherwise, the key must remain unchanged (return false).
> * remote: must return the same value as op#1
> * conditional remove: the key should be removed if checked with the op#1 return value (return true). Otherwise, the key must remain unchanged (return false)
> //TODO2: check if after a remove, the following get return null and the description above is still valid.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 9 months
[JBoss JIRA] (ISPN-3319) RemoteCacheStore should not be able to implement loadAllKeys
by William Burns (JIRA)
William Burns created ISPN-3319:
-----------------------------------
Summary: RemoteCacheStore should not be able to implement loadAllKeys
Key: ISPN-3319
URL: https://issues.jboss.org/browse/ISPN-3319
Project: Infinispan
Issue Type: Enhancement
Components: Loaders and Stores
Affects Versions: 5.3.0.Final
Reporter: William Burns
Assignee: Mircea Markus
Fix For: 6.0.0.Alpha2
RemoteCacheStore can now utilize the RemoteCache.keySet method to retrieve all the keys for the loadAllKeys method. Currently it throws an exception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 9 months