[JBoss JIRA] (ISPN-2802) Cache recovery fails due to missing responses
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2802?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2802:
--------------------------------
Fix Version/s: 5.3.0.Beta2
(was: 5.3.0.Beta1)
> Cache recovery fails due to missing responses
> ---------------------------------------------
>
> Key: ISPN-2802
> URL: https://issues.jboss.org/browse/ISPN-2802
> Project: Infinispan
> Issue Type: Bug
> Components: State transfer
> Affects Versions: 5.2.0.CR3
> Reporter: Radim Vansa
> Assignee: Dan Berindei
> Fix For: 5.3.0.Beta2, 5.3.0.Final
>
>
> When the cache recovery is started, the new coordinator sends CacheTopologyControlCommand.GET_STATUS to all nodes and waits for responses. However, I have a reproducible test-case where it always times out waiting for the responses.
> Here are the logs (TRACE is not doable here, but I added some byteman traces - see topology.btm in the archive): http://dl.dropbox.com/u/103079234/recovery.zip
> The problematic spot is on node3 at 05:37:57 receiving cluster view 34.
> All nodes (except the one which is killed, in this case node1) respond quickly to the GET_STATUS command (see BYTEMAN Receiving - Received pairs, these are bound to command execution in CommandAwareRpcDispatcher), but some responses are not received on node3 (look for Receiving rsp bound to GroupRequest).
> JGroups tracing could be useful here but it is not available (intensive logging often blocks on internal log4j locks and the node becomes unresponsive).
> As mentioned above, the case is reproducible, therefore if you can suggest any particular BYTEMAN hook, I can try it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 8 months
[JBoss JIRA] (ISPN-2771) WriteSkewTest.testWriteSkewWithOnlyPut fails randomly
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2771?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2771:
--------------------------------
Fix Version/s: 5.3.0.Beta2
(was: 5.3.0.Beta1)
> WriteSkewTest.testWriteSkewWithOnlyPut fails randomly
> -----------------------------------------------------
>
> Key: ISPN-2771
> URL: https://issues.jboss.org/browse/ISPN-2771
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite, Transactions
> Affects Versions: 5.2.0.CR3
> Reporter: Adrian Nistor
> Assignee: Adrian Nistor
> Fix For: 5.3.0.Beta2
>
> Attachments: WriteSkewTest.zip
>
>
> {noformat}
> org.infinispan.api.mvcc.repeatable_read.WriteSkewTest:testWriteSkewWithOnlyPut
> org.infinispan.transaction.WriteSkewException: Detected write skew.
> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at org.infinispan.api.mvcc.repeatable_read.WriteSkewTest.testWriteSkewWithOnlyPut(WriteSkewTest.java:315)
> 22 lines not shown
> Caused by Detected write skew.
> org.infinispan.container.entries.RepeatableReadEntry.performLocalWriteSkewCheck(RepeatableReadEntry.java:68)
> at org.infinispan.container.entries.RepeatableReadEntry.copyForUpdate(RepeatableReadEntry.java:52)
> at org.infinispan.container.EntryFactoryImpl.wrapEntryForPut(EntryFactoryImpl.java:170)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:164)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPutKeyValueCommand(OptimisticLockingInterceptor.java:142)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:251)
> at org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:191)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1162)
> at org.infinispan.CacheImpl.putInternal(CacheImpl.java:760)
> at org.infinispan.CacheImpl.put(CacheImpl.java:754)
> at org.infinispan.CacheImpl.put(CacheImpl.java:748)
> at org.infinispan.CacheSupport.put(CacheSupport.java:53)
> at org.infinispan.api.mvcc.repeatable_read.WriteSkewTest$EntryWriter.call(WriteSkewTest.java:433)
> at org.infinispan.api.mvcc.repeatable_read.WriteSkewTest$EntryWriter.call(WriteSkewTest.java:418)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 1 lines not shown
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 8 months
[JBoss JIRA] (ISPN-2835) Issues w/ M/R test cases if cache are not explicitly started on all nodes
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2835?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2835:
--------------------------------
Fix Version/s: 5.3.0.Beta2
(was: 5.3.0.Beta1)
> Issues w/ M/R test cases if cache are not explicitly started on all nodes
> -------------------------------------------------------------------------
>
> Key: ISPN-2835
> URL: https://issues.jboss.org/browse/ISPN-2835
> Project: Infinispan
> Issue Type: Feature Request
> Components: Core API, Distributed Execution and Map/Reduce
> Reporter: Ray Tsang
> Assignee: Galder Zamarreño
> Fix For: 5.3.0.Beta2, 5.3.0.Final
>
> Attachments: mr-test-src.zip
>
>
> I ran into some issues while using M/R. The gist of the issue I was seeing is that:
> Start a cluster of Embedded Caches, like 4 nodes
> Put in 100 elements
> Run a simple M/R job to count the number of keys
> If I run the M/R job using the node I'm inserting elements into as coordinator - the result is 100
> But if I run the M/R job using a different node as coordinator, the result is less than 100
> More interestingly, I can pause for 5 seconds and run the M/R jobs again, the results are always less than 100
> This behavior doens't occur if I explicitly run cacheManager.getCache() for each of the nodes...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 8 months
[JBoss JIRA] (ISPN-2950) In distributed mode cache store data should be read through the main data owner (vs directly from the store)
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2950?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2950:
--------------------------------
Fix Version/s: 5.3.0.Beta2
(was: 5.3.0.Beta1)
> In distributed mode cache store data should be read through the main data owner (vs directly from the store)
> ------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-2950
> URL: https://issues.jboss.org/browse/ISPN-2950
> Project: Infinispan
> Issue Type: Feature Request
> Components: Loaders and Stores
> Reporter: Sanne Grinovero
> Assignee: Mircea Markus
> Priority: Critical
> Fix For: 5.3.0.Beta2, 5.3.0.Final
>
>
> Dist cache with a cache store (shared or not), k owned by \{N1, N2\}. k is read on N3. What currently happens at this stage, if k is not present in N3's memory (likely unless L1 is configured), the N3's cache store is queried and data is loaded from there. This has several drawbacks:
> - the data might already be in the memory of the owner node (N1,N2) so reading it from the disk is highly inefficient. Especially for hot data: data requested from various nodes at the same time (see also mailing list discussion around lucene query performance depending on this)
> - if this is a local cache store, it might contain stale data which would be returned to the user
> - for async configured cache store this would result in dirty reads, given that a change might be in the async store's memory but not in the store at the moment when it is in read by N3. (Note that using async stores still leaves place to inconsistencies when a node leaves, e.g. because of node crashing before managing to flush the async store.)
> This JIRA is about changing the distribution mode: when asked for a specific key, a node would only touch a cache store if it is an owner of that key, otherwise would first go to the main owner of the key to read the value from there. The ClusterCacheLoader should be deprecated as well.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 8 months
[JBoss JIRA] (ISPN-2951) Infinispan try to register again the whole manager on jmx when deleting and recreating a cache
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2951?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2951:
--------------------------------
Fix Version/s: 5.3.0.Beta2
(was: 5.3.0.Beta1)
> Infinispan try to register again the whole manager on jmx when deleting and recreating a cache
> ----------------------------------------------------------------------------------------------
>
> Key: ISPN-2951
> URL: https://issues.jboss.org/browse/ISPN-2951
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 5.2.5.Final
> Environment: JDK Oracle 1.6.0_35
> Reporter: Thomas Mortagne
> Assignee: Mircea Markus
> Fix For: 5.3.0.Beta2, 5.3.0.Final
>
>
> When I do something like the following:
> {code:java}
> cacheManager.getCache("test");
> cacheManager.removeCache("test");
> cacheManager.getCache("test");
> {code}
> it fails with the following exception (this exception is not exactly with the code above but it's what is happening):
> {noformat}
> org.infinispan.CacheException: Unable to invoke method public void org.infinispan.jmx.CacheJmxRegistration.start() on object of type CacheJmxRegistration
> at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:205)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:886)
> at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:657)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:646)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:549)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:217)
> at org.infinispan.CacheImpl.start(CacheImpl.java:582)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:686)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:649)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:545)
> at org.xwiki.cache.infinispan.internal.InfinispanCache.<init>(InfinispanCache.java:71)
> at org.xwiki.cache.infinispan.internal.InfinispanCacheFactory.newCache(InfinispanCacheFactory.java:170)
> at org.xwiki.cache.infinispan.InfinispanCacheTest.testRecreateCache(InfinispanCacheTest.java:62)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.jmock.integration.junit4.JUnitRuleMockery$1.evaluate(JUnitRuleMockery.java:49)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
> Caused by: org.infinispan.jmx.JmxDomainConflictException: Domain already registered org.infinispan when trying to register: type=Cache,name="test(local)",manager="DefaultCacheManager"
> at org.infinispan.jmx.JmxUtil.buildJmxDomain(JmxUtil.java:75)
> at org.infinispan.jmx.CacheJmxRegistration.updateDomain(CacheJmxRegistration.java:153)
> at org.infinispan.jmx.CacheJmxRegistration.buildRegistrar(CacheJmxRegistration.java:145)
> at org.infinispan.jmx.AbstractJmxRegistration.registerMBeans(AbstractJmxRegistration.java:59)
> at org.infinispan.jmx.CacheJmxRegistration.start(CacheJmxRegistration.java:83)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203)
> ... 38 more
> {noformat}
> Note: this was working well with 5.1.8.Final
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 8 months
[JBoss JIRA] (ISPN-2906) SyncCacheListenerTest.testSyncTxRepl failing randomly
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2906?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2906:
--------------------------------
Fix Version/s: 5.3.0.Beta2
(was: 5.3.0.Beta1)
> SyncCacheListenerTest.testSyncTxRepl failing randomly
> -----------------------------------------------------
>
> Key: ISPN-2906
> URL: https://issues.jboss.org/browse/ISPN-2906
> Project: Infinispan
> Issue Type: Bug
> Components: Listeners, Transactions
> Reporter: Galder Zamarreño
> Assignee: Mircea Markus
> Labels: testsuite_stability
> Fix For: 5.3.0.Beta2
>
> Attachments: SyncCacheListenerTest_age_is_null.log, testSyncTxRepl-0.log
>
>
> {code}testSyncTxRepl(org.infinispan.replication.SyncCacheListenerTest) Time elapsed: 0.04 sec <<< FAILURE!
> java.lang.AssertionError: "age" obtained from cache2 must be non-null
> at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
> at org.testng.AssertJUnit.assertTrue(AssertJUnit.java:24)
> at org.testng.AssertJUnit.assertNotNull(AssertJUnit.java:267)
> at org.infinispan.replication.SyncCacheListenerTest.testSyncTxRepl(SyncCacheListenerTest.java:99)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722){code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 8 months