[JBoss JIRA] (ISPN-1838) State transfer takes more than 10 minutes with only 10K entries.
by Michal Linhard (JIRA)
Michal Linhard created ISPN-1838:
------------------------------------
Summary: State transfer takes more than 10 minutes with only 10K entries.
Key: ISPN-1838
URL: https://issues.jboss.org/browse/ISPN-1838
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.1.1.CR1
Reporter: Michal Linhard
Assignee: Manik Surtani
This could be categorized as a …
[View More]performance problem.
It happened in resilience test run: http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/j...
originally to verify ISPN-1826
It was run with infinispan special build from Galder's branch (https://github.com/galderz/infinispan/tree/t_1826_5)
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-QE/job/edg-60-build-...
test starts 4 nodes, kills node2, starts node2 and sees what happens
trace logging on server side was on. there were two runs
200 clients, 10K entries
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/j...
20 clients, 1K entries
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/j...
in run 24 everyting looks nice:
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/j...
in run 23 the state transfer takes forever (more than 10 min)
these important views are installed on coordinator (node03):
{code}
2012-02-02 05:11:00,560 TRACE [BaseStateTransferManagerImpl] (transport-thread-1) Received new cache view: testCache CacheView{viewId=6, members=[edg-perf04-45788, edg-perf03-36944, edg-perf02-51026, edg-perf01-47003]}
2012-02-02 05:15:13,591 TRACE [BaseStateTransferManagerImpl] (transport-thread-9) Received new cache view: testCache CacheView{viewId=7, members=[edg-perf04-45788, edg-perf03-36944, edg-perf01-47003]}
2012-02-02 05:18:17,219 TRACE [BaseStateTransferManagerImpl] (transport-thread-1) Received new cache view: testCache CacheView{viewId=8, members=[edg-perf04-45788, edg-perf03-36944, edg-perf01-47003, edg-perf02-21799]}
2012-02-02 05:28:17,511 TRACE [BaseStateTransferManagerImpl] (transport-thread-22) Received new cache view: testCache CacheView{viewId=10, members=[edg-perf04-45788, edg-perf03-36944, edg-perf01-47003, edg-perf02-21799]}
{code}
viewId=8 is the one that takes 10 min to prepare and after that the prepare fails:
{code}
2012-02-02 05:28:17,219 ERROR [CacheViewsManagerImpl] (CacheViewInstaller-9,edg-perf03-36944) ISPN000172: Failed to prepare view CacheView{viewId=8, members=[edg-perf04-45788, edg-perf03-36944, edg-perf01-47003, edg-perf02-21799]} for cache testCache, ro..
java.util.concurrent.TimeoutException
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:228)
at java.util.concurrent.FutureTask.get(FutureTask.java:91)
at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterPrepareView(CacheViewsManagerImpl.java:319)
at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterInstallView(CacheViewsManagerImpl.java:250)
at org.infinispan.cacheviews.CacheViewsManagerImpl$ViewInstallationTask.call(CacheViewsManagerImpl.java:877)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
{code}
viewId=10 is a retry and that succeeds quite quickly but the test is already ending about that time.
It might be worth looking at the tracelogs since they're already there...
10K entries and 200 clients isn't such a big load ...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[View Less]
12 years, 9 months
[JBoss JIRA] (ISPN-2103) Concurrent access after removal of an AtomicMap should NOT result in an IllegalStateException when accessed by other threads
by Adrian Nistor (JIRA)
Adrian Nistor created ISPN-2103:
-----------------------------------
Summary: Concurrent access after removal of an AtomicMap should NOT result in an IllegalStateException when accessed by other threads
Key: ISPN-2103
URL: https://issues.jboss.org/browse/ISPN-2103
Project: Infinispan
Issue Type: Bug
Components: Core API
Affects Versions: 5.1.5.FINAL
Reporter: Adrian Nistor
Assignee: …
[View More]Adrian Nistor
ISPN-1121 introduces an IllegalStateException that is being thrown from AtomicMap methods once the map handle has become stale (ie. removed from cache). In case of concurrent access, the exception is thrown to all threads not just to the thread that performed the removal. This was fine-ish in older versions of Infinispan before introduction of optimistic and pessimistic locking but it should be reconsidered now because:
1. It interferes/overlaps with transaction isolation. We should rely on the selected locking scheme (OL/PL) to detect conflicts between transactions and report the problem accordingly. Especially if the stale map is used just for reading - this should be allowed to work rather than stop it.
2. This exception is pretty disruptive and awkward to handle. All methods of an AtomicMap can result in this exception and the current thread has to be prepared for handling it if other threads remove the map. Not much transaction isolation.
3. Since the TreeCache is backed by AtomicMap nearly all Tree API can throw this.
The proposed fix consists of:
1. removing AtomicHashMap.removed flag and AtomicHashMap.markRemoved() method.
2. revising AtomicHashMapProxy.assertValid() method to check only if the map is null (ie. removed) but no longer use the removed flag.
3. revising ReadCommittedEntry.commit() method to no longer call markRemoved() method.
The consequences of these changes are:
1. Any further access to a stale map results in IllegalStateException ONLY in the thread that performed the removal. This thread 'knows' the map is stale so it is fine to punish it. Other threads remain unaffected until lock acquisition or commit is performed (depending on locking model).
2. Other threads can continue to use the previously obtained map handle for reads without danger of getting an exception.
3. If a write operation is done on the map, the results depend on the locking model:
3.1 optimistic locking + write skew check: a WriteSkewException will stop the commit during prepare
3.2 optimistic locking, no write skew check: the write is committed and the work of the transaction that removed the map is overwritten. The map is effectively revived.
3.3 pessimistic locking: same as 3.2
Please note 3.2 and 3.3 work the same as for normal values (not atomic maps).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[View Less]
12 years, 9 months
[JBoss JIRA] (ISPN-2091) CommandIdUniquenessTest fails when running the test suite with private maven repo that contains "infinispan" in its name
by Martin Gencur (JIRA)
Martin Gencur created ISPN-2091:
-----------------------------------
Summary: CommandIdUniquenessTest fails when running the test suite with private maven repo that contains "infinispan" in its name
Key: ISPN-2091
URL: https://issues.jboss.org/browse/ISPN-2091
Project: Infinispan
Issue Type: Bug
Components: Test Suite
Affects Versions: 5.1.5.FINAL
Reporter: Martin Gencur
Assignee: Mircea …
[View More]Markus
Priority: Minor
I recently tried to run the test suite with private maven repo and it failed with
{code}
WARN [ClassFinder] (testng-CommandIdUniquenessTest) From jar path javax/transaction/RollbackException.class could not load class null
java.lang.StringIndexOutOfBoundsException: String index out of range: -1
at java.lang.String.substring(String.java:1931)
at org.infinispan.util.ClassFinder.toClassName(ClassFinder.java:194)
at org.infinispan.util.ClassFinder.findClassesOnPath(ClassFinder.java:155)
at org.infinispan.util.ClassFinder.infinispanClasses(ClassFinder.java:112)
at org.infinispan.util.ClassFinder.infinispanClasses(ClassFinder.java:94)
at org.infinispan.util.ClassFinder.isAssignableFrom(ClassFinder.java:90)
at org.infinispan.commands.CommandIdUniquenessTest.testCommandIdUniqueness(CommandIdUniquenessTest.java:38)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:74)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:673)
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:846)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1170)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
at org.testng.TestRunner.runWorkers(TestRunner.java:1147)
at org.testng.TestRunner.privateRun(TestRunner.java:749)
at org.testng.TestRunner.run(TestRunner.java:600)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:317)
at org.testng.SuiteRunner.access$000(SuiteRunner.java:34)
at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:351)
at org.testng.internal.thread.ThreadUtil$CountDownLatchedRunnable.run(ThreadUtil.java:147)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
{code}
After few hours I figured out the cause. I had the private repo located in https://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-FUNC/job/ed... and the ClassFinder class was trying to parse all jars that contained "infinispan" in its name. The method toClassName then tried to find last index of "org" which was not there for some libraries in the private repo.
As a results, I cannot have a jenkins job (or my private repo) that contains "infinispan" in its name. I think this parsing could be done better.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[View Less]
12 years, 10 months