[JBoss JIRA] (ISPN-2240) Per-key lock container leads to superfluous TimeoutExceptions on concurrent access to same key
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2240?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2240:
--------------------------------
Fix Version/s: 6.0.0.Final
(was: 6.0.0.Alpha1)
> Per-key lock container leads to superfluous TimeoutExceptions on concurrent access to same key
> ----------------------------------------------------------------------------------------------
>
> Key: ISPN-2240
> URL: https://issues.jboss.org/browse/ISPN-2240
> Project: Infinispan
> Issue Type: Bug
> Components: Locking and Concurrency
> Affects Versions: 5.1.6.FINAL, 5.1.x
> Reporter: Robert Stupp
> Assignee: Mircea Markus
> Fix For: 6.0.0.Final
>
> Attachments: ISPN-2240_fix_TimeoutExceptions.patch, somehow.zip
>
>
> Hi,
> I've encountered a lot of TimeoutExceptions just running a load test against an infinispan cluster.
> I tracked down the reason and found out, that the code in org.infinispan.util.concurrent.locks.containers.AbstractPerEntryLockContainer#releaseLock() causes these superfluous TimeoutExceptions.
> A small test case (which just prints out timeouts, too late timeouts and "paints" a lot of dots to the console - more dots/second on the console means better throughput ;-)
> In a short test I extended the class ReentrantPerEntryLockContainer and changed the implementation of releaseLock() as follows:
> {noformat}
> public void releaseLock(Object lockOwner, Object key) {
> ReentrantLock l = locks.get(key);
> if (l != null) {
> if (!l.isHeldByCurrentThread())
> throw new IllegalStateException("Lock for [" + key + "] not held by current thread " + Thread.currentThread());
> while (l.isHeldByCurrentThread())
> unlock(l, lockOwner);
> if (!l.hasQueuedThreads())
> locks.remove(key);
> }
> else
> throw new IllegalStateException("No lock for [" + key + ']');
> }
> {noformat}
> The main improvement is that locks are not removed from the concurrent map as long as other threads are waiting on that lock.
> If the lock is removed from the map while other threads are waiting for it, they may run into timeouts and force TimeoutExceptions to the client.
> The above methods "paints more dots per second" - means: it gives a better throughput for concurrent accesses to the same key.
> The re-implemented method should also fix some replication timeout exceptions.
> Please, please add this to 5.1.7, if possible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 6 months
[JBoss JIRA] (ISPN-2240) Per-key lock container leads to superfluous TimeoutExceptions on concurrent access to same key
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2240?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2240:
--------------------------------
Priority: Critical (was: Major)
> Per-key lock container leads to superfluous TimeoutExceptions on concurrent access to same key
> ----------------------------------------------------------------------------------------------
>
> Key: ISPN-2240
> URL: https://issues.jboss.org/browse/ISPN-2240
> Project: Infinispan
> Issue Type: Bug
> Components: Locking and Concurrency
> Affects Versions: 5.1.6.FINAL, 5.1.x
> Reporter: Robert Stupp
> Assignee: Mircea Markus
> Priority: Critical
> Fix For: 6.0.0.Final
>
> Attachments: ISPN-2240_fix_TimeoutExceptions.patch, somehow.zip
>
>
> Hi,
> I've encountered a lot of TimeoutExceptions just running a load test against an infinispan cluster.
> I tracked down the reason and found out, that the code in org.infinispan.util.concurrent.locks.containers.AbstractPerEntryLockContainer#releaseLock() causes these superfluous TimeoutExceptions.
> A small test case (which just prints out timeouts, too late timeouts and "paints" a lot of dots to the console - more dots/second on the console means better throughput ;-)
> In a short test I extended the class ReentrantPerEntryLockContainer and changed the implementation of releaseLock() as follows:
> {noformat}
> public void releaseLock(Object lockOwner, Object key) {
> ReentrantLock l = locks.get(key);
> if (l != null) {
> if (!l.isHeldByCurrentThread())
> throw new IllegalStateException("Lock for [" + key + "] not held by current thread " + Thread.currentThread());
> while (l.isHeldByCurrentThread())
> unlock(l, lockOwner);
> if (!l.hasQueuedThreads())
> locks.remove(key);
> }
> else
> throw new IllegalStateException("No lock for [" + key + ']');
> }
> {noformat}
> The main improvement is that locks are not removed from the concurrent map as long as other threads are waiting on that lock.
> If the lock is removed from the map while other threads are waiting for it, they may run into timeouts and force TimeoutExceptions to the client.
> The above methods "paints more dots per second" - means: it gives a better throughput for concurrent accesses to the same key.
> The re-implemented method should also fix some replication timeout exceptions.
> Please, please add this to 5.1.7, if possible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 6 months
[JBoss JIRA] (ISPN-3029) IllegalMonitorStateException in LockSupportCacheStore.loadAllKeys
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-3029?page=com.atlassian.jira.plugin.... ]
Work on ISPN-3029 started by Pedro Ruivo.
> IllegalMonitorStateException in LockSupportCacheStore.loadAllKeys
> -----------------------------------------------------------------
>
> Key: ISPN-3029
> URL: https://issues.jboss.org/browse/ISPN-3029
> Project: Infinispan
> Issue Type: Bug
> Components: Loaders and Stores
> Affects Versions: 5.2.5.Final
> Reporter: Dan Berindei
> Assignee: Pedro Ruivo
> Priority: Critical
> Labels: onboard
> Fix For: 6.0.0.Final
>
>
> Most LockSupportCacheStore methods that call {{acquireGlobalLock()}} ignore its return value and proceed as if the lock was acquired on all the buckets. ISPN-2378 partially fixed this by only attempting to unlock the global lock if the lock was actually acquired, but the processing that was supposed to be protected by the global lock is still executed even if the lock acquisition failed.
> In {{loadAllKeys}}, this doesn't usually cause any problems. But if the cache store contains expired entries, it will try to upgrade a bucket lock to a write lock in order to update the bucket on disk, and the upgrade will fail with a IllegalMonitorStateException:
> {noformat}
> > 20:41:36,960 ERROR [org.infinispan.statetransfer.OutboundTransferTask] (undefined) Failed to execute outbound transfer: java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread
> > at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.unmatchedUnlockException(ReentrantReadWriteLock.java:447) [rt.jar:1.7.0_09-icedtea]
> > at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReleaseShared(ReentrantReadWriteLock.java:431) [rt.jar:1.7.0_09-icedtea]
> > at java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1340) [rt.jar:1.7.0_09-icedtea]
> > at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.unlock(ReentrantReadWriteLock.java:883) [rt.jar:1.7.0_09-icedtea]
> > at org.infinispan.util.concurrent.locks.StripedLock.upgradeLock(StripedLock.java:140) [infinispan-core-5.2.4.Final-redhat-2.jar:5.2.4.Final-redhat-2]
> > at org.infinispan.loaders.LockSupportCacheStore.upgradeLock(LockSupportCacheStore.java:106) [infinispan-core-5.2.4.Final-redhat-2.jar:5.2.4.Final-redhat-2]
> > at org.infinispan.loaders.bucket.BucketBasedCacheStore.access$000(BucketBasedCacheStore.java:49) [infinispan-core-5.2.4.Final-redhat-2.jar:5.2.4.Final-redhat-2]
> > at org.infinispan.loaders.bucket.BucketBasedCacheStore$CollectionGeneratingBucketHandler.handle(BucketBasedCacheStore.java:159) [infinispan-core-5.2.4.Final-redhat-2.jar:5.2.4.Final-redhat-2]
> > at org.infinispan.loaders.file.FileCacheStore.loopOverBuckets(FileCacheStore.java:102) [infinispan-core-5.2.4.Final-redhat-2.jar:5.2.4.Final-redhat-2]
> > at org.infinispan.loaders.bucket.BucketBasedCacheStore.loadAllKeysLockSafe(BucketBasedCacheStore.java:219) [infinispan-core-5.2.4.Final-redhat-2.jar:5.2.4.Final-redhat-2]
> > at org.infinispan.loaders.LockSupportCacheStore.loadAllKeys(LockSupportCacheStore.java:179) [infinispan-core-5.2.4.Final-redhat-2.jar:5.2.4.Final-redhat-2]
> > at org.infinispan.loaders.decorators.AbstractDelegatingStore.loadAllKeys(AbstractDelegatingStore.java:140) [infinispan-core-5.2.4.Final-redhat-2.jar:5.2.4.Final-redhat-2]
> > at org.infinispan.loaders.decorators.AsyncStore.loadKeys(AsyncStore.java:184) [infinispan-core-5.2.4.Final-redhat-2.jar:5.2.4.Final-redhat-2]
> > at org.infinispan.loaders.decorators.AsyncStore.loadAllKeys(AsyncStore.java:205) [infinispan-core-5.2.4.Final-redhat-2.jar:5.2.4.Final-redhat-2]
> > at org.infinispan.statetransfer.OutboundTransferTask.run(OutboundTransferTask.java:163) [infinispan-core-5.2.4.Final-redhat-2.jar:5.2.4.Final-redhat-2]
> > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_09-icedtea]
> > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_09-icedtea]
> > at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_09-icedtea]
> > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_09-icedtea]
> > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_09-icedtea]
> > at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_09-icedtea]
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_09-icedtea]
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_09-icedtea]
> > at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea]
> > at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.0.0.GA-redhat-2.jar:2.0.0.GA-redhat-2]
> {noformat}
> A simple solution would be to throw an exception any time the global lock acquisition failed, but the current global lock acquisition algorithm might need to change because it seems very deadlock-prone at the moment.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 6 months
[JBoss JIRA] (ISPN-2811) cassandraStore xml configuration gives parser error on attributes "username" and "password"
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2811?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2811:
--------------------------------
Assignee: Tristan Tarrant (was: Mircea Markus)
> cassandraStore xml configuration gives parser error on attributes "username" and "password"
> -------------------------------------------------------------------------------------------
>
> Key: ISPN-2811
> URL: https://issues.jboss.org/browse/ISPN-2811
> Project: Infinispan
> Issue Type: Bug
> Components: Loaders and Stores
> Affects Versions: 5.2.0.Final
> Reporter: Giovanni Mels
> Assignee: Tristan Tarrant
> Fix For: 6.0.0.Final
>
>
> This is because attributes "username" and "password" are in uppercase in [org.infinispan.loaders.cassandra.configuration.Attribute|https://github.c...], but in lowercase in the [schema|http://docs.jboss.org/infinispan/schemas/infinispan-cachestore-cas...].
> {quote}
> org.infinispan.config.ConfigurationException: javax.xml.stream.XMLStreamException: ParseError at [row,col]:[37,142]
> Message: Unexpected attribute 'password' encountered
> at org.infinispan.configuration.parsing.ParseUtils.unexpectedAttribute(ParseUtils.java:76)
> at org.infinispan.configuration.parsing.Parser52.parseCommonStoreAttributes(Parser52.java:696)
> at org.infinispan.loaders.cassandra.configuration.CassandraCacheStoreConfigurationParser52.parseCassandraStoreAttributes(CassandraCacheStoreConfigurationParser52.java:180)
> at org.infinispan.loaders.cassandra.configuration.CassandraCacheStoreConfigurationParser52.parseCassandraStore(CassandraCacheStoreConfigurationParser52.java:77)
> at org.infinispan.loaders.cassandra.configuration.CassandraCacheStoreConfigurationParser52.readElement(CassandraCacheStoreConfigurationParser52.java:65)
> at org.infinispan.loaders.cassandra.configuration.CassandraCacheStoreConfigurationParser52.readElement(CassandraCacheStoreConfigurationParser52.java:43)
> at org.jboss.staxmapper.XMLMapperImpl.processNested(XMLMapperImpl.java:110)
> at org.jboss.staxmapper.XMLExtendedStreamReaderImpl.handleAny(XMLExtendedStreamReaderImpl.java:69)
> at org.infinispan.configuration.parsing.Parser52.parseLoaders(Parser52.java:588)
> at org.infinispan.configuration.parsing.Parser52.parseCache(Parser52.java:180)
> at org.infinispan.configuration.parsing.Parser52.parseDefaultCache(Parser52.java:145)
> at org.infinispan.configuration.parsing.Parser52.readElement(Parser52.java:98)
> at org.infinispan.configuration.parsing.Parser52.readElement(Parser52.java:75)
> at org.jboss.staxmapper.XMLMapperImpl.processNested(XMLMapperImpl.java:110)
> at org.jboss.staxmapper.XMLMapperImpl.parseDocument(XMLMapperImpl.java:69)
> at org.infinispan.configuration.parsing.ParserRegistry.parse(ParserRegistry.java:77)
> ... 28 more
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 6 months
[JBoss JIRA] (ISPN-2876) AbstractComponentRegistry.getComponent can return an incomplete component
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2876?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2876:
--------------------------------
Assignee: Dan Berindei (was: Mircea Markus)
> AbstractComponentRegistry.getComponent can return an incomplete component
> -------------------------------------------------------------------------
>
> Key: ISPN-2876
> URL: https://issues.jboss.org/browse/ISPN-2876
> Project: Infinispan
> Issue Type: Bug
> Components: Locking and Concurrency
> Affects Versions: 5.2.2.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Fix For: 6.0.0.Final
>
>
> AbstractComponentRegistry.getComponent doesn't have any locking, so it's possible for a thread to retrieve a component while another thread is injecting that component's dependencies.
> Normally this isn't a problem, because the global registry is locked during cache manager startup, and the cache registry is locked during cache startup. However, some global components are only created "on-demand", because a cache component depends on them, and this can happen if someone uses getComponent instead of getOrCreateComponent.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 6 months
[JBoss JIRA] (ISPN-2838) Remote configuration in the arquillian.xml file should be commented - See JDF-215 for details
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2838?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2838:
--------------------------------
Fix Version/s: 6.0.0.Final
(was: 6.0.0.Alpha1)
> Remote configuration in the arquillian.xml file should be commented - See JDF-215 for details
> ---------------------------------------------------------------------------------------------
>
> Key: ISPN-2838
> URL: https://issues.jboss.org/browse/ISPN-2838
> Project: Infinispan
> Issue Type: Bug
> Components: Demos and Tutorials
> Reporter: Sande Gilda
> Assignee: Mircea Markus
> Fix For: 6.0.0.Final
>
>
> The kitchensink-deltaspike arquillian.xml file contains a configuration for jbossHome
> This should be commented out. All the remote configuration in the arquillian files should look like the following:
> <!-- Example configuration for a remote JBoss Enterprise Application Platform 6 or AS 7 instance -->
> <container qualifier="jboss" default="true">
> <!-- By default, arquillian will use the JBOSS_HOME environment variable. Alternatively, the configuration below can be uncommented. -->
> <!--<configuration> -->
> <!--<property name="jbossHome">/path/to/jboss/as</property> -->
> <!--</configuration> -->
> </container>
> You can copy and paste from the bean-validation arquillian.xml file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 6 months