6:45 p.m.
LRU data container endlesly looping or containing heavily
---------------------------------------------------------
Key: ISPN-277
URL: https://jira.jboss.org/jira/browse/ISPN-277
Project: Infinispan
Issue Type: Bug
Components: Eviction
Affects Versions: 4.0.0.CR2
Reporter: Galder Zamarreno
Assignee: Galder Zamarreno
Priority: Critical
Attachments: td2.txt
Something around LRU container is not working fine. The attached log from an concurrency
test in the 2nd level cache shows that in 3 thread dumps taken over 30 seconds appart,
UserRunnerThread-5 is stuck in:
"UserRunnerThread-5" prio=10 tid=0x6f65bc00 nid=0xdea runnable [0x05efb000]
java.lang.Thread.State: RUNNABLE
at
org.infinispan.container.FIFODataContainer$LinkedEntry.casNext(FIFODataContainer.java:180)
at org.infinispan.container.FIFODataContainer.correctPrev(FIFODataContainer.java:329)
at org.infinispan.container.FIFODataContainer.linkAtEnd(FIFODataContainer.java:252)
at org.infinispan.container.LRUDataContainer.put(LRUDataContainer.java:70)
at
org.infinispan.container.entries.ReadCommittedEntry.commit(ReadCommittedEntry.java:161)
at
org.infinispan.interceptors.LockingInterceptor.commitEntry(LockingInterceptor.java:298)
at
org.infinispan.interceptors.LockingInterceptor.cleanupLocks(LockingInterceptor.java:281)
at
org.infinispan.interceptors.LockingInterceptor.doAfterCall(LockingInterceptor.java:243)
at
org.infinispan.interceptors.LockingInterceptor.visitPutKeyValueCommand(LockingInterceptor.java:200)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.MarshalledValueInterceptor.visitPutKeyValueCommand(MarshalledValueInterceptor.java:93)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:185)
at
org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:132)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:48)
at
org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:34)
at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:269)
at org.infinispan.CacheDelegate.putIfAbsent(CacheDelegate.java:422)
at org.infinispan.CacheDelegate.putIfAbsent(CacheDelegate.java:153)
at org.infinispan.CacheDelegate.putForExternalRead(CacheDelegate.java:243)
at
org.hibernate.cache.infinispan.util.CacheAdapterImpl.putForExternalRead(CacheAdapterImpl.java:115)
at
org.hibernate.cache.infinispan.access.TransactionalAccessDelegate.putFromLoad(TransactionalAccessDelegate.java:91)
at
org.hibernate.cache.infinispan.collection.TransactionalAccess.putFromLoad(TransactionalAccess.java:44)
at
org.hibernate.engine.loading.CollectionLoadContext.addCollectionToCache(CollectionLoadContext.java:333)
at
org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollection(CollectionLoadContext.java:279)
at
org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollections(CollectionLoadContext.java:245)
at
org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollections(CollectionLoadContext.java:218)
at org.hibernate.loader.Loader.endCollectionLoad(Loader.java:901)
at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:886)
at org.hibernate.loader.Loader.doQuery(Loader.java:750)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:257)
at org.hibernate.loader.Loader.loadCollection(Loader.java:2019)
at org.hibernate.loader.collection.CollectionLoader.initialize(CollectionLoader.java:62)
at
org.hibernate.persister.collection.AbstractCollectionPersister.initialize(AbstractCollectionPersister.java:628)
at
org.hibernate.event.def.DefaultInitializeCollectionEventListener.onInitializeCollection(DefaultInitializeCollectionEventListener.java:83)
at org.hibernate.impl.SessionImpl.initializeCollection(SessionImpl.java:1817)
at
org.hibernate.collection.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:366)
at
org.hibernate.collection.AbstractPersistentCollection.read(AbstractPersistentCollection.java:108)
at
org.hibernate.collection.AbstractPersistentCollection.readSize(AbstractPersistentCollection.java:131)
at org.hibernate.collection.PersistentSet.isEmpty(PersistentSet.java:169)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest.getFirstContact(ConcurrentWriteTest.java:405)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest.access$0(ConcurrentWriteTest.java:397)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest$UserRunner.contactExists(ConcurrentWriteTest.java:519)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest$UserRunner.call(ConcurrentWriteTest.java:540)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest$UserRunner.call(ConcurrentWriteTest.java:1)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
This concurrency test does not get to finish in over a minute. However, once LRU is
switched to NONE or FIFO, the test runs in 5 seconds.
So, something looks fishy with LRU.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:45 p.m.
New subject: [JBoss JIRA] Updated: (ISPN-277) LRU data container endlesly looping or containing heavily
[
https://jira.jboss.org/jira/browse/ISPN-277?page=com.atlassian.jira.plugi...
]
Galder Zamarreno updated ISPN-277:
----------------------------------
Attachment: td2.txt
LRU data container endlesly looping or containing heavily
---------------------------------------------------------
Key: ISPN-277
URL: https://jira.jboss.org/jira/browse/ISPN-277
Project: Infinispan
Issue Type: Bug
Components: Eviction
Affects Versions: 4.0.0.CR2
Reporter: Galder Zamarreno
Assignee: Galder Zamarreno
Priority: Critical
Attachments: td2.txt
Something around LRU container is not working fine. The attached log from an concurrency
test in the 2nd level cache shows that in 3 thread dumps taken over 30 seconds appart,
UserRunnerThread-5 is stuck in:
"UserRunnerThread-5" prio=10 tid=0x6f65bc00 nid=0xdea runnable [0x05efb000]
java.lang.Thread.State: RUNNABLE
at
org.infinispan.container.FIFODataContainer$LinkedEntry.casNext(FIFODataContainer.java:180)
at org.infinispan.container.FIFODataContainer.correctPrev(FIFODataContainer.java:329)
at org.infinispan.container.FIFODataContainer.linkAtEnd(FIFODataContainer.java:252)
at org.infinispan.container.LRUDataContainer.put(LRUDataContainer.java:70)
at
org.infinispan.container.entries.ReadCommittedEntry.commit(ReadCommittedEntry.java:161)
at
org.infinispan.interceptors.LockingInterceptor.commitEntry(LockingInterceptor.java:298)
at
org.infinispan.interceptors.LockingInterceptor.cleanupLocks(LockingInterceptor.java:281)
at
org.infinispan.interceptors.LockingInterceptor.doAfterCall(LockingInterceptor.java:243)
at
org.infinispan.interceptors.LockingInterceptor.visitPutKeyValueCommand(LockingInterceptor.java:200)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.MarshalledValueInterceptor.visitPutKeyValueCommand(MarshalledValueInterceptor.java:93)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:185)
at
org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:132)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:48)
at
org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:34)
at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:269)
at org.infinispan.CacheDelegate.putIfAbsent(CacheDelegate.java:422)
at org.infinispan.CacheDelegate.putIfAbsent(CacheDelegate.java:153)
at org.infinispan.CacheDelegate.putForExternalRead(CacheDelegate.java:243)
at
org.hibernate.cache.infinispan.util.CacheAdapterImpl.putForExternalRead(CacheAdapterImpl.java:115)
at
org.hibernate.cache.infinispan.access.TransactionalAccessDelegate.putFromLoad(TransactionalAccessDelegate.java:91)
at
org.hibernate.cache.infinispan.collection.TransactionalAccess.putFromLoad(TransactionalAccess.java:44)
at
org.hibernate.engine.loading.CollectionLoadContext.addCollectionToCache(CollectionLoadContext.java:333)
at
org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollection(CollectionLoadContext.java:279)
at
org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollections(CollectionLoadContext.java:245)
at
org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollections(CollectionLoadContext.java:218)
at org.hibernate.loader.Loader.endCollectionLoad(Loader.java:901)
at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:886)
at org.hibernate.loader.Loader.doQuery(Loader.java:750)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:257)
at org.hibernate.loader.Loader.loadCollection(Loader.java:2019)
at
org.hibernate.loader.collection.CollectionLoader.initialize(CollectionLoader.java:62)
at
org.hibernate.persister.collection.AbstractCollectionPersister.initialize(AbstractCollectionPersister.java:628)
at
org.hibernate.event.def.DefaultInitializeCollectionEventListener.onInitializeCollection(DefaultInitializeCollectionEventListener.java:83)
at org.hibernate.impl.SessionImpl.initializeCollection(SessionImpl.java:1817)
at
org.hibernate.collection.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:366)
at
org.hibernate.collection.AbstractPersistentCollection.read(AbstractPersistentCollection.java:108)
at
org.hibernate.collection.AbstractPersistentCollection.readSize(AbstractPersistentCollection.java:131)
at org.hibernate.collection.PersistentSet.isEmpty(PersistentSet.java:169)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest.getFirstContact(ConcurrentWriteTest.java:405)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest.access$0(ConcurrentWriteTest.java:397)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest$UserRunner.contactExists(ConcurrentWriteTest.java:519)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest$UserRunner.call(ConcurrentWriteTest.java:540)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest$UserRunner.call(ConcurrentWriteTest.java:1)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
This concurrency test does not get to finish in over a minute. However, once LRU is
switched to NONE or FIFO, the test runs in 5 seconds.
So, something looks fishy with LRU.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:35 p.m.
New subject: [JBoss JIRA] Updated: (ISPN-277) LRU data container endlesly looping or containing heavily
[
https://jira.jboss.org/jira/browse/ISPN-277?page=com.atlassian.jira.plugi...
]
Galder Zamarreno updated ISPN-277:
----------------------------------
Fix Version/s: 4.0.0.GA
LRU data container endlesly looping or containing heavily
---------------------------------------------------------
Key: ISPN-277
URL: https://jira.jboss.org/jira/browse/ISPN-277
Project: Infinispan
Issue Type: Bug
Components: Eviction
Affects Versions: 4.0.0.CR2
Reporter: Galder Zamarreno
Assignee: Galder Zamarreno
Priority: Critical
Fix For: 4.0.0.GA
Attachments: td2.txt
Something around LRU container is not working fine. The attached log from an concurrency
test in the 2nd level cache shows that in 3 thread dumps taken over 30 seconds appart,
UserRunnerThread-5 is stuck in:
"UserRunnerThread-5" prio=10 tid=0x6f65bc00 nid=0xdea runnable [0x05efb000]
java.lang.Thread.State: RUNNABLE
at
org.infinispan.container.FIFODataContainer$LinkedEntry.casNext(FIFODataContainer.java:180)
at org.infinispan.container.FIFODataContainer.correctPrev(FIFODataContainer.java:329)
at org.infinispan.container.FIFODataContainer.linkAtEnd(FIFODataContainer.java:252)
at org.infinispan.container.LRUDataContainer.put(LRUDataContainer.java:70)
at
org.infinispan.container.entries.ReadCommittedEntry.commit(ReadCommittedEntry.java:161)
at
org.infinispan.interceptors.LockingInterceptor.commitEntry(LockingInterceptor.java:298)
at
org.infinispan.interceptors.LockingInterceptor.cleanupLocks(LockingInterceptor.java:281)
at
org.infinispan.interceptors.LockingInterceptor.doAfterCall(LockingInterceptor.java:243)
at
org.infinispan.interceptors.LockingInterceptor.visitPutKeyValueCommand(LockingInterceptor.java:200)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.MarshalledValueInterceptor.visitPutKeyValueCommand(MarshalledValueInterceptor.java:93)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:185)
at
org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:132)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:48)
at
org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:34)
at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:269)
at org.infinispan.CacheDelegate.putIfAbsent(CacheDelegate.java:422)
at org.infinispan.CacheDelegate.putIfAbsent(CacheDelegate.java:153)
at org.infinispan.CacheDelegate.putForExternalRead(CacheDelegate.java:243)
at
org.hibernate.cache.infinispan.util.CacheAdapterImpl.putForExternalRead(CacheAdapterImpl.java:115)
at
org.hibernate.cache.infinispan.access.TransactionalAccessDelegate.putFromLoad(TransactionalAccessDelegate.java:91)
at
org.hibernate.cache.infinispan.collection.TransactionalAccess.putFromLoad(TransactionalAccess.java:44)
at
org.hibernate.engine.loading.CollectionLoadContext.addCollectionToCache(CollectionLoadContext.java:333)
at
org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollection(CollectionLoadContext.java:279)
at
org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollections(CollectionLoadContext.java:245)
at
org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollections(CollectionLoadContext.java:218)
at org.hibernate.loader.Loader.endCollectionLoad(Loader.java:901)
at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:886)
at org.hibernate.loader.Loader.doQuery(Loader.java:750)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:257)
at org.hibernate.loader.Loader.loadCollection(Loader.java:2019)
at
org.hibernate.loader.collection.CollectionLoader.initialize(CollectionLoader.java:62)
at
org.hibernate.persister.collection.AbstractCollectionPersister.initialize(AbstractCollectionPersister.java:628)
at
org.hibernate.event.def.DefaultInitializeCollectionEventListener.onInitializeCollection(DefaultInitializeCollectionEventListener.java:83)
at org.hibernate.impl.SessionImpl.initializeCollection(SessionImpl.java:1817)
at
org.hibernate.collection.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:366)
at
org.hibernate.collection.AbstractPersistentCollection.read(AbstractPersistentCollection.java:108)
at
org.hibernate.collection.AbstractPersistentCollection.readSize(AbstractPersistentCollection.java:131)
at org.hibernate.collection.PersistentSet.isEmpty(PersistentSet.java:169)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest.getFirstContact(ConcurrentWriteTest.java:405)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest.access$0(ConcurrentWriteTest.java:397)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest$UserRunner.contactExists(ConcurrentWriteTest.java:519)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest$UserRunner.call(ConcurrentWriteTest.java:540)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest$UserRunner.call(ConcurrentWriteTest.java:1)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
This concurrency test does not get to finish in over a minute. However, once LRU is
switched to NONE or FIFO, the test runs in 5 seconds.
So, something looks fishy with LRU.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:37 p.m.
New subject: [JBoss JIRA] Updated: (ISPN-277) LRU data container endlesly looping or containing heavily
[
https://jira.jboss.org/jira/browse/ISPN-277?page=com.atlassian.jira.plugi...
]
Galder Zamarreno updated ISPN-277:
----------------------------------
Assignee: Manik Surtani (was: Galder Zamarreno)
LRU data container endlesly looping or containing heavily
---------------------------------------------------------
Key: ISPN-277
URL: https://jira.jboss.org/jira/browse/ISPN-277
Project: Infinispan
Issue Type: Bug
Components: Eviction
Affects Versions: 4.0.0.CR2
Reporter: Galder Zamarreno
Assignee: Manik Surtani
Priority: Critical
Fix For: 4.0.0.GA
Attachments: td2.txt
Something around LRU container is not working fine. The attached log from an concurrency
test in the 2nd level cache shows that in 3 thread dumps taken over 30 seconds appart,
UserRunnerThread-5 is stuck in:
"UserRunnerThread-5" prio=10 tid=0x6f65bc00 nid=0xdea runnable [0x05efb000]
java.lang.Thread.State: RUNNABLE
at
org.infinispan.container.FIFODataContainer$LinkedEntry.casNext(FIFODataContainer.java:180)
at org.infinispan.container.FIFODataContainer.correctPrev(FIFODataContainer.java:329)
at org.infinispan.container.FIFODataContainer.linkAtEnd(FIFODataContainer.java:252)
at org.infinispan.container.LRUDataContainer.put(LRUDataContainer.java:70)
at
org.infinispan.container.entries.ReadCommittedEntry.commit(ReadCommittedEntry.java:161)
at
org.infinispan.interceptors.LockingInterceptor.commitEntry(LockingInterceptor.java:298)
at
org.infinispan.interceptors.LockingInterceptor.cleanupLocks(LockingInterceptor.java:281)
at
org.infinispan.interceptors.LockingInterceptor.doAfterCall(LockingInterceptor.java:243)
at
org.infinispan.interceptors.LockingInterceptor.visitPutKeyValueCommand(LockingInterceptor.java:200)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.MarshalledValueInterceptor.visitPutKeyValueCommand(MarshalledValueInterceptor.java:93)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:185)
at
org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:132)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:48)
at
org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:34)
at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:269)
at org.infinispan.CacheDelegate.putIfAbsent(CacheDelegate.java:422)
at org.infinispan.CacheDelegate.putIfAbsent(CacheDelegate.java:153)
at org.infinispan.CacheDelegate.putForExternalRead(CacheDelegate.java:243)
at
org.hibernate.cache.infinispan.util.CacheAdapterImpl.putForExternalRead(CacheAdapterImpl.java:115)
at
org.hibernate.cache.infinispan.access.TransactionalAccessDelegate.putFromLoad(TransactionalAccessDelegate.java:91)
at
org.hibernate.cache.infinispan.collection.TransactionalAccess.putFromLoad(TransactionalAccess.java:44)
at
org.hibernate.engine.loading.CollectionLoadContext.addCollectionToCache(CollectionLoadContext.java:333)
at
org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollection(CollectionLoadContext.java:279)
at
org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollections(CollectionLoadContext.java:245)
at
org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollections(CollectionLoadContext.java:218)
at org.hibernate.loader.Loader.endCollectionLoad(Loader.java:901)
at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:886)
at org.hibernate.loader.Loader.doQuery(Loader.java:750)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:257)
at org.hibernate.loader.Loader.loadCollection(Loader.java:2019)
at
org.hibernate.loader.collection.CollectionLoader.initialize(CollectionLoader.java:62)
at
org.hibernate.persister.collection.AbstractCollectionPersister.initialize(AbstractCollectionPersister.java:628)
at
org.hibernate.event.def.DefaultInitializeCollectionEventListener.onInitializeCollection(DefaultInitializeCollectionEventListener.java:83)
at org.hibernate.impl.SessionImpl.initializeCollection(SessionImpl.java:1817)
at
org.hibernate.collection.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:366)
at
org.hibernate.collection.AbstractPersistentCollection.read(AbstractPersistentCollection.java:108)
at
org.hibernate.collection.AbstractPersistentCollection.readSize(AbstractPersistentCollection.java:131)
at org.hibernate.collection.PersistentSet.isEmpty(PersistentSet.java:169)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest.getFirstContact(ConcurrentWriteTest.java:405)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest.access$0(ConcurrentWriteTest.java:397)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest$UserRunner.contactExists(ConcurrentWriteTest.java:519)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest$UserRunner.call(ConcurrentWriteTest.java:540)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest$UserRunner.call(ConcurrentWriteTest.java:1)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
This concurrency test does not get to finish in over a minute. However, once LRU is
switched to NONE or FIFO, the test runs in 5 seconds.
So, something looks fishy with LRU.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:56 p.m.
New subject: [JBoss JIRA] Updated: (ISPN-277) LRU data container endlesly looping or exhibiting heavy contention
[
https://jira.jboss.org/jira/browse/ISPN-277?page=com.atlassian.jira.plugi...
]
Manik Surtani updated ISPN-277:
-------------------------------
Summary: LRU data container endlesly looping or exhibiting heavy contention (was:
LRU data container endlesly looping or containing heavily)
Complexity: High
LRU data container endlesly looping or exhibiting heavy contention
------------------------------------------------------------------
Key: ISPN-277
URL: https://jira.jboss.org/jira/browse/ISPN-277
Project: Infinispan
Issue Type: Bug
Components: Eviction
Affects Versions: 4.0.0.CR2
Reporter: Galder Zamarreno
Assignee: Manik Surtani
Priority: Critical
Fix For: 4.0.0.GA
Attachments: td2.txt
Something around LRU container is not working fine. The attached log from an concurrency
test in the 2nd level cache shows that in 3 thread dumps taken over 30 seconds appart,
UserRunnerThread-5 is stuck in:
"UserRunnerThread-5" prio=10 tid=0x6f65bc00 nid=0xdea runnable [0x05efb000]
java.lang.Thread.State: RUNNABLE
at
org.infinispan.container.FIFODataContainer$LinkedEntry.casNext(FIFODataContainer.java:180)
at org.infinispan.container.FIFODataContainer.correctPrev(FIFODataContainer.java:329)
at org.infinispan.container.FIFODataContainer.linkAtEnd(FIFODataContainer.java:252)
at org.infinispan.container.LRUDataContainer.put(LRUDataContainer.java:70)
at
org.infinispan.container.entries.ReadCommittedEntry.commit(ReadCommittedEntry.java:161)
at
org.infinispan.interceptors.LockingInterceptor.commitEntry(LockingInterceptor.java:298)
at
org.infinispan.interceptors.LockingInterceptor.cleanupLocks(LockingInterceptor.java:281)
at
org.infinispan.interceptors.LockingInterceptor.doAfterCall(LockingInterceptor.java:243)
at
org.infinispan.interceptors.LockingInterceptor.visitPutKeyValueCommand(LockingInterceptor.java:200)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.MarshalledValueInterceptor.visitPutKeyValueCommand(MarshalledValueInterceptor.java:93)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:185)
at
org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:132)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:48)
at
org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:34)
at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:269)
at org.infinispan.CacheDelegate.putIfAbsent(CacheDelegate.java:422)
at org.infinispan.CacheDelegate.putIfAbsent(CacheDelegate.java:153)
at org.infinispan.CacheDelegate.putForExternalRead(CacheDelegate.java:243)
at
org.hibernate.cache.infinispan.util.CacheAdapterImpl.putForExternalRead(CacheAdapterImpl.java:115)
at
org.hibernate.cache.infinispan.access.TransactionalAccessDelegate.putFromLoad(TransactionalAccessDelegate.java:91)
at
org.hibernate.cache.infinispan.collection.TransactionalAccess.putFromLoad(TransactionalAccess.java:44)
at
org.hibernate.engine.loading.CollectionLoadContext.addCollectionToCache(CollectionLoadContext.java:333)
at
org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollection(CollectionLoadContext.java:279)
at
org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollections(CollectionLoadContext.java:245)
at
org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollections(CollectionLoadContext.java:218)
at org.hibernate.loader.Loader.endCollectionLoad(Loader.java:901)
at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:886)
at org.hibernate.loader.Loader.doQuery(Loader.java:750)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:257)
at org.hibernate.loader.Loader.loadCollection(Loader.java:2019)
at
org.hibernate.loader.collection.CollectionLoader.initialize(CollectionLoader.java:62)
at
org.hibernate.persister.collection.AbstractCollectionPersister.initialize(AbstractCollectionPersister.java:628)
at
org.hibernate.event.def.DefaultInitializeCollectionEventListener.onInitializeCollection(DefaultInitializeCollectionEventListener.java:83)
at org.hibernate.impl.SessionImpl.initializeCollection(SessionImpl.java:1817)
at
org.hibernate.collection.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:366)
at
org.hibernate.collection.AbstractPersistentCollection.read(AbstractPersistentCollection.java:108)
at
org.hibernate.collection.AbstractPersistentCollection.readSize(AbstractPersistentCollection.java:131)
at org.hibernate.collection.PersistentSet.isEmpty(PersistentSet.java:169)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest.getFirstContact(ConcurrentWriteTest.java:405)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest.access$0(ConcurrentWriteTest.java:397)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest$UserRunner.contactExists(ConcurrentWriteTest.java:519)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest$UserRunner.call(ConcurrentWriteTest.java:540)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest$UserRunner.call(ConcurrentWriteTest.java:1)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
This concurrency test does not get to finish in over a minute. However, once LRU is
switched to NONE or FIFO, the test runs in 5 seconds.
So, something looks fishy with LRU.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:09 p.m.
New subject: [JBoss JIRA] Commented: (ISPN-277) LRU data container endlesly looping or exhibiting heavy contention
[
https://jira.jboss.org/jira/browse/ISPN-277?page=com.atlassian.jira.plugi...
]
Manik Surtani commented on ISPN-277:
------------------------------------
Added parallel implementations of FIFO and LRU containers that are implemented using JDK
AtomicMarkableReferences rather than delegating marker nodes (similar to the technique
used in Sun's ConcurrentSkipListMap impl). This second impl is in the source tree but
is not used, since it does perform worse than the delegating marker node technique - from
ConcurrentSkipListMap's sources:
" * Rather than using mark-bits to mark list deletions (which can
* be slow and space-intensive using AtomicMarkedReference), nodes
* use direct CAS'able next pointers. On deletion, instead of
* marking a pointer, they splice in another node that can be
* thought of as standing for a marked pointer (indicating this by
* using otherwise impossible field values). Using plain nodes
* acts roughly like "boxed" implementations of marked pointers,
* but uses new nodes only when nodes are deleted, not for every
* link. This requires less space and supports faster
* traversal. Even if marked references were better supported by
* JVMs, traversal using this technique might still be faster
* because any search need only read ahead one more node than
* otherwise required (to check for trailing marker) rather than
* unmasking mark bits or whatever on each read.
"
However, this approach is not as readable and using AtomicMarkableReferences (AMRs) does
allow for a cleaner implementation which is easier to validate against the algorithm
design.
My adding AMR variants of FIFO and LRU containers is purely as a reference and a
validation point. These should not be used, and the FIFODataContainer and
LRUDataContainer should be preferred instead.
LRU data container endlesly looping or exhibiting heavy contention
------------------------------------------------------------------
Key: ISPN-277
URL: https://jira.jboss.org/jira/browse/ISPN-277
Project: Infinispan
Issue Type: Bug
Components: Eviction
Affects Versions: 4.0.0.CR2
Reporter: Galder Zamarreno
Assignee: Manik Surtani
Priority: Critical
Fix For: 4.0.0.GA
Attachments: td2.txt
Something around LRU container is not working fine. The attached log from an concurrency
test in the 2nd level cache shows that in 3 thread dumps taken over 30 seconds appart,
UserRunnerThread-5 is stuck in:
"UserRunnerThread-5" prio=10 tid=0x6f65bc00 nid=0xdea runnable [0x05efb000]
java.lang.Thread.State: RUNNABLE
at
org.infinispan.container.FIFODataContainer$LinkedEntry.casNext(FIFODataContainer.java:180)
at org.infinispan.container.FIFODataContainer.correctPrev(FIFODataContainer.java:329)
at org.infinispan.container.FIFODataContainer.linkAtEnd(FIFODataContainer.java:252)
at org.infinispan.container.LRUDataContainer.put(LRUDataContainer.java:70)
at
org.infinispan.container.entries.ReadCommittedEntry.commit(ReadCommittedEntry.java:161)
at
org.infinispan.interceptors.LockingInterceptor.commitEntry(LockingInterceptor.java:298)
at
org.infinispan.interceptors.LockingInterceptor.cleanupLocks(LockingInterceptor.java:281)
at
org.infinispan.interceptors.LockingInterceptor.doAfterCall(LockingInterceptor.java:243)
at
org.infinispan.interceptors.LockingInterceptor.visitPutKeyValueCommand(LockingInterceptor.java:200)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.MarshalledValueInterceptor.visitPutKeyValueCommand(MarshalledValueInterceptor.java:93)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:185)
at
org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:132)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:48)
at
org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:34)
at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:269)
at org.infinispan.CacheDelegate.putIfAbsent(CacheDelegate.java:422)
at org.infinispan.CacheDelegate.putIfAbsent(CacheDelegate.java:153)
at org.infinispan.CacheDelegate.putForExternalRead(CacheDelegate.java:243)
at
org.hibernate.cache.infinispan.util.CacheAdapterImpl.putForExternalRead(CacheAdapterImpl.java:115)
at
org.hibernate.cache.infinispan.access.TransactionalAccessDelegate.putFromLoad(TransactionalAccessDelegate.java:91)
at
org.hibernate.cache.infinispan.collection.TransactionalAccess.putFromLoad(TransactionalAccess.java:44)
at
org.hibernate.engine.loading.CollectionLoadContext.addCollectionToCache(CollectionLoadContext.java:333)
at
org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollection(CollectionLoadContext.java:279)
at
org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollections(CollectionLoadContext.java:245)
at
org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollections(CollectionLoadContext.java:218)
at org.hibernate.loader.Loader.endCollectionLoad(Loader.java:901)
at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:886)
at org.hibernate.loader.Loader.doQuery(Loader.java:750)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:257)
at org.hibernate.loader.Loader.loadCollection(Loader.java:2019)
at
org.hibernate.loader.collection.CollectionLoader.initialize(CollectionLoader.java:62)
at
org.hibernate.persister.collection.AbstractCollectionPersister.initialize(AbstractCollectionPersister.java:628)
at
org.hibernate.event.def.DefaultInitializeCollectionEventListener.onInitializeCollection(DefaultInitializeCollectionEventListener.java:83)
at org.hibernate.impl.SessionImpl.initializeCollection(SessionImpl.java:1817)
at
org.hibernate.collection.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:366)
at
org.hibernate.collection.AbstractPersistentCollection.read(AbstractPersistentCollection.java:108)
at
org.hibernate.collection.AbstractPersistentCollection.readSize(AbstractPersistentCollection.java:131)
at org.hibernate.collection.PersistentSet.isEmpty(PersistentSet.java:169)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest.getFirstContact(ConcurrentWriteTest.java:405)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest.access$0(ConcurrentWriteTest.java:397)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest$UserRunner.contactExists(ConcurrentWriteTest.java:519)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest$UserRunner.call(ConcurrentWriteTest.java:540)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest$UserRunner.call(ConcurrentWriteTest.java:1)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
This concurrency test does not get to finish in over a minute. However, once LRU is
switched to NONE or FIFO, the test runs in 5 seconds.
So, something looks fishy with LRU.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:11 p.m.
New subject: [JBoss JIRA] Commented: (ISPN-277) LRU data container endlesly looping or exhibiting heavy contention
[
https://jira.jboss.org/jira/browse/ISPN-277?page=com.atlassian.jira.plugi...
]
Manik Surtani commented on ISPN-277:
------------------------------------
The crux of the problem - apart from a race condition in correctPrev() - is that LRU
containers need to lock segments on get() as well as other write commands. I've
updated the Javadocs on the LRU containers to include this paragraph:
" * Note though that this implementation does have a far lesser degree of concurrency
when compared with its FIFO variant
* due to the segment locking necessary even when doing a get() (since gets reorder
links). This has a knock-on effect
* not just on get() but even on other write() operations since they all compete for the
same segment lock (when working
* on keys mapped to the same segment, of course)."
LRU data container endlesly looping or exhibiting heavy contention
------------------------------------------------------------------
Key: ISPN-277
URL: https://jira.jboss.org/jira/browse/ISPN-277
Project: Infinispan
Issue Type: Bug
Components: Eviction
Affects Versions: 4.0.0.CR2
Reporter: Galder Zamarreno
Assignee: Manik Surtani
Priority: Critical
Fix For: 4.0.0.GA
Attachments: td2.txt
Something around LRU container is not working fine. The attached log from an concurrency
test in the 2nd level cache shows that in 3 thread dumps taken over 30 seconds appart,
UserRunnerThread-5 is stuck in:
"UserRunnerThread-5" prio=10 tid=0x6f65bc00 nid=0xdea runnable [0x05efb000]
java.lang.Thread.State: RUNNABLE
at
org.infinispan.container.FIFODataContainer$LinkedEntry.casNext(FIFODataContainer.java:180)
at org.infinispan.container.FIFODataContainer.correctPrev(FIFODataContainer.java:329)
at org.infinispan.container.FIFODataContainer.linkAtEnd(FIFODataContainer.java:252)
at org.infinispan.container.LRUDataContainer.put(LRUDataContainer.java:70)
at
org.infinispan.container.entries.ReadCommittedEntry.commit(ReadCommittedEntry.java:161)
at
org.infinispan.interceptors.LockingInterceptor.commitEntry(LockingInterceptor.java:298)
at
org.infinispan.interceptors.LockingInterceptor.cleanupLocks(LockingInterceptor.java:281)
at
org.infinispan.interceptors.LockingInterceptor.doAfterCall(LockingInterceptor.java:243)
at
org.infinispan.interceptors.LockingInterceptor.visitPutKeyValueCommand(LockingInterceptor.java:200)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.MarshalledValueInterceptor.visitPutKeyValueCommand(MarshalledValueInterceptor.java:93)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:185)
at
org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:132)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:48)
at
org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:34)
at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:269)
at org.infinispan.CacheDelegate.putIfAbsent(CacheDelegate.java:422)
at org.infinispan.CacheDelegate.putIfAbsent(CacheDelegate.java:153)
at org.infinispan.CacheDelegate.putForExternalRead(CacheDelegate.java:243)
at
org.hibernate.cache.infinispan.util.CacheAdapterImpl.putForExternalRead(CacheAdapterImpl.java:115)
at
org.hibernate.cache.infinispan.access.TransactionalAccessDelegate.putFromLoad(TransactionalAccessDelegate.java:91)
at
org.hibernate.cache.infinispan.collection.TransactionalAccess.putFromLoad(TransactionalAccess.java:44)
at
org.hibernate.engine.loading.CollectionLoadContext.addCollectionToCache(CollectionLoadContext.java:333)
at
org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollection(CollectionLoadContext.java:279)
at
org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollections(CollectionLoadContext.java:245)
at
org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollections(CollectionLoadContext.java:218)
at org.hibernate.loader.Loader.endCollectionLoad(Loader.java:901)
at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:886)
at org.hibernate.loader.Loader.doQuery(Loader.java:750)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:257)
at org.hibernate.loader.Loader.loadCollection(Loader.java:2019)
at
org.hibernate.loader.collection.CollectionLoader.initialize(CollectionLoader.java:62)
at
org.hibernate.persister.collection.AbstractCollectionPersister.initialize(AbstractCollectionPersister.java:628)
at
org.hibernate.event.def.DefaultInitializeCollectionEventListener.onInitializeCollection(DefaultInitializeCollectionEventListener.java:83)
at org.hibernate.impl.SessionImpl.initializeCollection(SessionImpl.java:1817)
at
org.hibernate.collection.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:366)
at
org.hibernate.collection.AbstractPersistentCollection.read(AbstractPersistentCollection.java:108)
at
org.hibernate.collection.AbstractPersistentCollection.readSize(AbstractPersistentCollection.java:131)
at org.hibernate.collection.PersistentSet.isEmpty(PersistentSet.java:169)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest.getFirstContact(ConcurrentWriteTest.java:405)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest.access$0(ConcurrentWriteTest.java:397)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest$UserRunner.contactExists(ConcurrentWriteTest.java:519)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest$UserRunner.call(ConcurrentWriteTest.java:540)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest$UserRunner.call(ConcurrentWriteTest.java:1)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
This concurrency test does not get to finish in over a minute. However, once LRU is
switched to NONE or FIFO, the test runs in 5 seconds.
So, something looks fishy with LRU.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:13 p.m.
New subject: [JBoss JIRA] Resolved: (ISPN-277) LRU data container endlesly looping or exhibiting heavy contention
[
https://jira.jboss.org/jira/browse/ISPN-277?page=com.atlassian.jira.plugi...
]
Manik Surtani resolved ISPN-277.
--------------------------------
Resolution: Done
LRU data container endlesly looping or exhibiting heavy contention
------------------------------------------------------------------
Key: ISPN-277
URL: https://jira.jboss.org/jira/browse/ISPN-277
Project: Infinispan
Issue Type: Bug
Components: Eviction
Affects Versions: 4.0.0.CR2
Reporter: Galder Zamarreno
Assignee: Manik Surtani
Priority: Critical
Fix For: 4.0.0.GA
Attachments: td2.txt
Something around LRU container is not working fine. The attached log from an concurrency
test in the 2nd level cache shows that in 3 thread dumps taken over 30 seconds appart,
UserRunnerThread-5 is stuck in:
"UserRunnerThread-5" prio=10 tid=0x6f65bc00 nid=0xdea runnable [0x05efb000]
java.lang.Thread.State: RUNNABLE
at
org.infinispan.container.FIFODataContainer$LinkedEntry.casNext(FIFODataContainer.java:180)
at org.infinispan.container.FIFODataContainer.correctPrev(FIFODataContainer.java:329)
at org.infinispan.container.FIFODataContainer.linkAtEnd(FIFODataContainer.java:252)
at org.infinispan.container.LRUDataContainer.put(LRUDataContainer.java:70)
at
org.infinispan.container.entries.ReadCommittedEntry.commit(ReadCommittedEntry.java:161)
at
org.infinispan.interceptors.LockingInterceptor.commitEntry(LockingInterceptor.java:298)
at
org.infinispan.interceptors.LockingInterceptor.cleanupLocks(LockingInterceptor.java:281)
at
org.infinispan.interceptors.LockingInterceptor.doAfterCall(LockingInterceptor.java:243)
at
org.infinispan.interceptors.LockingInterceptor.visitPutKeyValueCommand(LockingInterceptor.java:200)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.MarshalledValueInterceptor.visitPutKeyValueCommand(MarshalledValueInterceptor.java:93)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:185)
at
org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:132)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:48)
at
org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:34)
at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:269)
at org.infinispan.CacheDelegate.putIfAbsent(CacheDelegate.java:422)
at org.infinispan.CacheDelegate.putIfAbsent(CacheDelegate.java:153)
at org.infinispan.CacheDelegate.putForExternalRead(CacheDelegate.java:243)
at
org.hibernate.cache.infinispan.util.CacheAdapterImpl.putForExternalRead(CacheAdapterImpl.java:115)
at
org.hibernate.cache.infinispan.access.TransactionalAccessDelegate.putFromLoad(TransactionalAccessDelegate.java:91)
at
org.hibernate.cache.infinispan.collection.TransactionalAccess.putFromLoad(TransactionalAccess.java:44)
at
org.hibernate.engine.loading.CollectionLoadContext.addCollectionToCache(CollectionLoadContext.java:333)
at
org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollection(CollectionLoadContext.java:279)
at
org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollections(CollectionLoadContext.java:245)
at
org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollections(CollectionLoadContext.java:218)
at org.hibernate.loader.Loader.endCollectionLoad(Loader.java:901)
at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:886)
at org.hibernate.loader.Loader.doQuery(Loader.java:750)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:257)
at org.hibernate.loader.Loader.loadCollection(Loader.java:2019)
at
org.hibernate.loader.collection.CollectionLoader.initialize(CollectionLoader.java:62)
at
org.hibernate.persister.collection.AbstractCollectionPersister.initialize(AbstractCollectionPersister.java:628)
at
org.hibernate.event.def.DefaultInitializeCollectionEventListener.onInitializeCollection(DefaultInitializeCollectionEventListener.java:83)
at org.hibernate.impl.SessionImpl.initializeCollection(SessionImpl.java:1817)
at
org.hibernate.collection.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:366)
at
org.hibernate.collection.AbstractPersistentCollection.read(AbstractPersistentCollection.java:108)
at
org.hibernate.collection.AbstractPersistentCollection.readSize(AbstractPersistentCollection.java:131)
at org.hibernate.collection.PersistentSet.isEmpty(PersistentSet.java:169)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest.getFirstContact(ConcurrentWriteTest.java:405)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest.access$0(ConcurrentWriteTest.java:397)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest$UserRunner.contactExists(ConcurrentWriteTest.java:519)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest$UserRunner.call(ConcurrentWriteTest.java:540)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest$UserRunner.call(ConcurrentWriteTest.java:1)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
This concurrency test does not get to finish in over a minute. However, once LRU is
switched to NONE or FIFO, the test runs in 5 seconds.
So, something looks fishy with LRU.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:55 p.m.
New subject: [JBoss JIRA] Updated: (ISPN-277) LRU data container endlesly looping or exhibiting heavy contention
[
https://jira.jboss.org/jira/browse/ISPN-277?page=com.atlassian.jira.plugi...
]
Manik Surtani updated ISPN-277:
-------------------------------
Fix Version/s: 4.0.0.CR3
(was: 4.0.0.GA)
LRU data container endlesly looping or exhibiting heavy contention
------------------------------------------------------------------
Key: ISPN-277
URL: https://jira.jboss.org/jira/browse/ISPN-277
Project: Infinispan
Issue Type: Bug
Components: Eviction
Affects Versions: 4.0.0.CR2
Reporter: Galder Zamarreno
Assignee: Manik Surtani
Priority: Critical
Fix For: 4.0.0.CR3
Attachments: td2.txt
Something around LRU container is not working fine. The attached log from an concurrency
test in the 2nd level cache shows that in 3 thread dumps taken over 30 seconds appart,
UserRunnerThread-5 is stuck in:
"UserRunnerThread-5" prio=10 tid=0x6f65bc00 nid=0xdea runnable [0x05efb000]
java.lang.Thread.State: RUNNABLE
at
org.infinispan.container.FIFODataContainer$LinkedEntry.casNext(FIFODataContainer.java:180)
at org.infinispan.container.FIFODataContainer.correctPrev(FIFODataContainer.java:329)
at org.infinispan.container.FIFODataContainer.linkAtEnd(FIFODataContainer.java:252)
at org.infinispan.container.LRUDataContainer.put(LRUDataContainer.java:70)
at
org.infinispan.container.entries.ReadCommittedEntry.commit(ReadCommittedEntry.java:161)
at
org.infinispan.interceptors.LockingInterceptor.commitEntry(LockingInterceptor.java:298)
at
org.infinispan.interceptors.LockingInterceptor.cleanupLocks(LockingInterceptor.java:281)
at
org.infinispan.interceptors.LockingInterceptor.doAfterCall(LockingInterceptor.java:243)
at
org.infinispan.interceptors.LockingInterceptor.visitPutKeyValueCommand(LockingInterceptor.java:200)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.MarshalledValueInterceptor.visitPutKeyValueCommand(MarshalledValueInterceptor.java:93)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:185)
at
org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:132)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:48)
at
org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:34)
at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:269)
at org.infinispan.CacheDelegate.putIfAbsent(CacheDelegate.java:422)
at org.infinispan.CacheDelegate.putIfAbsent(CacheDelegate.java:153)
at org.infinispan.CacheDelegate.putForExternalRead(CacheDelegate.java:243)
at
org.hibernate.cache.infinispan.util.CacheAdapterImpl.putForExternalRead(CacheAdapterImpl.java:115)
at
org.hibernate.cache.infinispan.access.TransactionalAccessDelegate.putFromLoad(TransactionalAccessDelegate.java:91)
at
org.hibernate.cache.infinispan.collection.TransactionalAccess.putFromLoad(TransactionalAccess.java:44)
at
org.hibernate.engine.loading.CollectionLoadContext.addCollectionToCache(CollectionLoadContext.java:333)
at
org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollection(CollectionLoadContext.java:279)
at
org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollections(CollectionLoadContext.java:245)
at
org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollections(CollectionLoadContext.java:218)
at org.hibernate.loader.Loader.endCollectionLoad(Loader.java:901)
at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:886)
at org.hibernate.loader.Loader.doQuery(Loader.java:750)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:257)
at org.hibernate.loader.Loader.loadCollection(Loader.java:2019)
at
org.hibernate.loader.collection.CollectionLoader.initialize(CollectionLoader.java:62)
at
org.hibernate.persister.collection.AbstractCollectionPersister.initialize(AbstractCollectionPersister.java:628)
at
org.hibernate.event.def.DefaultInitializeCollectionEventListener.onInitializeCollection(DefaultInitializeCollectionEventListener.java:83)
at org.hibernate.impl.SessionImpl.initializeCollection(SessionImpl.java:1817)
at
org.hibernate.collection.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:366)
at
org.hibernate.collection.AbstractPersistentCollection.read(AbstractPersistentCollection.java:108)
at
org.hibernate.collection.AbstractPersistentCollection.readSize(AbstractPersistentCollection.java:131)
at org.hibernate.collection.PersistentSet.isEmpty(PersistentSet.java:169)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest.getFirstContact(ConcurrentWriteTest.java:405)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest.access$0(ConcurrentWriteTest.java:397)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest$UserRunner.contactExists(ConcurrentWriteTest.java:519)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest$UserRunner.call(ConcurrentWriteTest.java:540)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest$UserRunner.call(ConcurrentWriteTest.java:1)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
This concurrency test does not get to finish in over a minute. However, once LRU is
switched to NONE or FIFO, the test runs in 5 seconds.
So, something looks fishy with LRU.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:51 p.m.
New subject: [JBoss JIRA] Updated: (ISPN-277) LRU data container endlesly looping or exhibiting heavy contention
[
https://jira.jboss.org/jira/browse/ISPN-277?page=com.atlassian.jira.plugi...
]
Galder Zamarreno updated ISPN-277:
----------------------------------
Attachment: 3dumps-fail.txt
After 1 minute, org.hibernate.test.cache.infinispan.functional.testManyUsers still
timeouts after waiting for around 1 minute for the test to work.
I'm running latest trunk, just built minutes ago:
[galder(a)localhost]~/.m2/repository/org/infinispan/infinispan-core% ls -al
total 40
....
drwxr-xr-x. 2 galder galder 4096 2009-11-27 17:33 4.0.0-SNAPSHOT
And the infinispan version in hibernate-infinispan/pom.xml points there:
<version.infinispan>4.0.0-SNAPSHOT</version.infinispan>
LRU data container endlesly looping or exhibiting heavy contention
------------------------------------------------------------------
Key: ISPN-277
URL: https://jira.jboss.org/jira/browse/ISPN-277
Project: Infinispan
Issue Type: Bug
Components: Eviction
Affects Versions: 4.0.0.CR2
Reporter: Galder Zamarreno
Assignee: Manik Surtani
Priority: Critical
Fix For: 4.0.0.CR3
Attachments: 3dumps-fail.txt, td2.txt
Something around LRU container is not working fine. The attached log from an concurrency
test in the 2nd level cache shows that in 3 thread dumps taken over 30 seconds appart,
UserRunnerThread-5 is stuck in:
"UserRunnerThread-5" prio=10 tid=0x6f65bc00 nid=0xdea runnable [0x05efb000]
java.lang.Thread.State: RUNNABLE
at
org.infinispan.container.FIFODataContainer$LinkedEntry.casNext(FIFODataContainer.java:180)
at org.infinispan.container.FIFODataContainer.correctPrev(FIFODataContainer.java:329)
at org.infinispan.container.FIFODataContainer.linkAtEnd(FIFODataContainer.java:252)
at org.infinispan.container.LRUDataContainer.put(LRUDataContainer.java:70)
at
org.infinispan.container.entries.ReadCommittedEntry.commit(ReadCommittedEntry.java:161)
at
org.infinispan.interceptors.LockingInterceptor.commitEntry(LockingInterceptor.java:298)
at
org.infinispan.interceptors.LockingInterceptor.cleanupLocks(LockingInterceptor.java:281)
at
org.infinispan.interceptors.LockingInterceptor.doAfterCall(LockingInterceptor.java:243)
at
org.infinispan.interceptors.LockingInterceptor.visitPutKeyValueCommand(LockingInterceptor.java:200)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.MarshalledValueInterceptor.visitPutKeyValueCommand(MarshalledValueInterceptor.java:93)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:185)
at
org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:132)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:48)
at
org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:34)
at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:269)
at org.infinispan.CacheDelegate.putIfAbsent(CacheDelegate.java:422)
at org.infinispan.CacheDelegate.putIfAbsent(CacheDelegate.java:153)
at org.infinispan.CacheDelegate.putForExternalRead(CacheDelegate.java:243)
at
org.hibernate.cache.infinispan.util.CacheAdapterImpl.putForExternalRead(CacheAdapterImpl.java:115)
at
org.hibernate.cache.infinispan.access.TransactionalAccessDelegate.putFromLoad(TransactionalAccessDelegate.java:91)
at
org.hibernate.cache.infinispan.collection.TransactionalAccess.putFromLoad(TransactionalAccess.java:44)
at
org.hibernate.engine.loading.CollectionLoadContext.addCollectionToCache(CollectionLoadContext.java:333)
at
org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollection(CollectionLoadContext.java:279)
at
org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollections(CollectionLoadContext.java:245)
at
org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollections(CollectionLoadContext.java:218)
at org.hibernate.loader.Loader.endCollectionLoad(Loader.java:901)
at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:886)
at org.hibernate.loader.Loader.doQuery(Loader.java:750)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:257)
at org.hibernate.loader.Loader.loadCollection(Loader.java:2019)
at
org.hibernate.loader.collection.CollectionLoader.initialize(CollectionLoader.java:62)
at
org.hibernate.persister.collection.AbstractCollectionPersister.initialize(AbstractCollectionPersister.java:628)
at
org.hibernate.event.def.DefaultInitializeCollectionEventListener.onInitializeCollection(DefaultInitializeCollectionEventListener.java:83)
at org.hibernate.impl.SessionImpl.initializeCollection(SessionImpl.java:1817)
at
org.hibernate.collection.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:366)
at
org.hibernate.collection.AbstractPersistentCollection.read(AbstractPersistentCollection.java:108)
at
org.hibernate.collection.AbstractPersistentCollection.readSize(AbstractPersistentCollection.java:131)
at org.hibernate.collection.PersistentSet.isEmpty(PersistentSet.java:169)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest.getFirstContact(ConcurrentWriteTest.java:405)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest.access$0(ConcurrentWriteTest.java:397)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest$UserRunner.contactExists(ConcurrentWriteTest.java:519)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest$UserRunner.call(ConcurrentWriteTest.java:540)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest$UserRunner.call(ConcurrentWriteTest.java:1)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
This concurrency test does not get to finish in over a minute. However, once LRU is
switched to NONE or FIFO, the test runs in 5 seconds.
So, something looks fishy with LRU.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:59 p.m.
New subject: [JBoss JIRA] Commented: (ISPN-277) LRU data container endlesly looping or exhibiting heavy contention
[
https://jira.jboss.org/jira/browse/ISPN-277?page=com.atlassian.jira.plugi...
]
Manik Surtani commented on ISPN-277:
------------------------------------
Corrected this. Essentially,
* FIFODataContainer and LRUDataContainer are implementations of a concurrent linked
hashmap, similar to the JDK's LinkedHashMap, utilising the Sundell/Tsigas lock-free
linked list algorithm [1].
* There is a race in the CorrectPrev() function, which leads to an infinite loop under
very specific conditions. Easy to recreate in a stress test I have.
* Fixing this is very hard. I now have 2 alternate implementations (FIFODataContainer and
FIFOAMRDataContainer, see Javadocs on both to understand the differences)
It appears that the problem lies in the concurrent update of a link on get as well as put,
something that the algorithm may not support. I am in conversation with the algorithm
designers to detail this out some more.
In the meanwhile, I have implemented FIFO and LRU data containers using standard JDK
collections but it will perform much worse - O(N log N) during iteration although put, get
and remove have the same O(1) performance. There is a memory overhead too, since last
used or creation timestamps are collected for every entry (an extra 8 bytes per entry for
immortal entries, no extra overhead for others).
My plan is to go with these implementations for now, and once we have sorted the
concurrent linked list issues we look at reverting to the better performing
implementations.
[1] http://www.md.chalmers.se/~tsigas/papers/Lock-Free-Deques-Doubly-Lists-JP...
LRU data container endlesly looping or exhibiting heavy contention
------------------------------------------------------------------
Key: ISPN-277
URL: https://jira.jboss.org/jira/browse/ISPN-277
Project: Infinispan
Issue Type: Bug
Components: Eviction
Affects Versions: 4.0.0.CR2
Reporter: Galder Zamarreno
Assignee: Manik Surtani
Priority: Critical
Fix For: 4.0.0.CR3
Attachments: 3dumps-fail.txt, td2.txt
Something around LRU container is not working fine. The attached log from an concurrency
test in the 2nd level cache shows that in 3 thread dumps taken over 30 seconds appart,
UserRunnerThread-5 is stuck in:
"UserRunnerThread-5" prio=10 tid=0x6f65bc00 nid=0xdea runnable [0x05efb000]
java.lang.Thread.State: RUNNABLE
at
org.infinispan.container.FIFODataContainer$LinkedEntry.casNext(FIFODataContainer.java:180)
at org.infinispan.container.FIFODataContainer.correctPrev(FIFODataContainer.java:329)
at org.infinispan.container.FIFODataContainer.linkAtEnd(FIFODataContainer.java:252)
at org.infinispan.container.LRUDataContainer.put(LRUDataContainer.java:70)
at
org.infinispan.container.entries.ReadCommittedEntry.commit(ReadCommittedEntry.java:161)
at
org.infinispan.interceptors.LockingInterceptor.commitEntry(LockingInterceptor.java:298)
at
org.infinispan.interceptors.LockingInterceptor.cleanupLocks(LockingInterceptor.java:281)
at
org.infinispan.interceptors.LockingInterceptor.doAfterCall(LockingInterceptor.java:243)
at
org.infinispan.interceptors.LockingInterceptor.visitPutKeyValueCommand(LockingInterceptor.java:200)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.MarshalledValueInterceptor.visitPutKeyValueCommand(MarshalledValueInterceptor.java:93)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:185)
at
org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:132)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at
org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:48)
at
org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:34)
at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)
at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:269)
at org.infinispan.CacheDelegate.putIfAbsent(CacheDelegate.java:422)
at org.infinispan.CacheDelegate.putIfAbsent(CacheDelegate.java:153)
at org.infinispan.CacheDelegate.putForExternalRead(CacheDelegate.java:243)
at
org.hibernate.cache.infinispan.util.CacheAdapterImpl.putForExternalRead(CacheAdapterImpl.java:115)
at
org.hibernate.cache.infinispan.access.TransactionalAccessDelegate.putFromLoad(TransactionalAccessDelegate.java:91)
at
org.hibernate.cache.infinispan.collection.TransactionalAccess.putFromLoad(TransactionalAccess.java:44)
at
org.hibernate.engine.loading.CollectionLoadContext.addCollectionToCache(CollectionLoadContext.java:333)
at
org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollection(CollectionLoadContext.java:279)
at
org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollections(CollectionLoadContext.java:245)
at
org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollections(CollectionLoadContext.java:218)
at org.hibernate.loader.Loader.endCollectionLoad(Loader.java:901)
at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:886)
at org.hibernate.loader.Loader.doQuery(Loader.java:750)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:257)
at org.hibernate.loader.Loader.loadCollection(Loader.java:2019)
at
org.hibernate.loader.collection.CollectionLoader.initialize(CollectionLoader.java:62)
at
org.hibernate.persister.collection.AbstractCollectionPersister.initialize(AbstractCollectionPersister.java:628)
at
org.hibernate.event.def.DefaultInitializeCollectionEventListener.onInitializeCollection(DefaultInitializeCollectionEventListener.java:83)
at org.hibernate.impl.SessionImpl.initializeCollection(SessionImpl.java:1817)
at
org.hibernate.collection.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:366)
at
org.hibernate.collection.AbstractPersistentCollection.read(AbstractPersistentCollection.java:108)
at
org.hibernate.collection.AbstractPersistentCollection.readSize(AbstractPersistentCollection.java:131)
at org.hibernate.collection.PersistentSet.isEmpty(PersistentSet.java:169)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest.getFirstContact(ConcurrentWriteTest.java:405)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest.access$0(ConcurrentWriteTest.java:397)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest$UserRunner.contactExists(ConcurrentWriteTest.java:519)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest$UserRunner.call(ConcurrentWriteTest.java:540)
at
org.hibernate.test.cache.infinispan.functional.ConcurrentWriteTest$UserRunner.call(ConcurrentWriteTest.java:1)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
This concurrency test does not get to finish in over a minute. However, once LRU is
switched to NONE or FIFO, the test runs in 5 seconds.
So, something looks fishy with LRU.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5503
days inactive
5525
days old
infinispan-issues@lists.jboss.org
10 comments
2 participants
participants (2)
-
Galder Zamarreno (JIRA) -
Manik Surtani (JIRA)