2:59 p.m.
chunks might be deleted while Searcher is iterating
---------------------------------------------------
Key: ISPN-501
URL: https://jira.jboss.org/browse/ISPN-501
Project: Infinispan
Issue Type: Bug
Components: Lucene Directory
Reporter: Sanne Grinovero
Assignee: Sanne Grinovero
Priority: Critical
Fix For: 4.1.0.CR1, 5.0.0.BETA1
This one is quite nasty:
java.io.IOException: Chunk id = [ 5 ] does not exist for file [ _sc.cfs ] for index [
iname ], file position: [ 88420 ], file size: [ 93015 ]
at
org.infinispan.lucene.InfinispanIndexIO$InfinispanIndexInput.readBytes(InfinispanIndexIO.java:150)
at org.apache.lucene.store.IndexInput.readBytes(IndexInput.java:61)
at
org.apache.lucene.index.CompoundFileReader$CSIndexInput.readInternal(CompoundFileReader.java:247)
at org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:136)
at org.apache.lucene.index.SegmentReader$Norm.bytes(SegmentReader.java:469)
at org.apache.lucene.index.SegmentReader.getNorms(SegmentReader.java:1090)
at org.apache.lucene.index.SegmentReader.norms(SegmentReader.java:1096)
at org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:72)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:250)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:175)
at org.apache.lucene.search.Searcher.search(Searcher.java:181)
at
org.infinispan.lucene.profiling.LuceneReaderThread.testLoop(LuceneReaderThread.java:63)
at org.infinispan.lucene.profiling.LuceneUserThread.run(LuceneUserThread.java:55)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Easily reproducible by starting "mvn test -Dtest=PerformanceCompareStressTest"
with the currently committed parameters (r1925)
I'll likely need an AtomicMap or use transactions while reading?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:37 a.m.
New subject: [JBoss JIRA] Commented: (ISPN-501) chunks might be deleted while Searcher is iterating
[
https://jira.jboss.org/browse/ISPN-501?page=com.atlassian.jira.plugin.sys...
]
Sanne Grinovero commented on ISPN-501:
--------------------------------------
I have been thinking about several options:
1) don't delete elements but set them to expire after some minutes. Not happy about
this as it would require users to configure a sensible expiry time. Also there's not
an API in core to change the status of a value to expire, while editing a proof of concept
I had to retrieve the element's value to use put again - if we decide to go for this
solution I'll open a feature request to edit the expiry values for a given key without
retrieving the value.
2) reorder instructions: delete filename first, then use batching to delete all segments
at once. There are still race conditions possible so the issue wouldn't be solved,
just made even more unlikely.
I'd like to avoid having to read all chunks eagerly before returning "yes the
file exists", they might be huge and make the distribution useless.
3) use an AtomicMap
Seems to also prevent distribution of the index segment; we could workaround this by
configuring the index to not optimize to a single segment but to keep several smaller
segments around (not optimal for performance, but not too bad).
4) eagerly lock all segments.
As this would prevent concurrent searchers AFAIK, that would be nonsense; I would need a
"read lock", actually I only need to prevent values from being deleted until the
InfinispanIndexInput is closed. Also the opening of an InfinispanIndexInput should be
extremely fast.
First tests of attempts to work around this issue where affected by ISPN-512, which is
solved now but needs me to rerun experiments on the different approaches.
chunks might be deleted while Searcher is iterating
---------------------------------------------------
Key: ISPN-501
URL: https://jira.jboss.org/browse/ISPN-501
Project: Infinispan
Issue Type: Bug
Components: Lucene Directory
Reporter: Sanne Grinovero
Assignee: Sanne Grinovero
Priority: Critical
Fix For: 4.1.0.CR1, 5.0.0.BETA1
This one is quite nasty:
java.io.IOException: Chunk id = [ 5 ] does not exist for file [ _sc.cfs ] for index [
iname ], file position: [ 88420 ], file size: [ 93015 ]
at
org.infinispan.lucene.InfinispanIndexIO$InfinispanIndexInput.readBytes(InfinispanIndexIO.java:150)
at org.apache.lucene.store.IndexInput.readBytes(IndexInput.java:61)
at
org.apache.lucene.index.CompoundFileReader$CSIndexInput.readInternal(CompoundFileReader.java:247)
at org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:136)
at org.apache.lucene.index.SegmentReader$Norm.bytes(SegmentReader.java:469)
at org.apache.lucene.index.SegmentReader.getNorms(SegmentReader.java:1090)
at org.apache.lucene.index.SegmentReader.norms(SegmentReader.java:1096)
at org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:72)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:250)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:175)
at org.apache.lucene.search.Searcher.search(Searcher.java:181)
at
org.infinispan.lucene.profiling.LuceneReaderThread.testLoop(LuceneReaderThread.java:63)
at org.infinispan.lucene.profiling.LuceneUserThread.run(LuceneUserThread.java:55)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Easily reproducible by starting "mvn test -Dtest=PerformanceCompareStressTest"
with the currently committed parameters (r1925)
I'll likely need an AtomicMap or use transactions while reading?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:51 a.m.
New subject: [JBoss JIRA] Resolved: (ISPN-501) chunks might be deleted while Searcher is iterating
[
https://jira.jboss.org/browse/ISPN-501?page=com.atlassian.jira.plugin.sys...
]
Sanne Grinovero resolved ISPN-501.
----------------------------------
Resolution: Done
chunks might be deleted while Searcher is iterating
---------------------------------------------------
Key: ISPN-501
URL: https://jira.jboss.org/browse/ISPN-501
Project: Infinispan
Issue Type: Bug
Components: Lucene Directory
Reporter: Sanne Grinovero
Assignee: Sanne Grinovero
Priority: Critical
Fix For: 4.1.0.CR1, 5.0.0.BETA1
This one is quite nasty:
java.io.IOException: Chunk id = [ 5 ] does not exist for file [ _sc.cfs ] for index [
iname ], file position: [ 88420 ], file size: [ 93015 ]
at
org.infinispan.lucene.InfinispanIndexIO$InfinispanIndexInput.readBytes(InfinispanIndexIO.java:150)
at org.apache.lucene.store.IndexInput.readBytes(IndexInput.java:61)
at
org.apache.lucene.index.CompoundFileReader$CSIndexInput.readInternal(CompoundFileReader.java:247)
at org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:136)
at org.apache.lucene.index.SegmentReader$Norm.bytes(SegmentReader.java:469)
at org.apache.lucene.index.SegmentReader.getNorms(SegmentReader.java:1090)
at org.apache.lucene.index.SegmentReader.norms(SegmentReader.java:1096)
at org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:72)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:250)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:175)
at org.apache.lucene.search.Searcher.search(Searcher.java:181)
at
org.infinispan.lucene.profiling.LuceneReaderThread.testLoop(LuceneReaderThread.java:63)
at org.infinispan.lucene.profiling.LuceneUserThread.run(LuceneUserThread.java:55)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Easily reproducible by starting "mvn test -Dtest=PerformanceCompareStressTest"
with the currently committed parameters (r1925)
I'll likely need an AtomicMap or use transactions while reading?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:51 a.m.
New subject: [JBoss JIRA] Updated: (ISPN-501) chunks might be deleted while Searcher is iterating
[
https://issues.jboss.org/browse/ISPN-501?page=com.atlassian.jira.plugin.s...
]
Galder Zamarreño updated ISPN-501:
----------------------------------
Fix Version/s: (was: 5.0.0.BETA1)
chunks might be deleted while Searcher is iterating
---------------------------------------------------
Key: ISPN-501
URL: https://issues.jboss.org/browse/ISPN-501
Project: Infinispan
Issue Type: Bug
Components: Lucene Directory
Reporter: Sanne Grinovero
Assignee: Sanne Grinovero
Priority: Critical
Fix For: 4.1.0.CR1
This one is quite nasty:
java.io.IOException: Chunk id = [ 5 ] does not exist for file [ _sc.cfs ] for index [
iname ], file position: [ 88420 ], file size: [ 93015 ]
at
org.infinispan.lucene.InfinispanIndexIO$InfinispanIndexInput.readBytes(InfinispanIndexIO.java:150)
at org.apache.lucene.store.IndexInput.readBytes(IndexInput.java:61)
at
org.apache.lucene.index.CompoundFileReader$CSIndexInput.readInternal(CompoundFileReader.java:247)
at org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:136)
at org.apache.lucene.index.SegmentReader$Norm.bytes(SegmentReader.java:469)
at org.apache.lucene.index.SegmentReader.getNorms(SegmentReader.java:1090)
at org.apache.lucene.index.SegmentReader.norms(SegmentReader.java:1096)
at org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:72)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:250)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:175)
at org.apache.lucene.search.Searcher.search(Searcher.java:181)
at
org.infinispan.lucene.profiling.LuceneReaderThread.testLoop(LuceneReaderThread.java:63)
at org.infinispan.lucene.profiling.LuceneUserThread.run(LuceneUserThread.java:55)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Easily reproducible by starting "mvn test -Dtest=PerformanceCompareStressTest"
with the currently committed parameters (r1925)
I'll likely need an AtomicMap or use transactions while reading?
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5120
days inactive
5298
days old
infinispan-issues@lists.jboss.org
3 comments
2 participants
participants (2)
-
Galder Zamarreño (JIRA) -
Sanne Grinovero (JIRA)