[infinispan-issues] [JBoss JIRA] Commented: (ISPN-501) chunks might be deleted while Searcher is iterating

[Sanne Grinovero (JIRA)]

    [ https://jira.jboss.org/browse/ISPN-501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12537288#action_12537288 ] 

Sanne Grinovero commented on ISPN-501:
--------------------------------------

I have been thinking about several options:

1) don't delete elements but set them to expire after some minutes. Not happy about this as it would require users to configure a sensible expiry time. Also there's not an API in core to change the status of a value to expire, while editing a proof of concept I had to retrieve the element's value to use put again - if we decide to go for this solution I'll open a feature request to edit the expiry values for a given key without retrieving the value.

2) reorder instructions: delete filename first, then use batching to delete all segments at once. There are still race conditions possible so the issue wouldn't be solved, just made even more unlikely.
I'd like to avoid having to read all chunks eagerly before returning "yes the file exists", they might be huge and make the distribution useless.

3) use an AtomicMap
Seems to also prevent distribution of the index segment; we could workaround this by configuring the index to not optimize to a single segment but to keep several smaller segments around (not optimal for performance, but not too bad).

4) eagerly lock all segments.
As this would prevent concurrent searchers AFAIK, that would be nonsense; I would need a "read lock", actually I only need to prevent values from being deleted until the InfinispanIndexInput is closed. Also the opening of an InfinispanIndexInput should be extremely fast.

First tests of attempts to work around this issue where affected by ISPN-512, which is solved now but needs me to rerun experiments on the different approaches.

> chunks might be deleted while Searcher is iterating
> ---------------------------------------------------
>
>                 Key: ISPN-501
>                 URL: https://jira.jboss.org/browse/ISPN-501
>             Project: Infinispan
>          Issue Type: Bug
>          Components: Lucene Directory
>            Reporter: Sanne Grinovero
>            Assignee: Sanne Grinovero
>            Priority: Critical
>             Fix For: 4.1.0.CR1, 5.0.0.BETA1
>
>
> This one is quite nasty:
> java.io.IOException: Chunk id = [ 5 ] does not exist for file [ _sc.cfs ] for index [ iname ], file position: [ 88420 ], file size: [ 93015 ]
> 	at org.infinispan.lucene.InfinispanIndexIO$InfinispanIndexInput.readBytes(InfinispanIndexIO.java:150)
> 	at org.apache.lucene.store.IndexInput.readBytes(IndexInput.java:61)
> 	at org.apache.lucene.index.CompoundFileReader$CSIndexInput.readInternal(CompoundFileReader.java:247)
> 	at org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:136)
> 	at org.apache.lucene.index.SegmentReader$Norm.bytes(SegmentReader.java:469)
> 	at org.apache.lucene.index.SegmentReader.getNorms(SegmentReader.java:1090)
> 	at org.apache.lucene.index.SegmentReader.norms(SegmentReader.java:1096)
> 	at org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:72)
> 	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:250)
> 	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:175)
> 	at org.apache.lucene.search.Searcher.search(Searcher.java:181)
> 	at org.infinispan.lucene.profiling.LuceneReaderThread.testLoop(LuceneReaderThread.java:63)
> 	at org.infinispan.lucene.profiling.LuceneUserThread.run(LuceneUserThread.java:55)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:619)
> Easily reproducible by starting "mvn test -Dtest=PerformanceCompareStressTest" with the currently committed parameters (r1925)
> I'll likely need an AtomicMap or use transactions while reading?

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira