[hibernate-dev] Re: improving Search

Hello Emmanuel, as you asked how to use the SnapshotDeletionPolicy: when you create the IndexWriter do: IndexDeletionPolicy policy = new KeepOnlyLastCommitDeletionPolicy(); SnapshotDeletionPolicy snapshotter = new SnapshotDeletionPolicy(policy); IndexWriter writer = new IndexWriter(dir, autoCommit, analyzer, snapshotter); then when you want to make a copy you "freeze" the existence of current index segments, kindly asking the IndexWriter to avoid r emoving or otherwise change the files your are going to copy; it also tells you which files you should copy to get a clone of the index as at snapshot time: try { IndexCommitPoint commit = snapshotter.snapshot(); Collection fileNames = commit.getFileNames(); <iterate over & copy files from fileNames> } finally { snapshotter.release(); } (credits to Lucene in Action 2°edition MEAP) Should I add this improvement to JIRA? I'll add the code examples; also you opened already HSEARCH-152 is that meant for this same purpose?

>> Does it somehow involve not having cluster changes (ie intra VM policy rather than inter VM?) I don't really understand what you're asking; I hope previous example contains an answer; it just means you don't need to lock the index to make a "hotcopy" to anywhere.

>>The explanation is correct but a bit cataclysmic. We open the reader *only* if there is an actual deletion for the given Directory provider. So in some systems we might very well almost never open a rw reader. Well yes it doesn't happen always, but not only for deletions: also all updates are split to to delete+insert work AFAIK. if you combine this with the open issue that the index is updated on every dirty entity (not checking if only the indexed fields are changed) this translates to lots of re- opening, twice for every transaction updating some entity... (still correct?)

>> We need to check how efficient the implementation has been done. If it a mechanism similar to IndexModifier, it's not worth it. Agree, I'll try to discover that. >>This is what happens by default already, one index directory by entity type. We could compute a flag at init time to know that (it's alrady in place actually) and use it to use a Term query >> rather than the full query if only one entity is present in the index. Let's open a JIRA issue Well my purpose is to avoid the need to switch from reader to writer, this also simplifies the reordering code were you split all work in two sequences (would be unneded); For deletion and eviction we should check to see if the workspace has an indexwriter or indexreader available and use whatever we have available; but this would complicate code and don't like it, so I really hope we could really avoid the use of indexreader for modification operations; as we found already a good solutions for mass eviction and when the flag says it's ok to delete by id, there's just one case left we should think about. Should I open the JIRA for the partial solution as far as we got currently?

>> Not if it's updated in a cluster, right? >> Plus seeing the contention lock we have experienced (on IndexReader) in the recent test case, I want to be sure it's actually faster than opening every time. Why not? don't you update by using delete+insert in cluster too?

I'll write performance tests just to be sure, but for past experience I really expect it to be much faster. Also note that all document analysis is done inside the IndexWriter, so not using it concurrently is a bottleneck when the analysis is expensive (think about PDFs in Blobs..); during analysis the files are locked to a single thread/transaction but the time spent in really changing files is just a fraction, and Lucene has it's own proper locks in those points. cheers, Sanne