Hello! I think it is a very good idéa to go for Lucene 2.3.1 as soon as possible, since this version fixed a corruption issue introduced in 2.3. Se snipped text below: ----------------------------------------------------------------------- Release 2.3.1 of Lucene Java is now available! This release contains fixes for serious bugs in 2.3.0 that could cause index corruptions in autoCommit=false mode or in cases where multiple threads are adding documents where some have term-vector enabled fields and some don't. The autoCommit option was added to IndexWriter with release 2.2.0. If not explicitly set to false, the IndexWriter runs in autoCommit=true mode. 2.3.1 does not contain any new features, API or file format changes, which makes it fully compatible to 2.3.0. We would like to encourage everyone who currently uses Lucene Java 2.3.0 to upgrade to 2.3.1 to prevent possible index corruptions! ------------------------------------------------------------------------ Best regards, Aleksander M. Stensby On Sun, 24 Feb 2008 22:10:34 +0100, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; wrote: -- Aleksander M. Stensby Senior Software Developer Integrasco A/S aleksander.stensby(a)integrasco.no Tlf.: +47 41 22 82 72

On Sun, 24 Feb 2008 22:10:34 +0100, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; wrote: Cool. Well, I started on it and think it is not too hard, but Emmanuel recommended to wait until 3.1. I will look into it again, but will be very busy until mit March. Another issue which I wold like to investigate further is: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-135 Maybe it is time to give Seam another go. The last time I tried it (about a year ago, seam-gen was not even around :( ) I found it not quite ready yet. --Hardy

Hi all, I've been doing a lot of work with Hibernate Search recently and have been pushing the lucene search side of it pretty hard. I've made several changes to improve performance and functionality and have created patches for these locally, but would like some feedback on my approach: 1) Similarity annotation (HSEARCH-136) I've created a patch for this which does the following: * Added a Similarity annotation which can be added at the class level. * Modified Workspace.java to set Similarities when creating an IndexWriter * In CacheableMultiReader changed visibility of subReaders to package private so other classes can use them * In ReaderProviderHelper added methods which can resolve underlying IndexReaders from a Searcher or Reader passed in * In DocumentBuilder added code to find the similarity annotation and store its implementation locally * In FullTextQueryImpl added code which works out which similarity to use when creating a Reader and changed all finally instances to use a common piece of code to close the reader. This seems to work well in my dev environment, and I'll be sending out a patch for 3.0.1 later today as I've already had some feedback from Emmanuel on this one. The solution is a lot simpler than the first patch I uploaded to Jira. 2) Explaining results This uses the new DOCUMENT_ID projection introduced in 3.0.1 to explain query results (we need this so the customer can understand their search results in the backoffice interface). I added an explain method to both implementations of FullTextQueryImpl which is only available by casting (e.g. no interface changes). I think explain() is probably a fairly advanced function which it's acceptable to access by casting. 3) Counting results In the current implementation we only want to perform one Lucene query per search (all projected). In order to get a resultcount and the results themselves it is currently necessary to invoke the Lucene query twice. I have made changes to allow this information to propagate through to the user whilst only making one search invokation, which has obvious effects on performance: * Created a class called SearchResultList which extends ArrayList. This has an extra property for setting and retrieving the total hitcount. * Added the method "List load(int hitCount, EntityInfo ... entityInfos)" onto the Loader interface. Keep the existing "List load(EntityInfo ... entityInfos)" method which can be stubbed by passing a dummy value if used. * Changed the loader implementations themselves to create a SearchResultList with the hitcount instead of an ArrayList. There are two ways of then accessing the hitcount: a) Casting the list to a SearchResultList b) Returning a SearchResultList instead of a List from the Loader interface and propagating this through c) Creating an interface from SearchResultList that extends List and then having a private implementation but doing as per (2) I'm particularly interested in some feedback on this as it's a big performance gain for applications that need the total hit count, and contains the most breaking changes of any of the things I've done. 4) Caching filter BitSets In order to fix the problem with readers there's going to need to be a way of accessing the underlying readers of a CacheableMultiReader in order to store the appropriate references to cache by. I think it's going to be better to either make the subReaders property public or to define an accessor for it. I've done this locally so I can hack up a working caching strategy based on a weakreference to the first reader, which works for my case but not the general case. Any feedback on these would be very useful. I've made the changes locally, but would like some confirmation about direction before I start spraying patches around. Cheers, Nick

Hi Emmanuel, On 26/02/2008, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; wrote: The Lucene javadoc says "Computing an explanation is as expensive as executing the query over the entire index.". http://lucene.apache.org/java/2_3_1/api/core/org/apache/lucene/search/Sea... which is why I didn't consider projecting this. If that's true the effort to project an explanation onto the results will increase exponentially with the number of hits. For this reason I think the method of accessing an Explanation that I proposed is reasonable (although not necessarily right). You are right, and I don't need to make any alterations. I've worked out what the problem we encountered was that made me think this was a problem. It took a bit of digging around the source to work out what we'd done wrong, and perhaps it might be useful to include in an FAQ or the documentation. If you make the calls in this order: query.getResultSize(); // Hits retrieved, hitcount cached and returned query.list(); // Hits retrieved then the query gets run twice as resultCount is cached in FullTextQueryImpl but the Hits object is not. A subtle effect, but when you're using something like JSF you're not always sure in which order the properties of your underlying beans are retrieved during the render cycle. This was the cause of our double querying behaviour. Cheers, Nick

On 27/02/2008, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; wrote: Yes, our use case is that our customer will test a search on their page and then say "why does this particular result rank higher than the rest". We then need to provide an explanation (preferably without them having to ring us and ask). The think time should be fairly short and it's only for a backoffice system so it's not 100% critical it's correct, although it would be nice if it were. It's probably not necessary provided the information about correct usage is readily available. We have done pretty much that and created a bean which when asked for the hitcount runs the query and caches both the results and the total hitcount, which alleviates the problem for us. Nick