[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-194?pag... ] Sanne Grinovero commented on HSEARCH-194: ----------------------------------------- it looks like this is confirmed: http://forum.hibernate.org/viewtopic.php?p=2386280 -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://opensource.atlassian.com/projects/hibernate/secure/Administrators.... - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-194?pag... ] Emmanuel Bernard commented on HSEARCH-194: ------------------------------------------ I've spent a couple of hours to test all this. My test runs 100 threads in parallel. Each thread querying 'name:maria OR description:long<i>' (100 different queries), accessing the first 100 matching documents, retrieving one field and putting the result in a List<String>. The index is 5 Mb on disk. Tests do not warm up by preopening Readers. Plain Lucene 81423 84322 65698 86481 122023 = 87989.4 Hibernate Search regular settings 151810 172000 162000 157000 154000 = 159362 Hibernate Search with DumpReaderStrategy (*) 120000 148000 137000 133000 148000 = 137200 55783 84179 53086 74069 97635 = Hibernate Search without the class filter clause (**) 107000 103000 132000 86000 115000 = 108600 (*) The DumpStrategy stores in a static file the reader (prepare it eagerly) and reuses it: there is no lock latency (**) By default, Hibernate Search add a Boolean clause to all queries to filter to the targeted classes (and their subclasses) unless you do not specify a class type in createFullTextQuery. This test, explicitly remove the filter clause. So if we use Hibernate Search regular as a base line HSearch dump is 13.9% faster HSearch without clause is 31.8% faster Plain Lucene is 44.7% faster I could not reproduce the 30+ seconds per query as described in the post. Potential optimizations: - remove the unnecessary clause when we know that there is only one class per index (we need to add some more metadata): that is fairly easy - rework the SharedReaderProvider and use some of Sanne's ideas to warm in the background. As it is today, the SharedReaderProvider is very slow to warm up in this particular case (all queries arrive at the same time and we create a new Reader provider most of the time as we don't wait for one to be created today: reusability is limited) Still todo: I still cannot completely explain why queries ran in parallel take so long. Contrary to Pure Lucene, the cache does not seem to kick in very well (even if SharedReaderProvider is artificially prewarmed. We should redo the tests with a pre warm up. The very early tests I did show: Lucene 7500, HSearch no class clause + pure Reader reuse 40000, Regular HSearch 120000 We have enhancements to do :) Note the test shown in the forum is unfair as it compares Lucene *not* accessing the documents nor the fields to Hibernate Search applying the field access logic. If you want to avoid accessing fields, you can use getResultSize() (wo calling list()). -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://opensource.atlassian.com/projects/hibernate/secure/Administrators.... - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-194?pag... ] Emmanuel Bernard commented on HSEARCH-194: ------------------------------------------ OK I think I found the final bit of difference between Pure Lucene and Hibernate Search: When receiving 100 requests at the same time an IndexReader will reach it's scalability limit (hence the monitor contentions you have observed). Basically for such high request/sec, it is better to provide a pool of IndexReader/IndexSearcher. The sweet pot depends on the index size as warming up the IndexReader is also an important factor. I guess on top of the optimization proposals (from my last post) we should add a pooling mechanism for IndexReaders. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://opensource.atlassian.com/projects/hibernate/secure/Administrators.... - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-194?pag... ] Emmanuel Bernard commented on HSEARCH-194: ------------------------------------------ About a new ReaderProvider, let's start a discussion on the dev mailing list. With some design ideas. The scalability issues I saw compared to Lucene were due to the fact that my Lucene test did not reuse the IndexSearcher. It had one searcher per Thread. Which made it much faster in the end on my duo-core and on this extreme test (100 // queries) The DirectoryProvider itself is locked when you update data, not when you read. Pure read (ReaderProvider) are not locked in that regard. But for the structure keeping the cached IndexReaders for a given DirectoryProvider, you need to have a lock around. "//needed for same problem as the double-checked locking" is basically ensuring a memory barrier to make sure the instructions are not shuffled by the VM. I will commit my test, probably today or tomorrow. You can commit the experiment on the test directory and we can move it eventually. From he tests I've done, the isCurrent was not the issue really. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://opensource.atlassian.com/projects/hibernate/secure/Administrators.... - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-194?pag... ] Emmanuel Bernard resolved HSEARCH-194. -------------------------------------- Resolution: Fixed Fix Version/s: 3.1.0.Beta1 -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://opensource.atlassian.com/projects/hibernate/secure/Administrators.... - For more information on JIRA, see: http://www.atlassian.com/software/jira