[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-1019?pa... ] Sanne Grinovero updated HSEARCH-1019: ------------------------------------- Suitable for new contributors: [Yes, likely] Affects Version/s: (was: 4.0.0.Final) Fix Version/s: 4.1 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-1019?pa... ] Hardy Ferentschik commented on HSEARCH-1019: -------------------------------------------- See also https://forum.hibernate.org/viewtopic.php?f=9&t=1013883 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-1019?pa... ] Guillaume Smet commented on HSEARCH-1019: ----------------------------------------- Hi Emmanuel and Hardy, I work with Laurent who reported the problem. Moving from Lucene 3.3 to 3.4 really makes the difference between having the MassIndexer usable or unusable for us (with our 100k documents). We forced the use of Lucene 3.3 as a workaround for the moment but I'm pretty sure it's going to hit a lot of applications once people start to move to Hibernate Search 4. IMHO, we should have the ability to configure a number of documents between each commit on the mass indexer. Note: we can try snapshots on our application if you need us to test something. -- Guillaume -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-1019?pa... ] Emmanuel Bernard updated HSEARCH-1019: -------------------------------------- Fix Version/s: (was: 4.1) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-1019?pa... ] Sanne Grinovero commented on HSEARCH-1019: ------------------------------------------ brief update: this is being more complex than expected, we have indeed some commit operations which should be omitted, but not so much and solving it I've improved performance by only 40%; there is a mayor regression on the cost of initializing a new Analyzer in Lucene as it's doing some class consistency check via reflection; usually these check are guarded by the assert keyword but not in this case. I'm looking if we can work around it at least in the MassIndexer, or if it's no longer the case in Lucene 3.5 and if we need to patch Lucene. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-1019?pa... ] Sanne Grinovero updated HSEARCH-1019: ------------------------------------- Fix Version/s: (was: 4.1) 4.1.0.Alpha1 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-1019?pa... ] Sanne Grinovero commented on HSEARCH-1019: ------------------------------------------ Laurent, thank you so much for the test project. I've used it to create the patch, together with org.hibernate.search.test.batchindexing.IndexingGeneratedCorpusTest from the testsuite and some real-world project I have around. For the fun, here are my numbers after changing ENTITY_COUNT to 30000: Having the FSDirectory mounted on tmpfs: Search 4.0.0.Final with Lucene 3.4: INFO: Indexation last for 31468 ms. Search 4.0.0.Final with Lucene 3.3: INFO: Indexation last for 24496 ms. Search 4.1-SNAPSHOT with Lucene 3.5: INFO: Indexation last for 4647 ms. On SSD: Search 4.0.0.Final with Lucene 3.3: INFO: Indexation last for 29762 ms. Search 4.0.0.Final with Lucene 3.4: INFO: Indexation last for 117897 ms. Search 4.1-SNAPSHOT with Lucene 3.5: INFO: Indexation last for 4341 ms. Search 4.1-SNAPSHOT with Lucene 3.5, ENTITY_COUNT to 100000: INFO: Indexation last for 10127 ms. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-1019?pa... ] Sanne Grinovero commented on HSEARCH-1019: ------------------------------------------ Hi Guillaume, yes I've deployed a fresh 4.1.0-SNAPSHOT to the Maven repository (after Hardy merged this in master) The numbers above refer to an ext4 partition using default settings - but if you want to try to experiment with the write barriers, these are the "tuned" settings I usually use on the SSD drive on the laptop: rw,noatime,commit=60,max_batch_time=0,barrier=0,data=writeback These should provide better performance (that's why I use them for coding), but I don't suggest using them in a production system: I'm not a filesystems expert, have just been playing with them especially because of the write barriers you mention. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://hibernate.onjira.com/browse/HSEARCH-1019?page=com.atlassian.jira.... ] Laurent Almeras commented on HSEARCH-1019: ------------------------------------------ I try your fix this night. Indexation lasts 12 seconds now, so I also notice better performance than with 4.0 and 3.x with your patch. It's a very good job ! -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira