[hibernate-issues] [Hibernate-JIRA] Commented: (HSEARCH-1019) Profile mass indexer to reduce number of index commits

[Sanne Grinovero (JIRA)]

    [ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=45006#comment-45006 ] 

Sanne Grinovero commented on HSEARCH-1019:
------------------------------------------

Laurent,
thank you so much for the test project. I've used it to create the patch, together with org.hibernate.search.test.batchindexing.IndexingGeneratedCorpusTest from the testsuite and some real-world project I have around.

For the fun, here are my numbers after changing ENTITY_COUNT to 30000:

Having the FSDirectory mounted on tmpfs:
Search 4.0.0.Final with Lucene 3.4:
INFO: Indexation last for 31468 ms.

Search 4.0.0.Final with Lucene 3.3:
INFO: Indexation last for 24496 ms.

Search 4.1-SNAPSHOT with Lucene 3.5:
INFO: Indexation last for 4647 ms.

On SSD:
Search 4.0.0.Final with Lucene 3.3:
INFO: Indexation last for 29762 ms.

Search 4.0.0.Final with Lucene 3.4:
INFO: Indexation last for 117897 ms.

Search 4.1-SNAPSHOT with Lucene 3.5:
INFO: Indexation last for 4341 ms.

Search 4.1-SNAPSHOT with Lucene 3.5, ENTITY_COUNT to 100000:
INFO: Indexation last for 10127 ms.

> Profile mass indexer to reduce number of index commits
> ------------------------------------------------------
>
>                 Key: HSEARCH-1019
>                 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-1019
>             Project: Hibernate Search
>          Issue Type: Improvement
>          Components: massindexer
>    Affects Versions: 4.0.0.Final
>            Reporter: Hardy Ferentschik
>            Assignee: Sanne Grinovero
>            Priority: Blocker
>             Fix For: 4.1.0.Alpha1
>
>
> We need to profile the new mass indexer design and try to reduce the number of index commits. Performance of the filesystem based directory declined to a bug fix in Lucene (https://issues.apache.org/jira/browse/LUCENE-3418). Trying to reduce the amount of index commits (and hence fsync calls) might alleviate the performance loss. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira