[hibernate-issues] [JIRA] (HSEARCH-3871) Always enable delayed commits in the Lucene backend

[Yoann Rodière (JIRA)]

Yoann Rodière ( https://hibernate.atlassian.net/secure/ViewProfile.jspa?accountId=557058%3A58fa1ced-171a-4c00-97e8-5d70d442cc4b ) *created* an issue

Hibernate Search ( https://hibernate.atlassian.net/browse/HSEARCH?atlOrigin=eyJpIjoiY2RjZjQ3YzEzN2MxNDM3Mzg4MWUzNDBlOTljM2E1ZDUiLCJwIjoiaiJ9 ) / Improvement ( https://hibernate.atlassian.net/browse/HSEARCH-3871?atlOrigin=eyJpIjoiY2RjZjQ3YzEzN2MxNDM3Mzg4MWUzNDBlOTljM2E1ZDUiLCJwIjoiaiJ9 ) HSEARCH-3871 ( https://hibernate.atlassian.net/browse/HSEARCH-3871?atlOrigin=eyJpIjoiY2RjZjQ3YzEzN2MxNDM3Mzg4MWUzNDBlOTljM2E1ZDUiLCJwIjoiaiJ9 ) Always enable delayed commits in the Lucene backend ( https://hibernate.atlassian.net/browse/HSEARCH-3871?atlOrigin=eyJpIjoiY2RjZjQ3YzEzN2MxNDM3Mzg4MWUzNDBlOTljM2E1ZDUiLCJwIjoiaiJ9 )

Issue Type: Improvement Assignee: Yoann Rodière ( https://hibernate.atlassian.net/secure/ViewProfile.jspa?accountId=557058%3A58fa1ced-171a-4c00-97e8-5d70d442cc4b ) Components: backend-lucene Created: 25/Mar/2020 09:50 AM Fix Versions: 6.0.0.Beta-backlog-high-priority Priority: Major Reporter: Yoann Rodière ( https://hibernate.atlassian.net/secure/ViewProfile.jspa?accountId=557058%3A58fa1ced-171a-4c00-97e8-5d70d442cc4b )

Let's start by saying that delayed commits are nothing to be afraid of. When they are enabled, write operations that require an immediate commit (e.g. automatic indexing by default) are still committed immediately. The delayed commits are only effective when some write operations do *not* perform an immediate commit. In that case, the user already decided to take the chance of losing data in the event of a crash, so waiting 1 second before the commit will not change much to the data safety.
It does change much when it comes to performance, though. Instead of committing e.g. every 100 operations, we commit e.g. every second: maybe no additional operation will be performed by then, in which case which case performance is the same, but maybe thousands of write operations will have be performed by then, in which case we just reduced commits by a factor 10... and probably increased throughput by the same factor.

With that in mind... *not* enabling delayed commits is becoming a problem. Early numbers from the performance tests on HSEARCH-3822 ( https://hibernate.atlassian.net/browse/HSEARCH-3822 ) In Progress show that any improvement to the performance when delayed commits are enabled will probably lead to massive performance regressions when delayed commits are *not* enabled. We're talking about doubling the throughput when delayed commit are enabled, while dividing it by a factor 10 when delayed commits are disabled.

The reason for the regressions is that we have to commit after each batch of writes when delayed commits are disabled, and the improvements are about performing multiple, but smaller batche of writes in parallel. So when delayed commits are disabled, the "improvements" actually increase the amount of commits. When delayed commits are enabled, they do not.

( https://hibernate.atlassian.net/browse/HSEARCH-3871#add-comment?atlOrigin=eyJpIjoiY2RjZjQ3YzEzN2MxNDM3Mzg4MWUzNDBlOTljM2E1ZDUiLCJwIjoiaiJ9 ) Add Comment ( https://hibernate.atlassian.net/browse/HSEARCH-3871#add-comment?atlOrigin=eyJpIjoiY2RjZjQ3YzEzN2MxNDM3Mzg4MWUzNDBlOTljM2E1ZDUiLCJwIjoiaiJ9 )

Get Jira notifications on your phone! Download the Jira Cloud app for Android ( https://play.google.com/store/apps/details?id=com.atlassian.android.jira.core&referrer=utm_source%3DNotificationLink%26utm_medium%3DEmail ) or iOS ( https://itunes.apple.com/app/apple-store/id1006972087?pt=696495&ct=EmailNotificationLink&mt=8 ) This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100122- sha1:092c6a7 )
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/hibernate-issues/attachments/20200325/a30c5f40/attachment.html