<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0">
<base href="https://hibernate.atlassian.net">
<title>Message Title</title>
</head>
<body class="jira" style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 1.429">
<table id="background-table" cellpadding="0" cellspacing="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0; background-color: #f5f5f5; border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0" bgcolor="#f5f5f5">
<!-- header here -->
<tbody>
<tr>
<td id="header-pattern-container" style="padding: 0; border-collapse: collapse; padding: 10px 20px">
<table id="header-pattern" cellspacing="0" cellpadding="0" border="0" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0">
<tbody>
<tr>
<td id="header-avatar-image-container" valign="top" style="padding: 0; border-collapse: collapse; vertical-align: top; width: 32px; padding-right: 8px" width="32"> <img id="header-avatar-image" class="image_fix" src="https://secure.gravatar.com/avatar/2a8bdd4ffd282b7185c74b52ab452617?d=https%3A%2F%2Favatar-management--avatars.us-west-2.prod.public.atl-paas.net%2Finitials%2FYR-6.png&size=48&s=48" height="32" width="32" border="0" style="border-radius: 3px; vertical-align: top"> </td>
<td id="header-text-container" valign="middle" style="padding: 0; border-collapse: collapse; vertical-align: middle; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 1px"> <a class="user-hover" rel="yrodiere" style="color:#6c797f;; color: #3b73af; text-decoration: none" id="email_yrodiere" href="https://hibernate.atlassian.net/secure/ViewProfile.jspa?accountId=557058%3A58fa1ced-171a-4c00-97e8-5d70d442cc4b"> Yoann Rodière </a> <strong>created</strong> an issue </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td id="email-content-container" style="padding: 0; border-collapse: collapse; padding: 0 20px">
<table id="email-content-table" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0; border-spacing: 0; border-collapse: separate">
<tbody>
<tr>
<!-- there needs to be content in the cell for it to render in some clients -->
<td class="email-content-rounded-top mobile-expand" style="padding: 0; border-collapse: collapse; color: #ffffff; padding: 0 15px 0 16px; height: 15px; background-color: #ffffff; border-left: 1px solid #cccccc; border-top: 1px solid #cccccc; border-right: 1px solid #cccccc; border-bottom: 0; border-top-right-radius: 5px; border-top-left-radius: 5px; height: 10px; line-height: 10px; padding: 0 15px 0 16px; mso-line-height-rule: exactly" height="10" bgcolor="#ffffff"> </td>
</tr>
<tr>
<td class="email-content-main mobile-expand " style="padding: 0; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff" bgcolor="#ffffff">
<table class="page-title-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0">
<tbody>
<tr>
<td class="page-title-pattern-first-line " style="padding: 0; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; padding-top: 10px"> <a href="https://hibernate.atlassian.net/browse/HSEARCH?atlOrigin=eyJpIjoiMDU4NGMyYTlhY2JhNDA0MDgxNTI4ZjM1NjZiOWIxOTMiLCJwIjoiaiJ9" style="color: #3b73af; text-decoration: none">Hibernate Search</a> / <a href="https://hibernate.atlassian.net/browse/HSEARCH-3822?atlOrigin=eyJpIjoiMDU4NGMyYTlhY2JhNDA0MDgxNTI4ZjM1NjZiOWIxOTMiLCJwIjoiaiJ9" style="color: #3b73af; text-decoration: none"><img src="cid:jira-generated-image-avatar-56f00372-de2f-49a8-b049-2310589f1e8a" height="16" width="16" border="0" align="absmiddle" alt="Task" style="vertical-align: text-bottom"></a> <a href="https://hibernate.atlassian.net/browse/HSEARCH-3822?atlOrigin=eyJpIjoiMDU4NGMyYTlhY2JhNDA0MDgxNTI4ZjM1NjZiOWIxOTMiLCJwIjoiaiJ9" style="color: #3b73af; text-decoration: none">HSEARCH-3822</a> </td>
</tr>
<tr>
<td style="vertical-align: top;; padding: 0; border-collapse: collapse; padding-right: 5px; font-size: 20px; line-height: 30px; mso-line-height-rule: exactly" class="page-title-pattern-header-container"> <span class="page-title-pattern-header" style="font-family: Arial, sans-serif; padding: 0; font-size: 20px; line-height: 30px; mso-text-raise: 2px; mso-line-height-rule: exactly; vertical-align: middle"> <a href="https://hibernate.atlassian.net/browse/HSEARCH-3822?atlOrigin=eyJpIjoiMDU4NGMyYTlhY2JhNDA0MDgxNTI4ZjM1NjZiOWIxOTMiLCJwIjoiaiJ9" style="color: #3b73af; text-decoration: none">Restore support for concurrent Lucene flushes - at least during mass indexing</a> </span> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td class="email-content-main mobile-expand wrapper-special-margin" style="padding: 0; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff; padding-top: 10px; padding-bottom: 5px" bgcolor="#ffffff">
<table class="keyvalue-table" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0">
<tbody>
<tr>
<th style="color: #707070; font: normal 14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0">Issue Type:</th>
<td class="has-icon" style="padding: 0; border-collapse: collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; vertical-align: top"> <img src="cid:jira-generated-image-avatar-56f00372-de2f-49a8-b049-2310589f1e8a" height="16" width="16" border="0" align="absmiddle" alt="Task" style="vertical-align: text-bottom"> Task </td>
</tr>
<tr>
<th style="color: #707070; font: normal 14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0">Assignee:</th>
<td style="padding: 0; border-collapse: collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; vertical-align: top"> <a class="user-hover" rel="yrodiere" style="color:#6c797f;; color: #3b73af; text-decoration: none" id="email_yrodiere" href="https://hibernate.atlassian.net/secure/ViewProfile.jspa?accountId=557058%3A58fa1ced-171a-4c00-97e8-5d70d442cc4b"> Yoann Rodière </a> </td>
</tr>
<tr>
<th style="color: #707070; font: normal 14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0">Components:</th>
<td style="padding: 0; border-collapse: collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; vertical-align: top"> backend-lucene </td>
</tr>
<tr>
<th style="color: #707070; font: normal 14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0">Created:</th>
<td style="padding: 0; border-collapse: collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; vertical-align: top"> 03/Feb/2020 09:36 AM </td>
</tr>
<tr>
<th style="color: #707070; font: normal 14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0">Fix Versions:</th>
<td style="padding: 0; border-collapse: collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; vertical-align: top"> 6.0.0.Beta-backlog-high-priority </td>
</tr>
<tr>
<th style="color: #707070; font: normal 14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0">Priority:</th>
<td class="has-icon" style="padding: 0; border-collapse: collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; vertical-align: top"> <img src="cid:jira-generated-image-static-major-40c4dec0-79b4-41b5-bb62-4e60cb8478e0" height="16" width="16" border="0" align="absmiddle" alt="Major" style="vertical-align: text-bottom"> Major </td>
</tr>
<tr>
<th style="color: #707070; font: normal 14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0">Reporter:</th>
<td style="padding: 0; border-collapse: collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; vertical-align: top"> <a class="user-hover" rel="yrodiere" style="color:#6c797f;; color: #3b73af; text-decoration: none" id="email_yrodiere" href="https://hibernate.atlassian.net/secure/ViewProfile.jspa?accountId=557058%3A58fa1ced-171a-4c00-97e8-5d70d442cc4b"> Yoann Rodière </a> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td class="email-content-main mobile-expand issue-description-container" style="padding: 0; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff; padding-top: 5px; padding-bottom: 10px" bgcolor="#ffffff">
<table class="text-paragraph-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 2px">
<tbody>
<tr>
<td class="text-paragraph-pattern-container mobile-resize-text " style="padding: 0; border-collapse: collapse; padding: 0 0 10px"> <p style="margin-top:0;margin-bottom:10px;; margin: 10px 0 0; margin-top: 0">In Search 5, we used to execute calls to the IndexWriter from multiple threads when mass indexing. As explained <a href="http://blog.mikemccandless.com/2011/05/265-indexing-speedup-with-lucenes.html" class="external-link" rel="nofollow noreferrer" style="color: #3b73af; text-decoration: none">here</a>, Lucene does perform better when using multiple threads to index, at least if the hardware supports concurrent writes, thanks to concurrent flushing.</p> <p style="margin-top:0;margin-bottom:10px;; margin: 10px 0 0">In Search 6, we execute all calls to the IndexWriter from the same thread, with one thread assigned to each index manager. As a result, mass indexing performance is likely to degrade compared to Search 5 on hardware supporting concurrent writes.</p> <p style="margin-top:0;margin-bottom:10px;; margin: 10px 0 0">We should explore the following solutions, and pick whatever works best.</p> <p style="margin-top:0;margin-bottom:10px;; margin: 10px 0 0">All solutions below rely on the same concept of <a href="https://en.wikipedia.org/wiki/Work_stealing" class="external-link" rel="nofollow noreferrer" style="color: #3b73af; text-decoration: none">work stealing</a>: a thread that has nothing to do will try to help out the thread of another index. The different lies only in which thread will try to help out.</p> <p style="margin-top:0;margin-bottom:10px;; margin: 10px 0 0">All solutions will require some changes in the <tt>LuceneWriteWorkProcessor</tt>, which is currently thread-unsafe.</p> <h3 style="margin: 10px 0 0; font-size: 16px; line-height: 1.563; margin: 30px 0 0"><a name="Solution1%3Amassindexingthreadhelpingwithindexing" style="color: #3b73af; text-decoration: none"></a>Solution 1 : mass indexing thread helping with indexing</h3> <p style="margin-top:0;margin-bottom:10px;; margin: 10px 0 0">Restore the old behavior of executing writes from the mass indexing thread directly. This may prove difficult given the current architecture.</p> <p style="margin-top:0;margin-bottom:10px;; margin: 10px 0 0">One possible solution: return custom objects instead of CompletableFuture from <tt>org.hibernate.search.mapper.pojo.work.spi.PojoIndexer#add</tt>, and instead of calling <tt>future.join()</tt> from the massIndexer thread, call <tt>theReturnedObject.executeIfNotAlreadyExecuted()</tt>, which will execute the work unless it's already been executed by the per-index thread.</p> <h3 style="margin: 10px 0 0; font-size: 16px; line-height: 1.563; margin: 30px 0 0"><a name="Solution2%3Aotherperindexthreadshelpingwithindexing" style="color: #3b73af; text-decoration: none"></a>Solution 2: other per-index threads helping with indexing</h3> <p style="margin-top:0;margin-bottom:10px;; margin: 10px 0 0">Allow threads assigned to one index to steal the work from another index when they become idle.</p> <p style="margin-top:0;margin-bottom:10px;; margin: 10px 0 0">As a result, even "traditional" (non-massindexer) worksets will no longer be executed in the order they were submitted, but I do not think it will have much impact since we usually only submit one workset per transaction, and the order worksets are submitted in is not guaranteed to match the order of transactions anyway. If I'm wrong and fixing the order proves necessary, we can have a look at solving <span class="jira-issue-macro" data-jira-key="HSEARCH-3664"> <a href="https://hibernate.atlassian.net/browse/HSEARCH-3664" class="jira-issue-macro-key issue-link" title="Remove the requirement for works from workplans to be executed in order" style="color: #3b73af; text-decoration: none"> <img class="icon" src="https://hibernate.atlassian.net/secure/viewavatar?size=medium&avatarId=12150&avatarType=issuetype"> HSEARCH-3664 </a> <span class="aui-lozenge aui-lozenge-subtle aui-lozenge-complete jira-macro-single-issue-export-pdf" style="background: #cccccc; border: 1px solid #cccccc; border-radius: 3px; color: #333333; display: inline-block; font-size: 11px; font-weight: bold; line-height: 99%; margin: 0; padding: 2px 5px; text-align: center; text-decoration: none; text-transform: uppercase; background-color: #ffffff; border-color: #cccccc; color: #333333" bgcolor="#ffffff">Open</span> </span> first.</p> <p style="margin-top:0;margin-bottom:10px;; margin: 10px 0 0">This solution could get us decent performance improvement for all writes, not just mass indexer writes, as soon as there are more than one index and only some of the indexes are being written to. Which is typically the case when mass indexing: by default we only mass index one type (= one index) at a time, but there are generally multiple types (= multiple indexes) to mass index.</p> <p style="margin-top:0;margin-bottom:10px;; margin: 10px 0 0">We may even introduce settings to define a minimum and maximum number of threads, so that a Hibernate Search instance with 100 indexes still only uses 10 threads (max = 10), for example, with each thread sequentially working on each index, and a Hibernate Search instance with a single index still uses 2 threads (min = 2) to benefit from concurrent flushing.</p> <p style="margin-top:0;margin-bottom:10px;; margin: 10px 0 0">For Infinispan, where we usually have only one index per cache (so one index per Hibernate Search instance), we could provide an SPI that allows Infinispan to define its own thread pool shared across all caches. That way, Infinispan would benefit from concurrent flushing.</p> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td class="email-content-main mobile-expand " style="padding: 0; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff" bgcolor="#ffffff"> <script type="application/ld+json">
{
"@context": "http://schema.org",
"@type": "EmailMessage",
"description": "View Issue",
"potentialAction": {
"@type": "ViewAction",
"target": "https://hibernate.atlassian.net/browse/HSEARCH-3822?inbox=true&",
"name": "View Issue"
},
"publisher": {
"@type": "Organization",
"name": "Atlassian",
"url": "https://www.atlassian.com"
}
}
</script>
<table id="actions-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 1px">
<tbody>
<tr>
<td id="actions-pattern-container" valign="middle" style="padding: 0; border-collapse: collapse; padding: 10px 0 10px 24px; vertical-align: middle; padding-left: 0">
<table align="left" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0">
<tbody>
<tr>
<td class="actions-pattern-action-icon-container" style="padding: 0; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 0; vertical-align: middle"> <a href="https://hibernate.atlassian.net/browse/HSEARCH-3822#add-comment?atlOrigin=eyJpIjoiMDU4NGMyYTlhY2JhNDA0MDgxNTI4ZjM1NjZiOWIxOTMiLCJwIjoiaiJ9" target="_blank" title="Add Comment" style="color: #3b73af; text-decoration: none"> <img class="actions-pattern-action-icon-image" src="cid:jira-generated-image-static-comment-icon-d0a2d3bb-2dd2-4e45-b8d7-7915d23fc34b" alt="Add Comment" title="Add Comment" height="16" width="16" border="0" style="vertical-align: middle"> </a> </td>
<td class="actions-pattern-action-text-container" style="padding: 0; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 4px; padding-left: 5px"> <a href="https://hibernate.atlassian.net/browse/HSEARCH-3822#add-comment?atlOrigin=eyJpIjoiMDU4NGMyYTlhY2JhNDA0MDgxNTI4ZjM1NjZiOWIxOTMiLCJwIjoiaiJ9" target="_blank" title="Add Comment" style="color: #3b73af; text-decoration: none">Add Comment</a> </td>
</tr>
</tbody>
</table> </td>
</tr>
</tbody>
</table> </td>
</tr>
<!-- there needs to be content in the cell for it to render in some clients -->
<tr>
<td class="email-content-rounded-bottom mobile-expand" style="padding: 0; border-collapse: collapse; color: #ffffff; padding: 0 15px 0 16px; height: 5px; line-height: 5px; background-color: #ffffff; border-top: 0; border-left: 1px solid #cccccc; border-bottom: 1px solid #cccccc; border-right: 1px solid #cccccc; border-bottom-right-radius: 5px; border-bottom-left-radius: 5px; mso-line-height-rule: exactly" height="5" bgcolor="#ffffff"> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td id="footer-pattern" style="padding: 0; border-collapse: collapse; padding: 12px 20px">
<table id="footer-pattern-container" cellspacing="0" cellpadding="0" border="0" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0">
<tbody>
<tr>
<td id="footer-pattern-mobile-separated-links" class="mobile-resize-text" width="100%" colspan="2" style="padding: 0; border-collapse: collapse; color: #999999; font-size: 12px; line-height: 18px; font-family: Arial, sans-serif; mso-line-height-rule: exactly; mso-text-raise: 2px"> Get Jira notifications on your phone! Download the Jira Cloud app for <a href="https://play.google.com/store/apps/details?id=com.atlassian.android.jira.core&referrer=utm_source%3DNotificationLink%26utm_medium%3DEmail" style="color: #3b73af; text-decoration: none">Android</a> or <a href="https://itunes.apple.com/app/apple-store/id1006972087?pt=696495&ct=EmailNotificationLink&mt=8" style="color: #3b73af; text-decoration: none">iOS</a>
<hr> </td>
</tr>
<tr>
<td id="footer-pattern-text" class="mobile-resize-text" width="100%" style="padding: 0; border-collapse: collapse; color: #999999; font-size: 12px; line-height: 18px; font-family: Arial, sans-serif; mso-line-height-rule: exactly; mso-text-raise: 2px"> This message was sent by Atlassian Jira <span id="footer-build-information">(v1001.0.0-SNAPSHOT#100118-<span title="98956caf0dc5f7e583f56885fe315b977c6f0b6e" data-commit-id="98956caf0dc5f7e583f56885fe315b977c6f0b6e}">sha1:98956ca</span>)</span> </td>
<td id="footer-pattern-logo-desktop-container" valign="top" style="padding: 0; border-collapse: collapse; padding-left: 20px; vertical-align: top">
<table style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0">
<tbody>
<tr>
<td id="footer-pattern-logo-desktop-padding" style="padding: 0; border-collapse: collapse; padding-top: 3px; opacity: 0.150"> <img id="footer-pattern-logo-desktop" src="cid:jira-generated-image-static-footer-desktop-logo-874db987-0464-4cb2-8ecb-31df3de49b08" alt="Atlassian logo" title="Atlassian logo" width="192" height="24" class="image_fix"> </td>
</tr>
</tbody>
</table> </td>
</tr>
</tbody>
</table> </td>
</tr>
</tbody>
</table>
<img border="0" width="1" height="1" alt="" src="http://atlassian.et.e.sparkpost.com/q/-1rJaV4aR7npcjm-faJkcw~~/AAAAAQA~/RgRgGY2jPlcLYXRsYXNzaWFudXNCCgAAI1o4XmMIgsdSIGhpYmVybmF0ZS1pc3N1ZXNAbGlzdHMuamJvc3Mub3JnWAQAAAAE">
</body>
</html>