<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0">
<base href="https://hibernate.atlassian.net">
<title>Message Title</title>
</head>
<body class="jira" style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 1.429">
<table id="background-table" cellpadding="0" cellspacing="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0; background-color: #f5f5f5; border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0" bgcolor="#f5f5f5">
<!-- header here -->
<tbody>
<tr>
<td id="header-pattern-container" style="padding: 0; border-collapse: collapse; padding: 10px 20px">
<table id="header-pattern" cellspacing="0" cellpadding="0" border="0" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0">
<tbody>
<tr>
<td id="header-avatar-image-container" valign="top" style="padding: 0; border-collapse: collapse; vertical-align: top; width: 32px; padding-right: 8px" width="32"> <img id="header-avatar-image" class="image_fix" src="https://secure.gravatar.com/avatar/2a8bdd4ffd282b7185c74b52ab452617?d=https%3A%2F%2Favatar-management--avatars.us-west-2.prod.public.atl-paas.net%2Finitials%2FYR-6.png&size=48&s=48" height="32" width="32" border="0" style="border-radius: 3px; vertical-align: top"> </td>
<td id="header-text-container" valign="middle" style="padding: 0; border-collapse: collapse; vertical-align: middle; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 1px"> <a class="user-hover" rel="yrodiere" style="color:#6c797f;; color: #3b73af; text-decoration: none" id="email_yrodiere" href="https://hibernate.atlassian.net/secure/ViewProfile.jspa?accountId=557058%3A58fa1ced-171a-4c00-97e8-5d70d442cc4b"> Yoann Rodière </a> <strong>updated</strong> an issue </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td id="email-content-container" style="padding: 0; border-collapse: collapse; padding: 0 20px">
<table id="email-content-table" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0; border-spacing: 0; border-collapse: separate">
<tbody>
<tr>
<!-- there needs to be content in the cell for it to render in some clients -->
<td class="email-content-rounded-top mobile-expand" style="padding: 0; border-collapse: collapse; color: #ffffff; padding: 0 15px 0 16px; height: 15px; background-color: #ffffff; border-left: 1px solid #cccccc; border-top: 1px solid #cccccc; border-right: 1px solid #cccccc; border-bottom: 0; border-top-right-radius: 5px; border-top-left-radius: 5px; height: 10px; line-height: 10px; padding: 0 15px 0 16px; mso-line-height-rule: exactly" height="10" bgcolor="#ffffff"> </td>
</tr>
<tr>
<td class="email-content-main mobile-expand " style="padding: 0; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff" bgcolor="#ffffff">
<table class="page-title-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0">
<tbody>
<tr>
<td class="page-title-pattern-first-line " style="padding: 0; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; padding-top: 10px"> <a href="https://hibernate.atlassian.net/browse/HSEARCH?atlOrigin=eyJpIjoiNWE1MjMyNzE4MjMwNDA2YzgwNzk1ZmRhNTMwOWUxYTAiLCJwIjoiaiJ9" style="color: #3b73af; text-decoration: none">Hibernate Search</a> / <a href="https://hibernate.atlassian.net/browse/HSEARCH-3822?atlOrigin=eyJpIjoiNWE1MjMyNzE4MjMwNDA2YzgwNzk1ZmRhNTMwOWUxYTAiLCJwIjoiaiJ9" style="color: #3b73af; text-decoration: none"><img src="cid:jira-generated-image-avatar-4b80842c-89e4-4b55-9040-e8fa7aec8ea6" height="16" width="16" border="0" align="absmiddle" alt="Task" style="vertical-align: text-bottom"></a> <a href="https://hibernate.atlassian.net/browse/HSEARCH-3822?atlOrigin=eyJpIjoiNWE1MjMyNzE4MjMwNDA2YzgwNzk1ZmRhNTMwOWUxYTAiLCJwIjoiaiJ9" style="color: #3b73af; text-decoration: none">HSEARCH-3822</a> </td>
</tr>
<tr>
<td style="vertical-align: top;; padding: 0; border-collapse: collapse; padding-right: 5px; font-size: 20px; line-height: 30px; mso-line-height-rule: exactly" class="page-title-pattern-header-container"> <span class="page-title-pattern-header" style="font-family: Arial, sans-serif; padding: 0; font-size: 20px; line-height: 30px; mso-text-raise: 2px; mso-line-height-rule: exactly; vertical-align: middle"> <a href="https://hibernate.atlassian.net/browse/HSEARCH-3822?atlOrigin=eyJpIjoiNWE1MjMyNzE4MjMwNDA2YzgwNzk1ZmRhNTMwOWUxYTAiLCJwIjoiaiJ9" style="color: #3b73af; text-decoration: none">Restore support for concurrent Lucene flushes - at least during mass indexing</a> </span> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td class="email-content-main mobile-expand wrapper-special-margin" style="padding: 0; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff; padding-top: 10px; padding-bottom: 5px" bgcolor="#ffffff">
<table class="keyvalue-table" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0">
<tbody>
<tr>
<th style="color: #707070; font: normal 14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0">Change By:</th>
<td style="padding: 0; border-collapse: collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; vertical-align: top"> <a class="user-hover" rel="yrodiere" style="color:#6c797f;; color: #3b73af; text-decoration: none" id="email_yrodiere" href="https://hibernate.atlassian.net/secure/ViewProfile.jspa?accountId=557058%3A58fa1ced-171a-4c00-97e8-5d70d442cc4b"> Yoann Rodière </a> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td class="email-content-main mobile-expand issue-description-container" style="padding: 0; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff; padding-top: 5px; padding-bottom: 10px" bgcolor="#ffffff">
<table class="text-paragraph-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 2px">
<tbody>
<tr>
<td class="text-paragraph-pattern-container mobile-resize-text " style="padding: 0; border-collapse: collapse; padding: 0 0 10px"> <span class="diffcontext">In Search 5, we used to execute calls to the IndexWriter from multiple threads when mass indexing.</span> <span class="diffaddedchars" style="background-color:#ddfade;"><br><br></span> <span class="diffcontext"> As explained [here|http://blog.mikemccandless.com/2011/05/265-indexing-speedup-with-lucenes.html], Lucene does perform better when using multiple threads to index, at least if the hardware supports concurrent writes, thanks to concurrent flushing.<br><br></span> <span class="diffaddedchars" style="background-color:#ddfade;">Indexing is also faster when using multiple threads because analysis is relatively CPU-intensive.<br><br></span> <span class="diffcontext">In Search 6, we execute all calls to the IndexWriter from the same thread, with one thread assigned to each index manager. As a result, mass indexing performance is likely to degrade compared to Search 5</span> <span class="diffremovedchars" style="background-color: #ffe7e7; text-decoration:line-through;"> on hardware supporting concurrent writes</span> <span class="diffcontext">.<br><br>We should explore the following solutions, and pick whatever works best.<br><br>All solutions below rely on the same concept of [work stealing|https://en.wikipedia.org/wiki/Work_stealing]: a thread that has nothing to do will try to help out the thread of another index. The different lies only in which thread will try to help out.<br><br>All solutions will require some changes in the {{LuceneWriteWorkProcessor}}, which is currently thread-unsafe.<br><br>h3. Solution 1</span> <span class="diffaddedchars" style="background-color:#ddfade;"> (not recommended)</span> <span class="diffcontext"> : mass indexing thread helping with indexing<br><br>Restore the old behavior of executing writes from the mass indexing thread directly. This may prove difficult given the current architecture.<br><br>One possible solution: return custom objects instead of CompletableFuture from {{org.hibernate.search.mapper.pojo.work.spi.PojoIndexer#add}}, and instead of calling {{future.join()}} from the massIndexer thread, call {{theReturnedObject.executeIfNotAlreadyExecuted()}}, which will execute the work unless it's already been executed by the per-index thread.<br><br>h3. Solution 2</span> <span class="diffaddedchars" style="background-color:#ddfade;"> (better)</span> <span class="diffcontext">: other per-index threads helping with indexing<br><br>Allow threads assigned to one index to steal the work from another index when they become idle.<br><br></span> <span class="diffaddedchars" style="background-color:#ddfade;">h4. Ordering<br><br></span> <span class="diffcontext">As a result, even "traditional" (non-massindexer) worksets will no longer be executed in the order they were submitted</span> <span class="diffaddedchars" style="background-color:#ddfade;">.<br>We could give up on ordering completely</span> <span class="diffcontext">,</span> <span class="diffremovedchars" style="background-color: #ffe7e7; text-decoration:line-through;"> but I do not think it will have much impact since</span> <span class="diffaddedchars" style="background-color:#ddfade;"> because</span> <span class="diffcontext"> we usually only submit one workset per transaction, and the order worksets are submitted in is not guaranteed to match the order of transactions anyway.</span> <span class="diffremovedchars" style="background-color: #ffe7e7; text-decoration:line-through;"> If I</span> <span class="diffaddedchars" style="background-color:#ddfade;"> That</span> <span class="diffcontext">'</span> <span class="diffremovedchars" style="background-color: #ffe7e7; text-decoration:line-through;">m wrong and fixing</span> <span class="diffaddedchars" style="background-color:#ddfade;">s probably a bad idea, because we would definitely increase</span> <span class="diffcontext"> the</span> <span class="diffaddedchars" style="background-color:#ddfade;"> likelyhood of works being executed in the wrong</span> <span class="diffcontext"> order</span> <span class="diffremovedchars" style="background-color: #ffe7e7; text-decoration:line-through;"> proves necessary,</span> <span class="diffaddedchars" style="background-color:#ddfade;">.<br>Maybe</span> <span class="diffcontext"> we</span> <span class="diffaddedchars" style="background-color:#ddfade;"> should implement sharding by document ID in the queues, so that multiple threads</span> <span class="diffcontext"> can</span> <span class="diffremovedchars" style="background-color: #ffe7e7; text-decoration:line-through;"> have a look at solving HSEARCH-3664 first</span> <span class="diffaddedchars" style="background-color:#ddfade;"> actually process works on the same index, but from different queues?<br><br>h4</span> <span class="diffcontext">.</span> <span class="diffaddedchars" style="background-color:#ddfade;"> Benefits</span> <span class="diffcontext"><br><br>This solution could get us decent performance improvement for all writes, not just mass indexer writes, as soon as there are more than one index and only some of the indexes are being written to. Which is typically the case when mass indexing: by default we only mass index one type (= one index) at a time, but there are generally multiple types (= multiple indexes) to mass index.<br><br>We may even introduce settings to define a minimum and maximum number of threads, so that a Hibernate Search instance with 100 indexes still only uses 10 threads (max = 10), for example, with each thread sequentially working on each index, and a Hibernate Search instance with a single index still uses 2 threads (min = 2) to benefit from concurrent flushing.<br><br>For Infinispan, where we usually have only one index per cache (so one index per Hibernate Search instance), we could provide an SPI that allows Infinispan to define its own thread pool shared across all caches. That way, Infinispan would benefit from concurrent flushing.</span> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td class="email-content-main mobile-expand " style="padding: 0; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff" bgcolor="#ffffff"> <script type="application/ld+json">
{
"@context": "http://schema.org",
"@type": "EmailMessage",
"description": "View Issue",
"potentialAction": {
"@type": "ViewAction",
"target": "https://hibernate.atlassian.net/browse/HSEARCH-3822?inbox=true&",
"name": "View Issue"
},
"publisher": {
"@type": "Organization",
"name": "Atlassian",
"url": "https://www.atlassian.com"
}
}
</script>
<table id="actions-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 1px">
<tbody>
<tr>
<td id="actions-pattern-container" valign="middle" style="padding: 0; border-collapse: collapse; padding: 10px 0 10px 24px; vertical-align: middle; padding-left: 0">
<table align="left" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0">
<tbody>
<tr>
<td class="actions-pattern-action-icon-container" style="padding: 0; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 0; vertical-align: middle"> <a href="https://hibernate.atlassian.net/browse/HSEARCH-3822#add-comment?atlOrigin=eyJpIjoiNWE1MjMyNzE4MjMwNDA2YzgwNzk1ZmRhNTMwOWUxYTAiLCJwIjoiaiJ9" target="_blank" title="Add Comment" style="color: #3b73af; text-decoration: none"> <img class="actions-pattern-action-icon-image" src="cid:jira-generated-image-static-comment-icon-f7ca07e5-5e9e-484c-9c66-f3ac61bf3b68" alt="Add Comment" title="Add Comment" height="16" width="16" border="0" style="vertical-align: middle"> </a> </td>
<td class="actions-pattern-action-text-container" style="padding: 0; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 4px; padding-left: 5px"> <a href="https://hibernate.atlassian.net/browse/HSEARCH-3822#add-comment?atlOrigin=eyJpIjoiNWE1MjMyNzE4MjMwNDA2YzgwNzk1ZmRhNTMwOWUxYTAiLCJwIjoiaiJ9" target="_blank" title="Add Comment" style="color: #3b73af; text-decoration: none">Add Comment</a> </td>
</tr>
</tbody>
</table> </td>
</tr>
</tbody>
</table> </td>
</tr>
<!-- there needs to be content in the cell for it to render in some clients -->
<tr>
<td class="email-content-rounded-bottom mobile-expand" style="padding: 0; border-collapse: collapse; color: #ffffff; padding: 0 15px 0 16px; height: 5px; line-height: 5px; background-color: #ffffff; border-top: 0; border-left: 1px solid #cccccc; border-bottom: 1px solid #cccccc; border-right: 1px solid #cccccc; border-bottom-right-radius: 5px; border-bottom-left-radius: 5px; mso-line-height-rule: exactly" height="5" bgcolor="#ffffff"> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td id="footer-pattern" style="padding: 0; border-collapse: collapse; padding: 12px 20px">
<table id="footer-pattern-container" cellspacing="0" cellpadding="0" border="0" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0">
<tbody>
<tr>
<td id="footer-pattern-mobile-separated-links" class="mobile-resize-text" width="100%" colspan="2" style="padding: 0; border-collapse: collapse; color: #999999; font-size: 12px; line-height: 18px; font-family: Arial, sans-serif; mso-line-height-rule: exactly; mso-text-raise: 2px"> Get Jira notifications on your phone! Download the Jira Cloud app for <a href="https://play.google.com/store/apps/details?id=com.atlassian.android.jira.core&referrer=utm_source%3DNotificationLink%26utm_medium%3DEmail" style="color: #3b73af; text-decoration: none">Android</a> or <a href="https://itunes.apple.com/app/apple-store/id1006972087?pt=696495&ct=EmailNotificationLink&mt=8" style="color: #3b73af; text-decoration: none">iOS</a>
<hr> </td>
</tr>
<tr>
<td id="footer-pattern-text" class="mobile-resize-text" width="100%" style="padding: 0; border-collapse: collapse; color: #999999; font-size: 12px; line-height: 18px; font-family: Arial, sans-serif; mso-line-height-rule: exactly; mso-text-raise: 2px"> This message was sent by Atlassian Jira <span id="footer-build-information">(v1001.0.0-SNAPSHOT#100118-<span title="a758d7eb24011cd41321248c0c14a6d89605b72a" data-commit-id="a758d7eb24011cd41321248c0c14a6d89605b72a}">sha1:a758d7e</span>)</span> </td>
<td id="footer-pattern-logo-desktop-container" valign="top" style="padding: 0; border-collapse: collapse; padding-left: 20px; vertical-align: top">
<table style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0">
<tbody>
<tr>
<td id="footer-pattern-logo-desktop-padding" style="padding: 0; border-collapse: collapse; padding-top: 3px; opacity: 0.150"> <img id="footer-pattern-logo-desktop" src="cid:jira-generated-image-static-footer-desktop-logo-7d699ce0-1ce7-4794-b007-5cd3c1259447" alt="Atlassian logo" title="Atlassian logo" width="192" height="24" class="image_fix"> </td>
</tr>
</tbody>
</table> </td>
</tr>
</tbody>
</table> </td>
</tr>
</tbody>
</table>
<img border="0" width="1" height="1" alt="" src="http://atlassian.et.e.sparkpost.com/q/_Glu8Sg6y81HbdEjfRFg_w~~/AAAAAQA~/RgRgGr9OPlcLYXRsYXNzaWFudXNCCgAAzos5XuCvl89SIGhpYmVybmF0ZS1pc3N1ZXNAbGlzdHMuamJvc3Mub3JnWAQAAAAE">
</body>
</html>