<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0" /> <base href="https://hibernate.atlassian.net" />
<title>Message Title</title>
</head>
<body class="jira" style="color: #333; font-family: Arial, sans-serif; font-size: 14px; line-height: 1.429">
<table id="background-table" cellpadding="0" cellspacing="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; background-color: #f5f5f5; border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<!-- header here -->
<tr>
<td id="header-pattern-container" style="padding: 0px; border-collapse: collapse; padding: 10px 20px">
<table id="header-pattern" cellspacing="0" cellpadding="0" border="0" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tr>
<td id="header-avatar-image-container" valign="top" style="padding: 0px; border-collapse: collapse; vertical-align: top; width: 32px; padding-right: 8px"> <img id="header-avatar-image" class="image_fix" src="https://secure.gravatar.com/avatar/57637a2eb871b34eba14e700c78c6a5d?d=mm&s=48" height="32" width="32" border="0" style="border-radius: 3px; vertical-align: top" />
</td>
<td id="header-text-container" valign="middle" style="padding: 0px; border-collapse: collapse; vertical-align: middle; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 1px"> <a class="user-hover" rel="sanne" id="email_sanne" href="https://hibernate.atlassian.net/secure/ViewProfile.jspa?name=sanne" style="color:#6c797f;; color: #3b73af; text-decoration: none">Sanne Grinovero</a> <strong>updated</strong> an issue
</td>
</tr>
</table>
</td>
</tr>
<tr>
<td id="email-content-container" style="padding: 0px; border-collapse: collapse; padding: 0 20px">
<table id="email-content-table" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; border-spacing: 0; border-collapse: separate">
<tr>
<!-- there needs to be content in the cell for it to render in some clients -->
<td class="email-content-rounded-top mobile-expand" style="padding: 0px; border-collapse: collapse; color: #fff; padding: 0 15px 0 16px; height: 15px; background-color: #fff; border-left: 1px solid #ccc; border-top: 1px solid #ccc; border-right: 1px solid #ccc; border-bottom: 0; border-top-right-radius: 5px; border-top-left-radius: 5px; height: 10px; line-height: 10px; padding: 0 15px 0 16px; mso-line-height-rule: exactly">
</td>
</tr>
<tr>
<td class="email-content-main mobile-expand " style="padding: 0px; border-collapse: collapse; border-left: 1px solid #ccc; border-right: 1px solid #ccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #fff">
<table class="page-title-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tr>
<td class="page-title-pattern-first-line " style="padding: 0px; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; padding-top: 10px"> <a href="https://hibernate.atlassian.net/browse/HSEARCH" style="color: #3b73af; text-decoration: none">Hibernate Search</a> / <a href="https://hibernate.atlassian.net/browse/HSEARCH-1699" style="color: #3b73af; text-decoration: none"><img src="cid:jira-generated-image-static-improvement-075385b6-c98e-4faf-9f97-67dc07f3d791" height="16" width="16" border="0" align="absmiddle" alt="Improvement" style="vertical-align: text-bottom" /></a> <a href="https://hibernate.atlassian.net/browse/HSEARCH-1699" style="color: #3b73af; text-decoration: none">HSEARCH-1699</a>
</td>
</tr>
<tr>
<td style="vertical-align: top;; padding: 0px; border-collapse: collapse; padding-right: 5px; font-size: 20px; line-height: 30px; mso-line-height-rule: exactly" class="page-title-pattern-header-container"> <span class="page-title-pattern-header" style="font-family: Arial, sans-serif; padding: 0; font-size: 20px; line-height: 30px; mso-text-raise: 2px; mso-line-height-rule: exactly; vertical-align: middle"> <a href="https://hibernate.atlassian.net/browse/HSEARCH-1699" style="color: #3b73af; text-decoration: none">Batching IndexWriter commits for multiple synchronous worksets</a> </span>
</td>
</tr>
</table>
</td>
</tr>
<tr>
<td class="email-content-main mobile-expand wrapper-special-margin" style="padding: 0px; border-collapse: collapse; border-left: 1px solid #ccc; border-right: 1px solid #ccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #fff; padding-top: 10px; padding-bottom: 5px">
<table class="keyvalue-table" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tr>
<th style="color: #707070; font: normal 14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0">
Change By:
</th>
<td style="padding: 0px; border-collapse: collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; vertical-align: top"> <a class="user-hover" rel="sanne" id="email_sanne" href="https://hibernate.atlassian.net/secure/ViewProfile.jspa?name=sanne" style="color:#6c797f;; color: #3b73af; text-decoration: none">Sanne Grinovero</a>
</td>
</tr>
</table>
</td>
</tr>
<tr>
<td class="email-content-main mobile-expand issue-description-container" style="padding: 0px; border-collapse: collapse; border-left: 1px solid #ccc; border-right: 1px solid #ccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #fff; padding-top: 5px; padding-bottom: 10px">
<table class="text-paragraph-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 2px">
<tr>
<td class="text-paragraph-pattern-container mobile-resize-text " style="padding: 0px; border-collapse: collapse; padding: 0 0 10px 0"> <span class="diffcontext">An NRT based backend is orders of magnitude more efficient than a non-NRT one as it avoids a full flush cycle to the index for each {{LuceneBackendQueueTask}} which is applied to the index, and only triggers such flush to the Directory on specific needs (like when the in-memory buffers are running out of space).<br /><br />But since changes are buffered in memory rather than being pushed to the store right away, an application requiring synchronous changes but using the Infinispan Directory to replicate changes to other application nodes can't use NRT as these changes would not be propagated synchronously, they would only be visible eventually when a flush event is triggered.<br /><br />Consider that the Infinispan Directory can provide exceptional read efficiency but is not efficient at all at handling:<br /> - read locks as they need to be emulated with multiple RPCs doing optimistic CAS operations<br /> - index metadata replacements are also based on distributed CAS operations<br /> - a single segment write might trigger up to 8 different index metadata writes<br /> - one out of 10 (by default, or as configured the merging factor) writes on segment will trigger a rewrite of all segments<br /> - load on any IndexReader refresh will increase contention on the above CAS operations<br /> - and IndexReader refresh is triggered by visibility of a flush event<br /><br />If you combine the notion of each of these bullet points, it's clear that each increase of {{IndexReader}} refresh operations, and each {{IndexWriter}} commit have an exponential effect on worsening the mount of network traffic and make the CAS operations unlikely to ever spin out.<br />The most important observation is that the {{IndexReader}} refresh operations are affected linearly by the frequency of commits: a refresh is directly caused by a previous commit. So simply controlling the frequency of {{IndexWriter}} commit events will directly control the frequency of {{IndexReader}} refresh operations, and so has the potential to dramatically increase the write throughput on the {{InfinispanDirectoryProvider}}.<br /><br />So the core of the idea here is that for two incoming changesets A and B, incoming on behalf of two parallel user transactions TXA and TXB, normally we would apply this sequence of operations:<br /><br /># TXA enqueues A, and blocks waiting for it.<br /># TXB enqueues B, and blocks waiting for it.<br /># IndexWriter thread takes A from the Queue<br /># IndexWriter thread applies A changes to the index<br /># IndexWriter thread commits and flushes the index<br /># TXA is notified and is unblocked<br /># IndexWriter thread takes B from the Queue<br /># IndexWriter thread applies B changes to the index<br /># IndexWriter thread commits and flushes the index<br /># TXB is notified and is unblocked<br /><br />The above is the current strategy, and releases the waiting threads as soon as possible.<br />Let's assume that each write takes some nanoseconds, and each commit takes 200ms; actual figures will vary across systems and configurations but won't be too far from these orders of magnitude; so for the sake of this design we can approximate the cost of each write to nothing, while a commit time is significant.<br /><br />The latency of the above described schema will then be approximately:<br /> - TXA will return in 1 commit (plus a nanosecond)<br /> - TXB will return in 2 commits (plus a couple nanoseconds)<br /><br />If we now reorder the commit sequence to actually perform something like this:<br /><br /># TXA enqueues A, and blocks waiting for it.<br /># TXB enqueues B, and blocks waiting for it.<br /># IndexWriter thread takes A&B from the Queue<br /># IndexWriter thread applies A&B changes to the index (maintaining order)<br /># IndexWriter thread commits and flushes the index<br /># TXA is notified and is unblocked<br /># TXB is notified and is unblocked<br /><br />at this point the measured latency will be:<br /> - TXA will return in 1 commit (plus a couple nanoseconds)<br /> - TXB will return in 1 commit (plus a couple nanoseconds)<br /><br />Compared to the previous results, TXA was delayed by some nanoseconds but the order of magnitude of its response time is not affected.<br />TXB however did cut its waiting time in half.. more importantly, it's speed is increased by a factor N where N in this case is 2 as we have only two transactions writing, but N could be 100, you'd have a 100X performance improvement.<br /><br />So what's the throughput that we get out of this pattern?<br /><br />What matters most is to cap the _target frequency_ of commits to a specific upper bound, and we want to cap this frequency at exactly the maximum throughput that the Infinispan storage can actually take. So if the IndexWriter thread is designed as a busy loop which always takes all of N waiting changesets and applies them, if the commit is slow there will be more changes in the queue waiting for the next iteration of the write loop, making N larger and so scaling up the throughput: the more the storage is slow, the larger the batches.<br /><br />The upper bound</span> <span class="diffremovedchars" style="background-color: #ffe7e7; text-decoration:line-through;"> if</span> <span class="diffaddedchars" style="background-color:#ddfade;"> is</span> <span class="diffcontext"> of course determined by the speed of writes - the ones that we previously considered negligible as they operate in the nanoseconds range; they will</span> <span class="diffremovedchars" style="background-color: #ffe7e7; text-decoration:line-through;"> of course</span> <span class="diffcontext"> stack up and eventually become the limiting factor.</span> <span class="diffremovedchars" style="background-color: #ffe7e7; text-decoration:line-through;"> Also the frequency of commits might tend to zero for very slow stores and we don't want that to happen so the design</span> <span class="diffaddedchars" style="background-color:#ddfade;"> The implementation</span> <span class="diffcontext"> needs</span> <span class="diffaddedchars" style="background-color:#ddfade;"> also</span> <span class="diffcontext"> to take into account</span> <span class="diffremovedchars" style="background-color: #ffe7e7; text-decoration:line-through;"> also</span> <span class="diffaddedchars" style="background-color:#ddfade;"> at least</span> <span class="diffcontext"> two</span> <span class="diffremovedchars" style="background-color: #ffe7e7; text-decoration:line-through;"> boundaries</span> <span class="diffaddedchars" style="background-color:#ddfade;"> exceptional cases</span> <span class="diffcontext">:<br /> - what's the IndexWriter loop going to do when no changes are scheduled<br /> - do we need to cap the maximum size of batches<br /><br />Ultimately the latency of each user thread is still going to be close to the performance that we can get out of the store to perform a single commit operation, as at least one commit needs to happen before the synchronous user thread is unblocked. The benefit will be that different threads won't "pile up" changes creating larger wait times but the response time is expected to be quite predictable, whatever the load. Systems with an high degree of parallelism will benefit from this.</span>
</td>
</tr>
</table>
</td>
</tr>
<tr>
<td class="email-content-main mobile-expand " style="padding: 0px; border-collapse: collapse; border-left: 1px solid #ccc; border-right: 1px solid #ccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #fff">
<table id="actions-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 1px">
<tr>
<td id="actions-pattern-container" valign="middle" style="padding: 0px; border-collapse: collapse; padding: 10px 0 10px 24px; vertical-align: middle; padding-left: 0">
<table align="left" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tr>
<td class="actions-pattern-action-icon-container" style="padding: 0px; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 0px; vertical-align: middle"> <a href="https://hibernate.atlassian.net/browse/HSEARCH-1699#add-comment" target="_blank" title="Add Comment" style="color: #3b73af; text-decoration: none"> <img class="actions-pattern-action-icon-image" src="cid:jira-generated-image-static-comment-icon-d4dfad6b-ac1c-4c0a-8e7e-8312c210d31c" alt="Add Comment" title="Add Comment" height="16" width="16" border="0" style="vertical-align: middle" /> </a>
</td>
<td class="actions-pattern-action-text-container" style="padding: 0px; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 4px; padding-left: 5px"> <a href="https://hibernate.atlassian.net/browse/HSEARCH-1699#add-comment" target="_blank" title="Add Comment" style="color: #3b73af; text-decoration: none">Add Comment</a>
</td>
</tr>
</table>
</td>
</tr>
</table>
</td>
</tr>
<!-- there needs to be content in the cell for it to render in some clients -->
<tr>
<td class="email-content-rounded-bottom mobile-expand" style="padding: 0px; border-collapse: collapse; color: #fff; padding: 0 15px 0 16px; height: 5px; line-height: 5px; background-color: #fff; border-top: 0; border-left: 1px solid #ccc; border-bottom: 1px solid #ccc; border-right: 1px solid #ccc; border-bottom-right-radius: 5px; border-bottom-left-radius: 5px; mso-line-height-rule: exactly">
</td>
</tr>
</table>
</td>
</tr>
<tr>
<td id="footer-pattern" style="padding: 0px; border-collapse: collapse; padding: 12px 20px">
<table id="footer-pattern-container" cellspacing="0" cellpadding="0" border="0" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tr>
<td id="footer-pattern-text" class="mobile-resize-text" width="100%" style="padding: 0px; border-collapse: collapse; color: #999; font-size: 12px; line-height: 18px; font-family: Arial, sans-serif; mso-line-height-rule: exactly; mso-text-raise: 2px">
This message was sent by Atlassian JIRA <span id="footer-build-information">(v6.4-OD-07-004#64005-<span title="29a0beaca7c693975974e1d5d9f869ecb60599fc" data-commit-id="29a0beaca7c693975974e1d5d9f869ecb60599fc}">sha1:29a0bea</span>)</span>
</td>
<td id="footer-pattern-logo-desktop-container" valign="top" style="padding: 0px; border-collapse: collapse; padding-left: 20px; vertical-align: top">
<table style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tr>
<td id="footer-pattern-logo-desktop-padding" style="padding: 0px; border-collapse: collapse; padding-top: 3px"> <img id="footer-pattern-logo-desktop" src="cid:jira-generated-image-static-footer-desktop-logo-4393208a-a6f6-4f54-85dc-ed34aec33523" alt="Atlassian logo" title="Atlassian logo" width="169" height="36" class="image_fix" />
</td>
</tr>
</table>
</td>
</tr>
</table>
</td>
</tr>
</table>
</body>
</html>