<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0">
<base href="https://hibernate.atlassian.net">
<title>Message Title</title>
</head>
<body class="jira" style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 1.429">
<table id="background-table" cellpadding="0" cellspacing="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0; background-color: #f5f5f5; border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0" bgcolor="#f5f5f5">
<!-- header here -->
<tbody>
<tr>
<td id="header-pattern-container" style="padding: 0; border-collapse: collapse; padding: 10px 20px">
<table id="header-pattern" cellspacing="0" cellpadding="0" border="0" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0">
<tbody>
<tr>
<td id="header-avatar-image-container" valign="top" style="padding: 0; border-collapse: collapse; vertical-align: top; width: 32px; padding-right: 8px" width="32"> <img id="header-avatar-image" class="image_fix" src="https://avatar-cdn.atlassian.com/3673815784047b7e0673677a0bc7dde0?s=48&d=https%3A%2F%2Fsecure.gravatar.com%2Favatar%2F3673815784047b7e0673677a0bc7dde0%3Fd%3Dmm%26s%3D48%26noRedirect%3Dtrue" height="32" width="32" border="0" style="border-radius: 3px; vertical-align: top"> </td>
<td id="header-text-container" valign="middle" style="padding: 0; border-collapse: collapse; vertical-align: middle; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 1px"> <a class="user-hover" rel="yrodiere" id="email_yrodiere" href="https://hibernate.atlassian.net/secure/ViewProfile.jspa?name=yrodiere" style="color:#6c797f;; color: #3b73af; text-decoration: none">Yoann Rodière</a> <strong>updated</strong> an issue </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td id="email-content-container" style="padding: 0; border-collapse: collapse; padding: 0 20px">
<table id="email-content-table" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0; border-spacing: 0; border-collapse: separate">
<tbody>
<tr>
<!-- there needs to be content in the cell for it to render in some clients -->
<td class="email-content-rounded-top mobile-expand" style="padding: 0; border-collapse: collapse; color: #ffffff; padding: 0 15px 0 16px; height: 15px; background-color: #ffffff; border-left: 1px solid #cccccc; border-top: 1px solid #cccccc; border-right: 1px solid #cccccc; border-bottom: 0; border-top-right-radius: 5px; border-top-left-radius: 5px; height: 10px; line-height: 10px; padding: 0 15px 0 16px; mso-line-height-rule: exactly" height="10" bgcolor="#ffffff"> </td>
</tr>
<tr>
<td class="email-content-main mobile-expand " style="padding: 0; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff" bgcolor="#ffffff">
<table class="page-title-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0">
<tbody>
<tr>
<td class="page-title-pattern-first-line " style="padding: 0; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; padding-top: 10px"> <a href="https://hibernate.atlassian.net/browse/HSEARCH" style="color: #3b73af; text-decoration: none">Hibernate Search</a> / <a href="https://hibernate.atlassian.net/browse/HSEARCH-2761" style="color: #3b73af; text-decoration: none"><img src="cid:jira-generated-image-avatar-a912cf5a-bc3d-437d-a597-6db9ac37f2f0" height="16" width="16" border="0" align="absmiddle" alt="Bug" style="vertical-align: text-bottom"></a> <a href="https://hibernate.atlassian.net/browse/HSEARCH-2761" style="color: #3b73af; text-decoration: none">HSEARCH-2761</a> </td>
</tr>
<tr>
<td style="vertical-align: top;; padding: 0; border-collapse: collapse; padding-right: 5px; font-size: 20px; line-height: 30px; mso-line-height-rule: exactly" class="page-title-pattern-header-container"> <span class="page-title-pattern-header" style="font-family: Arial, sans-serif; padding: 0; font-size: 20px; line-height: 30px; mso-text-raise: 2px; mso-line-height-rule: exactly; vertical-align: middle"> <a href="https://hibernate.atlassian.net/browse/HSEARCH-2761" style="color: #3b73af; text-decoration: none">Elasticsearch purges may fail when purging multiple classes of the same inheritance tree in the same transaction</a> </span> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td id="text-paragraph-pattern-top" class="email-content-main mobile-expand comment-top-pattern" style="padding: 0; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff; border-bottom: none; padding-bottom: 0" bgcolor="#ffffff">
<table class="text-paragraph-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 2px">
<tbody>
<tr>
<td class="text-paragraph-pattern-container mobile-resize-text " style="padding: 0; border-collapse: collapse; padding: 0 0 10px"> <p style="margin: 10px 0 0; margin-top: 0">So. After a bit of investigation, the problem was actually that we performed mutliple purges, one after another, which targeted the same index (since they targeted classes in the same inheritance tree), but not the same types.</p> <p style="margin: 10px 0 0">Several separate things combined make this fail. I numbered each fact for easier reference below.</p>
<ol>
<li>Even with refreshAfterWrite enabled, we don't refresh between each single write, but simply after each changeset</li>
<li>When someone calls <tt>fullTextSession.purge</tt>, the fullTextSession will create one work per concrete type, not just one work for the given type</li>
<li>... and we also do that in the Elasticsearch integration when a purge is requested. As a result, when calling <tt>fullTextSession.purge</tt> on classes A, B and C (with C extends B extends A), we'll execute the following works: purge(A, B, C), purge(B, C), purge(C), instead of simply executing purge(A, B, C) OR executing purge(A) and purge(B) and purge(C). We effectively purge each type multiple times.</li>
<li>DeleteByQuery (used for purges on Elasticsearch) is a two-step operation: it first performs a search to retrieve the set of documents to delete, and then perform the actual deletion, failing if the (internal) version number retrieved when searching doesn't match the version number retrieved when deleting. As a result, using delete by query on a outdated (not refreshed) index may fail, even if the document was simply deleted in-between...</li>
<li>When multiple purges are requested in a transaction, their relative order may be altered randomly, because works are first stored in a HashMap (whose order may change upon inserts), and are transferred from this map to a list before we execute them (see <tt>org.hibernate.search.engine.impl.WorkPlan.getPlannedLuceneWork()</tt>).</li>
<li>Since <a href="https://github.com/hibernate/hibernate-search/commit/0ad3bab4450da36005dc801f4a046bb27fc1d57a" class="external-link" rel="nofollow" style="color: #3b73af; text-decoration: none">https://github.com/hibernate/hibernate-search/commit/0ad3bab4450da36005dc801f4a046bb27fc1d57a</a>, when one work in a changeset (list of works) fails, we don't execute the remaining works.</li>
</ol> <p style="margin: 10px 0 0">All of this combined will make the test fail:</p>
<ul>
<li>Because of 1, 2, 3 and 4, executing multiple purges in the same changeset on the same class hierarchy may fail (semi-randomly, because when we're lucky Elasticsearch may execute an automatic refresh between each purge)</li>
<li>Because of 5 and 6, the test failure is random (if purge(A, B, C) is executed first, then everything will be purged, and then even though the background thread will log a failure, the test will pass)</li>
</ul> <p style="margin: 10px 0 0">I believe there are multiple problems here. Fixing just one may be enough to make the test pass again, but I'd like to fix as many as possible...</p> <p style="margin: 10px 0 0">To be precise:</p>
<ul>
<li>Issue 1 should <b>at least</b> be mitigated. I see two (non-exclusive) approaches:</li>
</ul>
<ol>
<li>When refreshAfterWrite is enabled, I think we should just refresh after each write operation. Most of the time it will not change anything, because most of the time we just execute one bulked work. When executing multiple bulked works, the impact on performance should be small (one refresh every 250 works, and we could make this a configuration property)</li>
<li>When refreshAfterWrite is not enabled, I think we should force a refresh just before each DeleteByQuery, so that a purge between two additions to the index will work fine. DeleteByQuery is not something</li>
</ol>
<ul>
<li>Issue 3 (concrete types are resolved both in the ORM integration and Elasticsearch integration) should be fixed. We should not resolve concrete types in the Elasticsearch integration.</li>
<li>Issue 5 (the order of works within a transaction is not deterministic) should, in my opinion, be fixed. This kind of behavior is simply hell when we debug it. Using LinkedHashMaps instead of a HashMaps will solve the issue easily, and I doubt the impact on performance will be dramatic. At <b>least</b> we should do it at the class level (we rarely index hundreds of different classes in a single transaction, so this should really not change much performance-wise).</li>
<li>Issue 6 could be fixed, even though it's not absolutely necessary. But if we consider that the order of works doesn't matter (see issue 5), then stopping the execution of a changeset at the first failure doesn't make much sense, because the next works in the list should not be considered as depending on the failing one (it may even be the reverse: the failing one depended on the next works in the list).</li>
</ul> <p style="margin: 10px 0 0">Re-qualifying as bug (the problem was already there in previous releases, even though it may simply have resulted in errors being logged) and changing the title to better reflect the issue.</p> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td class="email-content-main mobile-expand wrapper-special-margin" style="padding: 0; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff; padding-top: 10px; padding-bottom: 5px" bgcolor="#ffffff">
<table class="keyvalue-table" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0">
<tbody>
<tr>
<th style="color: #707070; font: normal 14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0">Change By:</th>
<td style="padding: 0; border-collapse: collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; vertical-align: top"> <a class="user-hover" rel="yrodiere" id="email_yrodiere" href="https://hibernate.atlassian.net/secure/ViewProfile.jspa?name=yrodiere" style="color:#6c797f;; color: #3b73af; text-decoration: none">Yoann Rodière</a> </td>
</tr>
<tr>
<th style="color: #707070; font: normal 14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0">Summary:</th>
<td style="padding: 0; border-collapse: collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; vertical-align: top"> <span class="diffremovedchars" style="background-color:#ffe7e7;text-decoration:line-through;">IndexingActionInterceptorTest failing randomly</span> <span class="diffaddedchars" style="background-color:#ddfade;">Elasticsearch purges may fail when purging multiple classes of the same inheritance tree in the same transaction</span> </td>
</tr>
<tr>
<th style="color: #707070; font: normal 14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0">Issue Type:</th>
<td style="padding: 0; border-collapse: collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; vertical-align: top"> <span class="diffremovedchars" style="background-color:#ffe7e7;text-decoration:line-through;">Task</span> <span class="diffaddedchars" style="background-color:#ddfade;">Bug</span> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td class="email-content-main mobile-expand " style="padding: 0; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff" bgcolor="#ffffff"> <script type="application/ld+json">
{
"@context": "http://schema.org",
"@type": "EmailMessage",
"description": "View Issue",
"potentialAction": {
"@type": "ViewAction",
"target": "https://hibernate.atlassian.net/browse/HSEARCH-2761?inbox=true&focusedCommentId=95303&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-95303",
"name": "View Comment"
},
"publisher": {
"@type": "Organization",
"name": "Atlassian",
"url": "https://www.atlassian.com"
}
}
</script>
<table id="actions-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 1px">
<tbody>
<tr>
<td id="actions-pattern-container" valign="middle" style="padding: 0; border-collapse: collapse; padding: 10px 0 10px 24px; vertical-align: middle; padding-left: 0">
<table align="left" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0">
<tbody>
<tr>
<td class="actions-pattern-action-icon-container" style="padding: 0; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 0; vertical-align: middle"> <a href="https://hibernate.atlassian.net/browse/HSEARCH-2761#add-comment" target="_blank" title="Add Comment" style="color: #3b73af; text-decoration: none"> <img class="actions-pattern-action-icon-image" src="cid:jira-generated-image-static-comment-icon-9c3c766a-7228-470f-afd3-e5d872e9b904" alt="Add Comment" title="Add Comment" height="16" width="16" border="0" style="vertical-align: middle"> </a> </td>
<td class="actions-pattern-action-text-container" style="padding: 0; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 4px; padding-left: 5px"> <a href="https://hibernate.atlassian.net/browse/HSEARCH-2761#add-comment" target="_blank" title="Add Comment" style="color: #3b73af; text-decoration: none">Add Comment</a> </td>
</tr>
</tbody>
</table> </td>
</tr>
</tbody>
</table> </td>
</tr>
<!-- there needs to be content in the cell for it to render in some clients -->
<tr>
<td class="email-content-rounded-bottom mobile-expand" style="padding: 0; border-collapse: collapse; color: #ffffff; padding: 0 15px 0 16px; height: 5px; line-height: 5px; background-color: #ffffff; border-top: 0; border-left: 1px solid #cccccc; border-bottom: 1px solid #cccccc; border-right: 1px solid #cccccc; border-bottom-right-radius: 5px; border-bottom-left-radius: 5px; mso-line-height-rule: exactly" height="5" bgcolor="#ffffff"> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td id="footer-pattern" style="padding: 0; border-collapse: collapse; padding: 12px 20px">
<table id="footer-pattern-container" cellspacing="0" cellpadding="0" border="0" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0">
<tbody>
<tr>
</tr>
<tr>
<td id="footer-pattern-text" class="mobile-resize-text" width="100%" style="padding: 0; border-collapse: collapse; color: #999999; font-size: 12px; line-height: 18px; font-family: Arial, sans-serif; mso-line-height-rule: exactly; mso-text-raise: 2px"> This message was sent by Atlassian JIRA <span id="footer-build-information">(v1000.1045.1#100046-<span title="c375a71b67f3fdccea1a271097e8228667701032" data-commit-id="c375a71b67f3fdccea1a271097e8228667701032}">sha1:c375a71</span>)</span> </td>
<td id="footer-pattern-logo-desktop-container" valign="top" style="padding: 0; border-collapse: collapse; padding-left: 20px; vertical-align: top">
<table style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0">
<tbody>
<tr>
<td id="footer-pattern-logo-desktop-padding" style="padding: 0; border-collapse: collapse; padding-top: 3px"> <img id="footer-pattern-logo-desktop" src="cid:jira-generated-image-static-footer-desktop-logo-8b9df5b1-9044-4206-a02b-df070ab73116" alt="Atlassian logo" title="Atlassian logo" width="169" height="36" class="image_fix"> </td>
</tr>
</tbody>
</table> </td>
</tr>
</tbody>
</table> </td>
</tr>
</tbody>
</table>
</body>
</html>