<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0">
<base href="https://hibernate.atlassian.net">
<title>Message Title</title>
</head>
<body class="jira" style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 1.429">
<table id="background-table" cellpadding="0" cellspacing="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0; background-color: #f5f5f5; border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0" bgcolor="#f5f5f5">
<!-- header here -->
<tbody>
<tr>
<td id="header-pattern-container" style="padding: 0; border-collapse: collapse; padding: 10px 20px">
<table id="header-pattern" cellspacing="0" cellpadding="0" border="0" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0">
<tbody>
<tr>
<td id="header-avatar-image-container" valign="top" style="padding: 0; border-collapse: collapse; vertical-align: top; width: 32px; padding-right: 8px" width="32"> <img id="header-avatar-image" class="image_fix" src="cid:jira-generated-image-avatar-e2669480-88b7-4212-94d1-46923b116771" height="32" width="32" border="0" style="border-radius: 3px; vertical-align: top"> </td>
<td id="header-text-container" valign="middle" style="padding: 0; border-collapse: collapse; vertical-align: middle; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 1px"> <a class="user-hover" rel="yrodiere" id="email_yrodiere" href="https://hibernate.atlassian.net/secure/ViewProfile.jspa?name=yrodiere" style="color:#6c797f;; color: #3b73af; text-decoration: none">Yoann Rodière</a> <strong>commented</strong> on <a href="https://hibernate.atlassian.net/browse/HSEARCH-2189" style="color: #3b73af; text-decoration: none"><img src="cid:jira-generated-image-avatar-4bc7ef0c-0867-40cc-a1b6-dea82440687e" height="16" width="16" border="0" align="absmiddle" alt="Bug"> HSEARCH-2189</a> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td id="email-content-container" style="padding: 0; border-collapse: collapse; padding: 0 20px">
<table id="email-content-table" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0; border-spacing: 0; border-collapse: separate">
<tbody>
<tr>
<!-- there needs to be content in the cell for it to render in some clients -->
<td class="email-content-rounded-top mobile-expand" style="padding: 0; border-collapse: collapse; color: #ffffff; padding: 0 15px 0 16px; height: 15px; background-color: #ffffff; border-left: 1px solid #cccccc; border-top: 1px solid #cccccc; border-right: 1px solid #cccccc; border-bottom: 0; border-top-right-radius: 5px; border-top-left-radius: 5px; height: 10px; line-height: 10px; padding: 0 15px 0 16px; mso-line-height-rule: exactly" height="10" bgcolor="#ffffff"> </td>
</tr>
<tr>
<td class="email-content-main mobile-expand " style="padding: 0; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff" bgcolor="#ffffff">
<table class="page-title-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0">
<tbody>
<tr>
<td style="vertical-align: top;; padding: 0; border-collapse: collapse; padding-right: 5px; font-size: 20px; line-height: 30px; mso-line-height-rule: exactly" class="page-title-pattern-header-container"> <span class="page-title-pattern-header" style="font-family: Arial, sans-serif; padding: 0; font-size: 20px; line-height: 30px; mso-text-raise: 2px; mso-line-height-rule: exactly; vertical-align: middle"> <a href="https://hibernate.atlassian.net/browse/HSEARCH-2189" style="color: #3b73af; text-decoration: none">Re: Querying elastic search triggers exception "Result window is too large"</a> </span> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td id="text-paragraph-pattern-top" class="email-content-main mobile-expand comment-top-pattern" style="padding: 0; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff; border-bottom: none; padding-bottom: 0" bgcolor="#ffffff">
<table class="text-paragraph-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 2px">
<tbody>
<tr>
<td class="text-paragraph-pattern-container mobile-resize-text " style="padding: 0; border-collapse: collapse; padding: 0 0 10px"> <p style="margin: 10px 0 0; margin-top: 0"><a href="https://hibernate.atlassian.net/secure/ViewProfile.jspa?name=gsmet" class="user-hover" rel="gsmet" style="color: #3b73af; text-decoration: none">Guillaume Smet</a> <a href="https://hibernate.atlassian.net/secure/ViewProfile.jspa?name=crancran" class="user-hover" rel="crancran" style="color: #3b73af; text-decoration: none">Chris Cranford</a> I just investigated the Scroll API to see if we could wire the <tt>ScrollableResults</tt> from a <tt>FullTextQuery</tt> (<tt>fullTextQuery.scroll()</tt>) to some object taking advantage of Elasticsearch's Scroll API. Well, we cannot, because <tt>ScrollableResults</tt> offers far more methods than what Elasticsearch provides. So using the Scroll API to implement <tt>ScrollableResults</tt> would mean throwing <tt>UnsupportedOperationException</tt> in most methods.<br> Implementing the basic queries (<tt>getResultList()</tt>/<tt>list()</tt>, with an offset and a maximum number of results) is not possible either with the Scroll API, or at least not in an efficient way: the scroll API does not allow using an offset (the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-from-size.html" class="external-link" rel="nofollow" style="color: #3b73af; text-decoration: none"><tt>from</tt></a> attribute is ignored), so we would have to scroll through every previous result each time a user uses an offset. For the same performance reasons, we cannot use the Scroll API as a fallback when Elasticsearch throws a "Result window is too large" error at us.</p> <p style="margin: 10px 0 0">Also to be noted, increasing the value of <tt>index.max_result_window</tt> seems to be discouraged for performance reasons: <a href="https://www.elastic.co/guide/en/elasticsearch/reference/2.4/breaking_21_search_changes.html#_from_size_limits" class="external-link" rel="nofollow" style="color: #3b73af; text-decoration: none">https://www.elastic.co/guide/en/elasticsearch/reference/2.4/breaking_21_search_changes.html#_from_size_limits</a></p> <p style="margin: 10px 0 0">Here are the solutions:</p>
<ol>
<li>implementing the pre-existing Hibernate ORM <tt>scroll()</tt> method in such a way that it will work as usual with the Lucene backend, will also work (with unlimited scrolling) with the Elasticsearch backend, but that the scrollable results with the Elasticsearch backend will through <tt>UnsupportedOperationException</tt> in most methods (<tt>previous()</tt>, <tt>last()</tt>, <tt>setRowNumber(int)</tt>, ...)</li>
<li>implementing the pre-existing Hibernate ORM <tt>scroll()</tt> fully for both the Lucene and Elasticsearch backend, using horribly inefficient workarounds for methods not supported by the Elasticsearch Scroll API (<tt>previous()</tt>, <tt>last()</tt>, <tt>setRowNumber(int)</tt>, ...).</li>
</ol> <p style="margin: 10px 0 0">Personally, I'd be in favor of solution 1. Offering inefficient methods in an API that's primarily aiming at processing large datasets efficiently seems a nonsense to me. But some implementors chose to do just that, like <a href="http://grepcode.com/file/repo1.maven.org/maven2/com.h2database/h2/1.3.164/org/h2/jdbc/JdbcResultSet.java#JdbcResultSet.previous%28%29" class="external-link" rel="nofollow" style="color: #3b73af; text-decoration: none">H2</a>, so...</p> <p style="margin: 10px 0 0">In any case, this will require non-trivial SPI additions (most notably a "scroll" method in <tt>org.hibernate.search.query.engine.spi.HSQuery</tt>). This might mean that the fix will only be merged in 6.0; I'll have to check with Sanne, I guess.</p> <p style="margin: 10px 0 0">I'm starting the work on solution 1. Feel free to ping me if you disagree with the whole approach (better now than when I submit a PR <img class="emoticon" src="cid:jira-generated-image-static-wink-b541f1ea-6c63-45da-9bda-e56243a27565" height="16" width="16" align="absmiddle" alt="" border="0"> )</p> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td class="email-content-main mobile-expand " style="padding: 0; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff" bgcolor="#ffffff">
<table id="actions-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 1px">
<tbody>
<tr>
<td id="actions-pattern-container" valign="middle" style="padding: 0; border-collapse: collapse; padding: 10px 0 10px 24px; vertical-align: middle; padding-left: 0">
<table align="left" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0">
<tbody>
<tr>
<td class="actions-pattern-action-icon-container" style="padding: 0; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 0; vertical-align: middle"> <a href="https://hibernate.atlassian.net/browse/HSEARCH-2189#add-comment" target="_blank" title="Add Comment" style="color: #3b73af; text-decoration: none"> <img class="actions-pattern-action-icon-image" src="cid:jira-generated-image-static-comment-icon-8133756e-2712-450d-abd4-33ba9024b79d" alt="Add Comment" title="Add Comment" height="16" width="16" border="0" style="vertical-align: middle"> </a> </td>
<td class="actions-pattern-action-text-container" style="padding: 0; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 4px; padding-left: 5px"> <a href="https://hibernate.atlassian.net/browse/HSEARCH-2189#add-comment" target="_blank" title="Add Comment" style="color: #3b73af; text-decoration: none">Add Comment</a> </td>
</tr>
</tbody>
</table> </td>
</tr>
</tbody>
</table> </td>
</tr>
<!-- there needs to be content in the cell for it to render in some clients -->
<tr>
<td class="email-content-rounded-bottom mobile-expand" style="padding: 0; border-collapse: collapse; color: #ffffff; padding: 0 15px 0 16px; height: 5px; line-height: 5px; background-color: #ffffff; border-top: 0; border-left: 1px solid #cccccc; border-bottom: 1px solid #cccccc; border-right: 1px solid #cccccc; border-bottom-right-radius: 5px; border-bottom-left-radius: 5px; mso-line-height-rule: exactly" height="5" bgcolor="#ffffff"> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td id="footer-pattern" style="padding: 0; border-collapse: collapse; padding: 12px 20px">
<table id="footer-pattern-container" cellspacing="0" cellpadding="0" border="0" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0">
<tbody>
<tr>
<td id="footer-pattern-text" class="mobile-resize-text" width="100%" style="padding: 0; border-collapse: collapse; color: #999999; font-size: 12px; line-height: 18px; font-family: Arial, sans-serif; mso-line-height-rule: exactly; mso-text-raise: 2px"> This message was sent by Atlassian JIRA <span id="footer-build-information">(v1000.362.1#100014-<span title="6f27b8c49e62d4306cf33f09d846010573a9610b" data-commit-id="6f27b8c49e62d4306cf33f09d846010573a9610b}">sha1:6f27b8c</span>)</span> </td>
<td id="footer-pattern-logo-desktop-container" valign="top" style="padding: 0; border-collapse: collapse; padding-left: 20px; vertical-align: top">
<table style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0">
<tbody>
<tr>
<td id="footer-pattern-logo-desktop-padding" style="padding: 0; border-collapse: collapse; padding-top: 3px"> <img id="footer-pattern-logo-desktop" src="cid:jira-generated-image-static-footer-desktop-logo-500589d3-1d05-4610-ac41-b77836f27515" alt="Atlassian logo" title="Atlassian logo" width="169" height="36" class="image_fix"> </td>
</tr>
</tbody>
</table> </td>
</tr>
</tbody>
</table> </td>
</tr>
</tbody>
</table>
</body>
</html>