<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0">
<base href="https://hibernate.atlassian.net">
<title>Message Title</title>
</head>
<body class="jira" style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 1.429">
<table id="background-table" cellpadding="0" cellspacing="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0; background-color: #f5f5f5; border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0" bgcolor="#f5f5f5"> <!-- header here -->
<tbody>
<tr>
<td id="header-pattern-container" style="padding: 0; border-collapse: collapse; padding: 10px 20px">
<table id="header-pattern" cellspacing="0" cellpadding="0" border="0" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0">
<tbody>
<tr>
<td id="header-avatar-image-container" valign="top" style="padding: 0; border-collapse: collapse; vertical-align: top; width: 32px; padding-right: 8px" width="32"> <img id="header-avatar-image" class="image_fix" src="https://secure.gravatar.com/avatar/2a8bdd4ffd282b7185c74b52ab452617?d=https%3A%2F%2Favatar-management--avatars.us-west-2.prod.public.atl-paas.net%2Finitials%2FYR-6.png&size=48&s=48" height="32" width="32" border="0" style="border-radius: 3px; vertical-align: top"> </td>
<td id="header-text-container" valign="middle" style="padding: 0; border-collapse: collapse; vertical-align: middle; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 1px"> <a class="user-hover" rel="yrodiere" style="color:#6c797f;; color: #3b73af; text-decoration: none" id="email_yrodiere" href="https://hibernate.atlassian.net/secure/ViewProfile.jspa?accountId=557058%3A58fa1ced-171a-4c00-97e8-5d70d442cc4b"> Yoann Rodière </a> <strong>updated</strong> an issue </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td id="email-content-container" style="padding: 0; border-collapse: collapse; padding: 0 20px">
<table id="email-content-table" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0; border-spacing: 0; border-collapse: separate">
<tbody>
<tr> <!-- there needs to be content in the cell for it to render in some clients -->
<td class="email-content-rounded-top mobile-expand" style="padding: 0; border-collapse: collapse; color: #ffffff; padding: 0 15px 0 16px; height: 15px; background-color: #ffffff; border-left: 1px solid #cccccc; border-top: 1px solid #cccccc; border-right: 1px solid #cccccc; border-bottom: 0; border-top-right-radius: 5px; border-top-left-radius: 5px; height: 10px; line-height: 10px; padding: 0 15px 0 16px; mso-line-height-rule: exactly" height="10" bgcolor="#ffffff"> </td>
</tr>
<tr>
<td class="email-content-main mobile-expand " style="padding: 0; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff" bgcolor="#ffffff">
<table class="page-title-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0">
<tbody>
<tr>
<td class="page-title-pattern-first-line " style="padding: 0; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; padding-top: 10px"> <a href="https://hibernate.atlassian.net/browse/HSEARCH?atlOrigin=eyJpIjoiZWJiZDk3NWU3OTIxNDYzOGI5ZGY5MmY3YjY3OTNmY2YiLCJwIjoiaiJ9" style="color: #3b73af; text-decoration: none">Hibernate Search</a> / <a href="https://hibernate.atlassian.net/browse/HSEARCH-3323?atlOrigin=eyJpIjoiZWJiZDk3NWU3OTIxNDYzOGI5ZGY5MmY3YjY3OTNmY2YiLCJwIjoiaiJ9" style="color: #3b73af; text-decoration: none"><img src="cid:jira-generated-image-avatar-70d63334-4693-4ae1-81d3-baa4a872956e" height="16" width="16" border="0" align="absmiddle" alt="Task" style="vertical-align: text-bottom"></a> <a href="https://hibernate.atlassian.net/browse/HSEARCH-3323?atlOrigin=eyJpIjoiZWJiZDk3NWU3OTIxNDYzOGI5ZGY5MmY3YjY3OTNmY2YiLCJwIjoiaiJ9" style="color: #3b73af; text-decoration: none">HSEARCH-3323</a> </td>
</tr>
<tr>
<td style="vertical-align: top;; padding: 0; border-collapse: collapse; padding-right: 5px; font-size: 20px; line-height: 30px; mso-line-height-rule: exactly" class="page-title-pattern-header-container"> <span class="page-title-pattern-header" style="font-family: Arial, sans-serif; padding: 0; font-size: 20px; line-height: 30px; mso-text-raise: 2px; mso-line-height-rule: exactly; vertical-align: middle"> <a href="https://hibernate.atlassian.net/browse/HSEARCH-3323?atlOrigin=eyJpIjoiZWJiZDk3NWU3OTIxNDYzOGI5ZGY5MmY3YjY3OTNmY2YiLCJwIjoiaiJ9" style="color: #3b73af; text-decoration: none">Search 6 groundwork - Restore support for scrolling</a> </span> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td class="email-content-main mobile-expand wrapper-special-margin" style="padding: 0; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff; padding-top: 10px; padding-bottom: 5px" bgcolor="#ffffff">
<table class="keyvalue-table" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0">
<tbody>
<tr>
<th style="color: #707070; font: normal 14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0">Change By:</th>
<td style="padding: 0; border-collapse: collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; vertical-align: top"> <a class="user-hover" rel="yrodiere" style="color:#6c797f;; color: #3b73af; text-decoration: none" id="email_yrodiere" href="https://hibernate.atlassian.net/secure/ViewProfile.jspa?accountId=557058%3A58fa1ced-171a-4c00-97e8-5d70d442cc4b"> Yoann Rodière </a> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td class="email-content-main mobile-expand issue-description-container" style="padding: 0; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff; padding-top: 5px; padding-bottom: 10px" bgcolor="#ffffff">
<table class="text-paragraph-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 2px">
<tbody>
<tr>
<td class="text-paragraph-pattern-container mobile-resize-text " style="padding: 0; border-collapse: collapse; padding: 0 0 10px"> <span class="diffcontext">h3. Goal<br><br>Restore the scroll feature exposed in Search 5 through {{org.hibernate.search.query.hibernate.impl.FullTextQueryImpl#scroll()}}.<br><br>h3. API<br><br>All located in the {{org.hibernate.search.engine.search.query}} package.<br><br>{code}<br>public interface SearchFetchable {<br><br> // ... there's already some code here...<br><br> // Add this (+ javadoc):<br> // Throws IllegalArgumentException if passed 0 or less (see the class Contracts).<br> SearchScroll<H> scroll(Integer pageSize);<br><br> // Add this (+ javadoc):<br> // Throws IllegalArgumentException if passed 0 or less for pageSize (see the class Contracts).<br> // Throws IllegalArgumentException if passed less than 0 for offset (see the class Contracts).<br> // TODO maybe it's not possible to implement this efficiently for Elasticsearch (not sure it accepts an offset when scrolling is enabled). In that case, remove this method.<br> SearchScroll<H> scroll(Integer offset, Integer pageSize);<br><br>}<br>{code}<br><br>{code}<br>// This will be used like this:<br>// try (SearchScroll<H> scroll = query.scroll(20)) {<br>// for (SearchScrollResult<H> page = scroll.next(); page.hasHits(); page = scroll.next()) {<br>// List<H> hits = page.getHits();<br>// // ... do something with the page ...<br>// }<br>// }<br>public interface SearchScroll<H> extends AutoCloseable {<br><br> @Override<br> void close();<br><br> // TODO: javadoc<br> // Returns the next page, with at most "pageSize" hits ("pageSize" defined in the call to query.scroll()).<br> // May return a result with less than "pageSize" elements if only that many hits are left.<br> // This should *not* rely on pre-fetching. Fetching should happen when this method is called, not before.<br> // This is necessary if we want to make it easy for users to clear the ORM session between two pages.<br> // Note there is no "hasNext" method precisely because we do not do pre-fetching.<br> SearchScrollResult<H> next();<br> <br>}<br>{code}<br><br>{code}<br>public interface SearchScrollResult<H> {<br><br> // TODO: javadoc<br> // This returns true if there are still hits, false otherwise.<br> // Note hasHits() == true && getHits().isEmpty() *is possible*, in particular if matching entities could not be found in the database.<br> // This methods is mainly useful as a stop condition in loops.<br> boolean hasHits();<br><br> // TODO: javadoc<br> List<H> getHits();<br><br> // TO BE CHECKED: these may not be implementable efficiently.<br> // First, let's check if Elasticsearch returns the total hit count/aggregations to the first search API call when scrolling is enabled.<br> // If it does, let's check the performance impact... Getting this information might require to execute the search query twice, in which case I'd rather not expose this information here and require users to execute the search query twice, explicitly.<br> // Note that *if* we end up implementing these methods, they will return the same data for every single page.<br> long getTotalHitCount();<br> <A> A getAggregation(AggregationKey<A> key);<br><br> // TO BE DISCUSSED: if we add this, it will probably be better to wrap this information into a SearchExecutionMetadata object, and implement getLastExecutionMetadata() here.<br> // As a first step, I would not implement this and would just create a ticket about it.<br> Duration getTook();<br> boolean isTimedOut();<br> <br>}<br>{code}<br><br>h3. To-do list<br><br>In order:<br><br># Add APIs, with stub implementations (throw UnsupportedOperationException( "Not yet implemented" );<br>## Ignore getTotalHitCount/getAggregation/getTook/isTimeout for now.<br># Copy-paste {{org.hibernate.search.integrationtest.backend.tck.search.query.SearchQueryFetchIT}} to {{SearchQueryScrollIT}} and adapt it to test scrolling.<br>## Don't forget to test edge cases: not fetching any result (should work fine), fetching some results but not all of them (should work fine), trying to fetch more than the total hit count (should throw an exception).<br>## Don't forget to check that {{hasMoreHits()}} returns the correct information.<br># Add tests for timeouts (failAfter/truncateAfter) when scrolling.<br># Implement scrolling for the stub backend.<br># Add tests to the ORM mapper. Will probably need to copy/paste {{org.hibernate.search.integrationtest.mapper.orm.search.loading.SearchQueryEntityLoadingBaseIT}} and adapt it to test loading when calling {{scroll()}} instead of just loading when calling {{fetch()}}.<br># Implement scrolling for Elasticsearch.<br>## This should be easy enough: the first call to fetch*() will execute a search work with the {{scroll}} parameter set, the next calls with execute a scroll work (already implemented, see {{org.hibernate.search.elasticsearch.work.impl.factory.ElasticsearchWorkFactory#scroll}}).<br>## On close, we will execute a clearScroll work (already implemented, see {{org.hibernate.search.elasticsearch.work.impl.factory.ElasticsearchWorkFactory#clearScroll}}).<br># Implement scrolling for Lucene.<br>## Search 5 code will not be very useful in that regard, as it addresses a lot of problems that are no longer relevant in Search 6.<br>##</span> <span class="diffremovedchars" style="background-color: #ffe7e7; text-decoration:line-through;"> Basically, in</span> <span class="diffaddedchars" style="background-color:#ddfade;"> In</span> <span class="diffcontext"> the SearchScroll implementation we will need to keep around some of the context that we currently store as local variables in {{LuceneSearcherImpl#search}}: the</span> <span class="diffremovedchars" style="background-color: #ffe7e7; text-decoration:line-through;"> `</span> <span class="diffaddedchars" style="background-color:#ddfade;"> {{</span> <span class="diffcontext">IndexSearcher</span> <span class="diffremovedchars" style="background-color: #ffe7e7; text-decoration:line-through;">`</span> <span class="diffaddedchars" style="background-color:#ddfade;">}}</span> <span class="diffcontext"> and the</span> <span class="diffremovedchars" style="background-color: #ffe7e7; text-decoration:line-through;"> `</span> <span class="diffaddedchars" style="background-color:#ddfade;"> {{</span> <span class="diffcontext">LuceneCollectors</span> <span class="diffremovedchars" style="background-color: #ffe7e7; text-decoration:line-through;">`</span> <span class="diffaddedchars" style="background-color:#ddfade;">}}</span> <span class="diffcontext"> instance in particular.<br>##</span> <span class="diffaddedchars" style="background-color:#ddfade;"> When calling {{next()}}:<br>### First we will need to update the topDocs if necessary: if the topDocs do not include the next page, then update the topDocs<br>#### See {{org.hibernate.search.query.engine.impl.QueryHits#scoreDoc}} for how to decide how many topDocs to retrieve<br>#### See phase 1 in {{org.hibernate.search.backend.lucene.search.extraction.impl.LuceneCollectors#collect}}, but *only phase 1*<br>### Then we will need to collect information for the next page; see the call to {{extractTopDocs}} and phase 2 in {{org.hibernate.search.backend.lucene.search.extraction.impl.LuceneCollectors#collect}}.<br>##</span> <span class="diffcontext"> This may prove difficult, maybe let's organize a pair-programming session for that?<br># Add Lucene-specific extensions to Scrolling<br>## This is mainly necessary for Infinispan<br>## Expose a way to force Lucene to extract TopDocs up to a specific index and retrieve them: {{LuceneSearchScroll#preloadTopDocsUpTo(), returns TopDocs}}<br>## Expose a way to load a specific document specified by its index: {{LuceneSearchScroll#loadHitByIndex(), returns H}}<br>## Maybe we can improve on that later; ideally Infinispan should load multiple hits in one call ({{LuceneSearchScroll#loadHitsByIndex(int ...), returns List<H>}}) otherwise the cost of creating collectors for each retrieved hit will be a bit too much.<br># Implement {{scroll()}} and {{scroll(ScrollMode)}} in {{HibernateOrmSearchQueryAdapter}}, relying on {{scrollAll()}} under the scene.<br>## Only {{ScrollMode.FORWARD_ONLY}} will be supported.<br>## We will need to decide on a page size. Let's use the same size as the loading fetch size, which should be accessible from {{org.hibernate.search.mapper.orm.search.loading.impl.MutableEntityLoadingOptions#getFetchSize}}.<br>## Some internal windowing will probably be necessary. Just copy/paste the {{org.hibernate.search.elasticsearch.util.impl.Window}} class from Search 5 and adapt it. Do not forget to also copy the unit test, {{org.hibernate.search.elasticsearch.test.WindowTest}}.<br>## See {{org.hibernate.search.query.hibernate.impl.ScrollableResultsImpl}} for an example of how it was done in Search 5 (may or may not be helpful).<br># Add tests for {{scroll()}} and {{scroll(ScrollMode)}} in {{org.hibernate.search.integrationtest.mapper.orm.hibernateormapis.ToHibernateOrmIT}}:<br>## Nominal case (create scroll, fetch some hits until all hits have been consumed, close).<br>## Edge cases: not fetching any result (should work fine), fetching some results but not all of them (should work fine), trying to fetch more than the total hit count (should throw an exception).<br>## Error cases: trying to scroll back, trying to call the {{get*(int)}} methods...<br>## Check that using any scroll mode other than ScrollMode.FORWARD_ONLY fails.<br>## Test {{query.stream()}} too (it's based on {{scroll()}}).<br># Add tests for {{getResultStream()}} in {{org.hibernate.search.integrationtest.mapper.orm.hibernateormapis.ToJpaIT}}.<br># Allow backends to extend the SearchScroll interfaces, like they currently do with {{SearchQuery}} ({{ElasticsearchSearchQuery}}, {{LuceneSearchQuery}}):<br>## Add a generic parameter {{S extends SearchScroll<H>}} to {{ExtendedSearchFetchable}} and override its {{scroll}} methods to return that type.<br>## Adapt the interfaces that extend {{ExtendedSearchFetchable}} as necessary.<br>## Create a new {{ExtendedSearchScroll<H>}} interface using the same principle.<br>## Create specific interfaces for Elasticsearch and Lucene: {{ElasticsearchSearchScroll}} and {{LuceneSearchScroll}}.<br>## Implement these interfaces where appropriate.<br>## Test extensions for Lucene and Elasticsearch. Mainly, check that the scroll has the correct type. See how it's done for SearchResult in {{org.hibernate.search.integrationtest.backend.elasticsearch.ElasticsearchExtensionIT#query}}.<br># Add getTotalHitCount/getAggregation to APIs if relevant and implement them.<br># Add getTook/isTimeout to APIs if relevant and implement them.</span> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td class="email-content-main mobile-expand " style="padding: 0; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff" bgcolor="#ffffff"> <script type="application/ld+json">
{
"@context": "http://schema.org",
"@type": "EmailMessage",
"description": "View Issue",
"potentialAction": {
"@type": "ViewAction",
"target": "https://hibernate.atlassian.net/browse/HSEARCH-3323?inbox=true&",
"name": "View Issue"
},
"publisher": {
"@type": "Organization",
"name": "Atlassian",
"url": "https://www.atlassian.com"
}
}
</script>
<table id="actions-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 1px">
<tbody>
<tr>
<td id="actions-pattern-container" valign="middle" style="padding: 0; border-collapse: collapse; padding: 10px 0 10px 24px; vertical-align: middle; padding-left: 0">
<table align="left" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0">
<tbody>
<tr>
<td class="actions-pattern-action-icon-container" style="padding: 0; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 0; vertical-align: middle"> <a href="https://hibernate.atlassian.net/browse/HSEARCH-3323#add-comment?atlOrigin=eyJpIjoiZWJiZDk3NWU3OTIxNDYzOGI5ZGY5MmY3YjY3OTNmY2YiLCJwIjoiaiJ9" target="_blank" title="Add Comment" style="color: #3b73af; text-decoration: none"> <img class="actions-pattern-action-icon-image" src="cid:jira-generated-image-static-comment-icon-5b8ddf58-97fa-4744-b642-a0b28a90dfd8" alt="Add Comment" title="Add Comment" height="16" width="16" border="0" style="vertical-align: middle"> </a> </td>
<td class="actions-pattern-action-text-container" style="padding: 0; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 4px; padding-left: 5px"> <a href="https://hibernate.atlassian.net/browse/HSEARCH-3323#add-comment?atlOrigin=eyJpIjoiZWJiZDk3NWU3OTIxNDYzOGI5ZGY5MmY3YjY3OTNmY2YiLCJwIjoiaiJ9" target="_blank" title="Add Comment" style="color: #3b73af; text-decoration: none">Add Comment</a> </td>
</tr>
</tbody>
</table> </td>
</tr>
</tbody>
</table> </td>
</tr> <!-- there needs to be content in the cell for it to render in some clients -->
<tr>
<td class="email-content-rounded-bottom mobile-expand" style="padding: 0; border-collapse: collapse; color: #ffffff; padding: 0 15px 0 16px; height: 5px; line-height: 5px; background-color: #ffffff; border-top: 0; border-left: 1px solid #cccccc; border-bottom: 1px solid #cccccc; border-right: 1px solid #cccccc; border-bottom-right-radius: 5px; border-bottom-left-radius: 5px; mso-line-height-rule: exactly" height="5" bgcolor="#ffffff"> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td id="footer-pattern" style="padding: 0; border-collapse: collapse; padding: 12px 20px">
<table id="footer-pattern-container" cellspacing="0" cellpadding="0" border="0" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0">
<tbody>
<tr>
<td id="footer-pattern-mobile-separated-links" class="mobile-resize-text" width="100%" colspan="2" style="padding: 0; border-collapse: collapse; color: #999999; font-size: 12px; line-height: 18px; font-family: Arial, sans-serif; mso-line-height-rule: exactly; mso-text-raise: 2px"> Get Jira notifications on your phone! Download the Jira Cloud app for <a href="https://play.google.com/store/apps/details?id=com.atlassian.android.jira.core&referrer=utm_source%3DNotificationLink%26utm_medium%3DEmail" style="color: #3b73af; text-decoration: none">Android</a> or <a href="https://itunes.apple.com/app/apple-store/id1006972087?pt=696495&ct=EmailNotificationLink&mt=8" style="color: #3b73af; text-decoration: none">iOS</a>
<hr> </td>
</tr>
<tr>
<td id="footer-pattern-text" class="mobile-resize-text" width="100%" style="padding: 0; border-collapse: collapse; color: #999999; font-size: 12px; line-height: 18px; font-family: Arial, sans-serif; mso-line-height-rule: exactly; mso-text-raise: 2px"> This message was sent by Atlassian Jira <span id="footer-build-information">(v1001.0.0-SNAPSHOT#100126-<span title="053c92411572c3fb53ddf3c3d5467e48193c9b08" data-commit-id="053c92411572c3fb53ddf3c3d5467e48193c9b08}">sha1:053c924</span>)</span> </td>
<td id="footer-pattern-logo-desktop-container" valign="top" style="padding: 0; border-collapse: collapse; padding-left: 20px; vertical-align: top">
<table style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0">
<tbody>
<tr>
<td id="footer-pattern-logo-desktop-padding" style="padding: 0; border-collapse: collapse; padding-top: 3px; opacity: 0.150"> <img id="footer-pattern-logo-desktop" src="cid:jira-generated-image-static-footer-desktop-logo-04eb620a-e389-412d-b1b2-d2b7a92fb492" alt="Atlassian logo" title="Atlassian logo" width="192" height="24" class="image_fix"> </td>
</tr>
</tbody>
</table> </td>
</tr>
</tbody>
</table> </td>
</tr>
</tbody>
</table>
</body>
</html>