| Guillaume Smet Chris Cranford I just investigated the Scroll API to see if we could wire the ScrollableResults from a FullTextQuery (fullTextQuery.scroll()) to some object taking advantage of Elasticsearch's Scroll API. Well, we cannot, because ScrollableResults offers far more methods than what Elasticsearch provides. So using the Scroll API to implement ScrollableResults would mean throwing UnsupportedOperationException in most methods.
Implementing the basic queries (getResultList()/list(), with an offset and a maximum number of results) is not possible either with the Scroll API, or at least not in an efficient way: the scroll API does not allow using an offset (the from attribute is ignored), so we would have to scroll through every previous result each time a user uses an offset. For the same performance reasons, we cannot use the Scroll API as a fallback when Elasticsearch throws a "Result window is too large" error at us. Also to be noted, increasing the value of index.max_result_window seems to be discouraged for performance reasons: https://www.elastic.co/guide/en/elasticsearch/reference/2.4/breaking_21_search_changes.html#_from_size_limits Here are the solutions:
- implementing the pre-existing Hibernate ORM scroll() method in such a way that it will work as usual with the Lucene backend, will also work (with unlimited scrolling) with the Elasticsearch backend, but that the scrollable results with the Elasticsearch backend will through UnsupportedOperationException in most methods (previous(), last(), setRowNumber(int), ...)
- implementing the pre-existing Hibernate ORM scroll() fully for both the Lucene and Elasticsearch backend, using horribly inefficient workarounds for methods not supported by the Elasticsearch Scroll API (previous(), last(), setRowNumber(int), ...).
Personally, I'd be in favor of solution 1. Offering inefficient methods in an API that's primarily aiming at processing large datasets efficiently seems a nonsense to me. But some implementors chose to do just that, like H2, so... In any case, this will require non-trivial SPI additions (most notably a "scroll" method in org.hibernate.search.query.engine.spi.HSQuery). This might mean that the fix will only be merged in 6.0; I'll have to check with Sanne, I guess. I'm starting the work on solution 1. Feel free to ping me if you disagree with the whole approach (better now than when I submit a PR  ) |