Hi all, we are exposing a nice feature inherited from the Search engine via the "simple" DSL version, the one which is also available via Hot Rod: org.infinispan.query.dsl.Query.getResultSize() To be fair I hadn't noticed we do expose this, I just noticed after a recent PR review and I found it surprising. This method returns the size of the full resultset, disregarding pagination options; you can imagine it fit for situations like: "found 6 million matches, these are the top 20: " A peculiarity of Hibernate Search is that the total number of matches is extremely cheap to figure out as it's generally a side effect of finding the 20 results. Essentially we're just exposing an int value which was already computed: very cheap, and happens to be useful in practice. This is not the case with a SQL statement, in this case you'd have to craft 2 different SQL statements, often incurring the cost of 2 round trips to the database. So this getResultSize() is not available on the Hibernate ORM Query, only on our FullTextQuery extension. Now my doubt is if it is indeed a wise move to expose this method on the simplified DSL. Of course some people might find it useful, still I'm wondering how much we'll be swearing at needing to maintain this feature vs its usefulness when we'll implement alternative execution engines to run queries, not least on Map/Reduce based filtering, and ultimately hybrid strategies. In case of Map/Reduce I think we'll need to keep track of possible de-duplication of results, in case of a Teiid integration it might need a second expensive query; so in this case I'd expect this method to be lazily evaluated. Should we rather remove this functionality? Sanne _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Ok you make some good points, and I've no doubts of it being useful. My only concern is that this could slow us down significantly in providing other features which might be even more useful or pressing. You have to pick your battles and be wise on where to spend energy first. Considering that it's easier to add methods than to remove them, what would you think of marking this as experimental for now? I'd prefer to see the non-indexed query engine delivered first; this sounds like being a stone on the critical path so it might be wise to have the option to drop the requirement from a first implementation. Definitely you're right that we should then implement "some" COUNT strategy, I'm just not comfortable in committing on this one yet.

Now on a general purpose COUNT: for sure we need one but it's a pandora's box you're opening. In a sense there is a parallelism conceptually with my concerns on the API contract we provide for the clear() method. too keep it short in this context as we're changing subject, I don't think we'll ever be able to provide a solid guarantee of a fully reliable value: indexes are not updated in transaction yet, and M/R does cross boundaries of nodes and datacontainer/cachestore without making a consistent read snapshot. We should document any such API as to providing a best effort estimate.

On 10 March 2014 13:16, Adrian Nistor <anistor(a)redhat.com&gt; wrote: > I'd vote for keeping it, and executing it lazily in environments where it is > costly to compute it upfront. > > And off course, document this properly so users will be aware it can incur a > second execution, with significant performance impact and also possibly a > data visibility/consistency impact. I'd do this because the api is meant to > be first of all user friendly and useful, not just machine friendly and > efficient. > > There's another reason for having it. Say we remove it, how will users be > able to know the total number of matching results? Our DSL does not > currently have a 'count' function. Maybe we should add such a thing first, > and then think about removing Query.getResultsSize(). > > But, if we implement a proper 'count', getResultsSize() could be trivially > implemented as some kind of syntactic sugar on top of it, so I would still > consider it worth being in the API. > > And then it all boils down to the question: should the DSL provide a count > function? (+1 from me) > > Cheers > > > On 03/10/2014 02:23 PM, Sanne Grinovero wrote: > > Hi all, > we are exposing a nice feature inherited from the Search engine via > the "simple" DSL version, the one which is also available via Hot Rod: > > org.infinispan.query.dsl. > Query.getResultSize() > > To be fair I hadn't noticed we do expose this, I just noticed after a > recent PR review and I found it surprising. > > This method returns the size of the full resultset, disregarding > pagination options; you can imagine it fit for situations like: > > "found 6 million matches, these are the top 20: " > > A peculiarity of Hibernate Search is that the total number of matches > is extremely cheap to figure out as it's generally a side effect of > finding the 20 results. Essentially we're just exposing an int value > which was already computed: very cheap, and happens to be useful in > practice. > > This is not the case with a SQL statement, in this case you'd have to > craft 2 different SQL statements, often incurring the cost of 2 round > trips to the database. So this getResultSize() is not available on the > Hibernate ORM Query, only on our FullTextQuery extension. > > Now my doubt is if it is indeed a wise move to expose this method on > the simplified DSL. Of course some people might find it useful, still > I'm wondering how much we'll be swearing at needing to maintain this > feature vs its usefulness when we'll implement alternative execution > engines to run queries, not least on Map/Reduce based filtering, and > ultimately hybrid strategies. > > In case of Map/Reduce I think we'll need to keep track of possible > de-duplication of results, in case of a Teiid integration it might > need a second expensive query; so in this case I'd expect this method > to be lazily evaluated. > > Should we rather remove this functionality? > > Sanne > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 10 March 2014 17:09, Mircea Markus <mmarkus(a)redhat.com&gt; wrote: > On Mar 10, 2014, at 15:12, Sanne Grinovero <sanne(a)infinispan.org&gt; wrote: > >> Ok you make some good points, and I've no doubts of it being useful. >> >> My only concern is that this could slow us down significantly in >> providing other features which might be even more useful or pressing. >> You have to pick your battles and be wise on where to spend energy >> first. >> >> Considering that it's easier to add methods than to remove them, what >> would you think of marking this as experimental for now? >> I'd prefer to see the non-indexed query engine delivered first; this >> sounds like being a stone on the critical path so it might be wise to >> have the option to drop the requirement from a first implementation. >> Definitely you're right that we should then implement "some" COUNT >> strategy, I'm just not comfortable in committing on this one yet. > I can imagine a lot of users emulating this by simply iterating over the entries in the result set. Even if we do just that and document it as slow, I think it's still worth exposing this somewhere. I'm not questioning it to be useful. But the implementation is tricky, for example simply iterating would require a global (and distributed) lock to be accurate. Otherwise we can only document it as an approximation, and worst is we can't even estimate an error margin: under steady load it would probably be a reasonable estimation, but there are corner cases in which you can get off by several orders of magnitude. Among others, your assumptions would need to include: - no nodes failing (no state transfers happening) - no write spikes (probably one of the best reasons to deploy infinispan is to be able to absorb spikes) - no wild churning across datastore/cachestores - expirations happening in a homogeneus pattern (this entirely depends on the use case) Also there is no way to make this work in the context of a transaction, as it would pretty much violate any promise of repeatable read properties. So the question is, if a user would still consider it useful after (hopefully) understanding all the attached strings. In other contexts we discussed the need for Infinispan to provide something like a snapshot capability based on TOA. If we had that, we could implement a count operation on top of it. Sanne _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

I'd vote for keeping it, and executing it lazily in environments where it is costly to compute it upfront.

And off course, document this properly so users will be aware it can incur a second execution, with significant performance impact and also possibly a data visibility/consistency impact. I'd do this because the api is meant to be first of all user friendly and useful, not just machine friendly and efficient. There's another reason for having it. Say we remove it, how will users be able to know the total number of matching results? Our DSL does not currently have a 'count' function. Maybe we should add such a thing first, and then think about removing Query.getResultsSize(). But, if we implement a proper 'count', getResultsSize() could be trivially implemented as some kind of syntactic sugar on top of it, so I would still consider it worth being in the API. And then it all boils down to the question: should the DSL provide a count function? (+1 from me) Cheers On 03/10/2014 02:23 PM, Sanne Grinovero wrote: > Hi all, > we are exposing a nice feature inherited from the Search engine via > the "simple" DSL version, the one which is also available via Hot Rod: > > org.infinispan.query.dsl.Query.getResultSize() > > To be fair I hadn't noticed we do expose this, I just noticed after a > recent PR review and I found it surprising. > > This method returns the size of the full resultset, disregarding > pagination options; you can imagine it fit for situations like: > > "found 6 million matches, these are the top 20: " > > A peculiarity of Hibernate Search is that the total number of matches > is extremely cheap to figure out as it's generally a side effect of > finding the 20 results. Essentially we're just exposing an int value > which was already computed: very cheap, and happens to be useful in > practice. > > This is not the case with a SQL statement, in this case you'd have to > craft 2 different SQL statements, often incurring the cost of 2 round > trips to the database. So this getResultSize() is not available on the > Hibernate ORM Query, only on our FullTextQuery extension. > > Now my doubt is if it is indeed a wise move to expose this method on > the simplified DSL. Of course some people might find it useful, still > I'm wondering how much we'll be swearing at needing to maintain this > feature vs its usefulness when we'll implement alternative execution > engines to run queries, not least on Map/Reduce based filtering, and > ultimately hybrid strategies. > > In case of Map/Reduce I think we'll need to keep track of possible > de-duplication of results, in case of a Teiid integration it might > need a second expensive query; so in this case I'd expect this method > to be lazily evaluated. > > Should we rather remove this functionality? > > Sanne > _______________________________________________ > infinispan-dev mailing list > > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

I’m struggling with this same question in ModeShape. The JCR API exposes a method that returns the number of results, but at least the spec allows the implementation to return -1 if the size is not known (or very expensive to compute). Yet this still does not satisfy all cases. Depending upon the technology, computing the **exact size** ranges from very cheap to extremely expensive to calculate. For example, consider a system that has to take into account access control limitations of the user. My current opinion is that few applications actually need an exact size, and if they do there may be alternatives (like counting as they iterate over the results). An alternative is to expose an **approximate size**, which is likely to be sufficient for generating display or other pre-computed information such as links or paging details. I think that this is sufficient for most needs, and that even an order of magnitude is sufficient. When the results are known to be small, the system might want to determine the exact size (e.g., by iterating). So one option is to expose both methods, but allow the exact size method to return -1 if the system can’t determine the size or if doing so is very expensive. This allows the system a way out for large/complex queries and flexibility in the implementation technology. The approximate size method probably always needs to return at least some usable value. BTW, computing an exact size by iterating can be expensive unless you can keep all the results in memory. That’s not ideal - a query with large results could fill up available memory. If you don’t keep all results in memory, then if you’re going to allow clients to access the results more than once you have to provide a way to buffer the results. On Mar 10, 2014, at 7:23 AM, Sanne Grinovero <sanne(a)infinispan.org&gt; wrote: > Hi all, > we are exposing a nice feature inherited from the Search engine via > the "simple" DSL version, the one which is also available via Hot Rod: > > org.infinispan.query.dsl.Query.getResultSize() > > To be fair I hadn't noticed we do expose this, I just noticed after a > recent PR review and I found it surprising. > > This method returns the size of the full resultset, disregarding > pagination options; you can imagine it fit for situations like: > > "found 6 million matches, these are the top 20: " > > A peculiarity of Hibernate Search is that the total number of matches > is extremely cheap to figure out as it's generally a side effect of > finding the 20 results. Essentially we're just exposing an int value > which was already computed: very cheap, and happens to be useful in > practice. > > This is not the case with a SQL statement, in this case you'd have to > craft 2 different SQL statements, often incurring the cost of 2 round > trips to the database. So this getResultSize() is not available on the > Hibernate ORM Query, only on our FullTextQuery extension. > > Now my doubt is if it is indeed a wise move to expose this method on > the simplified DSL. Of course some people might find it useful, still > I'm wondering how much we'll be swearing at needing to maintain this > feature vs its usefulness when we'll implement alternative execution > engines to run queries, not least on Map/Reduce based filtering, and > ultimately hybrid strategies. > > In case of Map/Reduce I think we'll need to keep track of possible > de-duplication of results, in case of a Teiid integration it might > need a second expensive query; so in this case I'd expect this method > to be lazily evaluated. > > Should we rather remove this functionality? > > Sanne > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

I disagree. Most developers have access to the JavaDoc, and if even moderately well-written, they will find out what the method returns and when. It’s no different than a method sometimes returning null rather than an object reference. On Mar 11, 2014, at 12:16 PM, Dennis Reed <dereed(a)redhat.com&gt; wrote: > Providing methods that work sometimes and don't work other times is > generally a bad idea. > > No matter how much you document it, users *will* try to use it and > expect it to always work > (either because they didn't read the docs that say otherwise, they think > they'll stick to a configuration where it does work, etc.) > > And then when it doesn't work (because they pushed something to > production which has a different configuration than dev, etc) > it's a frustrating experience. > > -Dennis > > On 03/11/2014 09:37 AM, Randall Hauch wrote: >> I’m struggling with this same question in ModeShape. The JCR API exposes a method that returns the number of results, but at least the spec allows the implementation to return -1 if the size is not known (or very expensive to compute). Yet this still does not satisfy all cases. >> >> Depending upon the technology, computing the **exact size** ranges from very cheap to extremely expensive to calculate. For example, consider a system that has to take into account access control limitations of the user. My current opinion is that few applications actually need an exact size, and if they do there may be alternatives (like counting as they iterate over the results). >> >> An alternative is to expose an **approximate size**, which is likely to be sufficient for generating display or other pre-computed information such as links or paging details. I think that this is sufficient for most needs, and that even an order of magnitude is sufficient. When the results are known to be small, the system might want to determine the exact size (e.g., by iterating). >> >> So one option is to expose both methods, but allow the exact size method to return -1 if the system can’t determine the size or if doing so is very expensive. This allows the system a way out for large/complex queries and flexibility in the implementation technology. The approximate size method probably always needs to return at least some usable value. >> >> BTW, computing an exact size by iterating can be expensive unless you can keep all the results in memory. That’s not ideal - a query with large results could fill up available memory. If you don’t keep all results in memory, then if you’re going to allow clients to access the results more than once you have to provide a way to buffer the results. >> >> >> On Mar 10, 2014, at 7:23 AM, Sanne Grinovero <sanne(a)infinispan.org&gt; wrote: >> >>> Hi all, >>> we are exposing a nice feature inherited from the Search engine via >>> the "simple" DSL version, the one which is also available via Hot Rod: >>> >>> org.infinispan.query.dsl.Query.getResultSize() >>> >>> To be fair I hadn't noticed we do expose this, I just noticed after a >>> recent PR review and I found it surprising. >>> >>> This method returns the size of the full resultset, disregarding >>> pagination options; you can imagine it fit for situations like: >>> >>> "found 6 million matches, these are the top 20: " >>> >>> A peculiarity of Hibernate Search is that the total number of matches >>> is extremely cheap to figure out as it's generally a side effect of >>> finding the 20 results. Essentially we're just exposing an int value >>> which was already computed: very cheap, and happens to be useful in >>> practice. >>> >>> This is not the case with a SQL statement, in this case you'd have to >>> craft 2 different SQL statements, often incurring the cost of 2 round >>> trips to the database. So this getResultSize() is not available on the >>> Hibernate ORM Query, only on our FullTextQuery extension. >>> >>> Now my doubt is if it is indeed a wise move to expose this method on >>> the simplified DSL. Of course some people might find it useful, still >>> I'm wondering how much we'll be swearing at needing to maintain this >>> feature vs its usefulness when we'll implement alternative execution >>> engines to run queries, not least on Map/Reduce based filtering, and >>> ultimately hybrid strategies. >>> >>> In case of Map/Reduce I think we'll need to keep track of possible >>> de-duplication of results, in case of a Teiid integration it might >>> need a second expensive query; so in this case I'd expect this method >>> to be lazily evaluated. >>> >>> Should we rather remove this functionality? >>> >>> Sanne >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

what about we call it int getEstimatedResultSize() ? Having such a method occasionally return null looks very bad to me, I'd rather remove the functionality. -- Sanne On 11 March 2014 19:08, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; wrote: > I agree with Randall. > > I tend to be very conservative about my public APIs. And offering an API that I think will block me in the future is something I tend to avoid. > > Something like .guessNbrOfMatchingElements() / .guessResultSize() would provide a better clue about the gamble the user takes. Note that the size is irrespective of the pagination applied which renders this result quite cool even if approximate. > > I’d be tempted not to put getResultSize() with an exact value in the public contract as iterating is probably going to as “fast”. > > An alternative is something like that (needs to be refined though) > > /** > * Get the result size. > * Approximate results are to be preferred as it is usually very cheap to compute. > * If the computation is too expensive, the approximate accuracy returns null. > * > * Exact results are likely to be costly and require two queries. > */ > Integer getResultSize(Accuracy); > enum Accuracy { EXACT, APPROXIMATE_OR_NULL } > > Emmanuel > > On 11 Mar 2014, at 18:23, Randall Hauch <rhauch(a)redhat.com&gt; wrote: > >> I disagree. Most developers have access to the JavaDoc, and if even moderately well-written, they will find out what the method returns and when. It’s no different than a method sometimes returning null rather than an object reference. >> >> On Mar 11, 2014, at 12:16 PM, Dennis Reed <dereed(a)redhat.com&gt; wrote: >> >>> Providing methods that work sometimes and don't work other times is >>> generally a bad idea. >>> >>> No matter how much you document it, users *will* try to use it and >>> expect it to always work >>> (either because they didn't read the docs that say otherwise, they think >>> they'll stick to a configuration where it does work, etc.) >>> >>> And then when it doesn't work (because they pushed something to >>> production which has a different configuration than dev, etc) >>> it's a frustrating experience. >>> >>> -Dennis >>> >>> On 03/11/2014 09:37 AM, Randall Hauch wrote: >>>> I’m struggling with this same question in ModeShape. The JCR API exposes a method that returns the number of results, but at least the spec allows the implementation to return -1 if the size is not known (or very expensive to compute). Yet this still does not satisfy all cases. >>>> >>>> Depending upon the technology, computing the **exact size** ranges from very cheap to extremely expensive to calculate. For example, consider a system that has to take into account access control limitations of the user. My current opinion is that few applications actually need an exact size, and if they do there may be alternatives (like counting as they iterate over the results). >>>> >>>> An alternative is to expose an **approximate size**, which is likely to be sufficient for generating display or other pre-computed information such as links or paging details. I think that this is sufficient for most needs, and that even an order of magnitude is sufficient. When the results are known to be small, the system might want to determine the exact size (e.g., by iterating). >>>> >>>> So one option is to expose both methods, but allow the exact size method to return -1 if the system can’t determine the size or if doing so is very expensive. This allows the system a way out for large/complex queries and flexibility in the implementation technology. The approximate size method probably always needs to return at least some usable value. >>>> >>>> BTW, computing an exact size by iterating can be expensive unless you can keep all the results in memory. That’s not ideal - a query with large results could fill up available memory. If you don’t keep all results in memory, then if you’re going to allow clients to access the results more than once you have to provide a way to buffer the results. >>>> >>>> >>>> On Mar 10, 2014, at 7:23 AM, Sanne Grinovero <sanne(a)infinispan.org&gt; wrote: >>>> >>>>> Hi all, >>>>> we are exposing a nice feature inherited from the Search engine via >>>>> the "simple" DSL version, the one which is also available via Hot Rod: >>>>> >>>>> org.infinispan.query.dsl.Query.getResultSize() >>>>> >>>>> To be fair I hadn't noticed we do expose this, I just noticed after a >>>>> recent PR review and I found it surprising. >>>>> >>>>> This method returns the size of the full resultset, disregarding >>>>> pagination options; you can imagine it fit for situations like: >>>>> >>>>> "found 6 million matches, these are the top 20: " >>>>> >>>>> A peculiarity of Hibernate Search is that the total number of matches >>>>> is extremely cheap to figure out as it's generally a side effect of >>>>> finding the 20 results. Essentially we're just exposing an int value >>>>> which was already computed: very cheap, and happens to be useful in >>>>> practice. >>>>> >>>>> This is not the case with a SQL statement, in this case you'd have to >>>>> craft 2 different SQL statements, often incurring the cost of 2 round >>>>> trips to the database. So this getResultSize() is not available on the >>>>> Hibernate ORM Query, only on our FullTextQuery extension. >>>>> >>>>> Now my doubt is if it is indeed a wise move to expose this method on >>>>> the simplified DSL. Of course some people might find it useful, still >>>>> I'm wondering how much we'll be swearing at needing to maintain this >>>>> feature vs its usefulness when we'll implement alternative execution >>>>> engines to run queries, not least on Map/Reduce based filtering, and >>>>> ultimately hybrid strategies. >>>>> >>>>> In case of Map/Reduce I think we'll need to keep track of possible >>>>> de-duplication of results, in case of a Teiid integration it might >>>>> need a second expensive query; so in this case I'd expect this method >>>>> to be lazily evaluated. >>>>> >>>>> Should we rather remove this functionality? >>>>> >>>>> Sanne >>>>> _______________________________________________ >>>>> infinispan-dev mailing list >>>>> infinispan-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 11 mars 2014, at 21:59, Randall Hauch <rhauch(a)redhat.com&gt; wrote: Maybe a Long rather than an Integer? Ints are so last year. :-) And, what about using a primitive that returns -1 when the method cannot determine the size (if allowed by the parameter). Just as easy to check -1 than it is to check null, IMO. > On Mar 11, 2014, at 2:21 PM, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; wrote: > > It does not work, I think, because if you implement your query via some map reduce and you do pagination, it will be costly to compute the size and you might want not to return it. > Hence my Accuracy idea to clarify the intend to the API user. > >> On 11 Mar 2014, at 19:18, Sanne Grinovero <sanne(a)infinispan.org&gt; wrote: >> >> what about we call it >> >> int getEstimatedResultSize() ? >> >> Having such a method occasionally return null looks very bad to me, >> I'd rather remove the functionality. >> >> -- Sanne >> >>> On 11 March 2014 19:08, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; wrote: >>> I agree with Randall. >>> >>> I tend to be very conservative about my public APIs. And offering an API that I think will block me in the future is something I tend to avoid. >>> >>> Something like .guessNbrOfMatchingElements() / .guessResultSize() would provide a better clue about the gamble the user takes. Note that the size is irrespective of the pagination applied which renders this result quite cool even if approximate. >>> >>> I’d be tempted not to put getResultSize() with an exact value in the public contract as iterating is probably going to as “fast”. >>> >>> An alternative is something like that (needs to be refined though) >>> >>> /** >>> * Get the result size. >>> * Approximate results are to be preferred as it is usually very cheap to compute. >>> * If the computation is too expensive, the approximate accuracy returns null. >>> * >>> * Exact results are likely to be costly and require two queries. >>> */ >>> Integer getResultSize(Accuracy); >>> enum Accuracy { EXACT, APPROXIMATE_OR_NULL } >>> >>> Emmanuel >>> >>>> On 11 Mar 2014, at 18:23, Randall Hauch <rhauch(a)redhat.com&gt; wrote: >>>> >>>> I disagree. Most developers have access to the JavaDoc, and if even moderately well-written, they will find out what the method returns and when. It’s no different than a method sometimes returning null rather than an object reference. >>>> >>>>> On Mar 11, 2014, at 12:16 PM, Dennis Reed <dereed(a)redhat.com&gt; wrote: >>>>> >>>>> Providing methods that work sometimes and don't work other times is >>>>> generally a bad idea. >>>>> >>>>> No matter how much you document it, users *will* try to use it and >>>>> expect it to always work >>>>> (either because they didn't read the docs that say otherwise, they think >>>>> they'll stick to a configuration where it does work, etc.) >>>>> >>>>> And then when it doesn't work (because they pushed something to >>>>> production which has a different configuration than dev, etc) >>>>> it's a frustrating experience. >>>>> >>>>> -Dennis >>>>> >>>>>> On 03/11/2014 09:37 AM, Randall Hauch wrote: >>>>>> I’m struggling with this same question in ModeShape. The JCR API exposes a method that returns the number of results, but at least the spec allows the implementation to return -1 if the size is not known (or very expensive to compute). Yet this still does not satisfy all cases. >>>>>> >>>>>> Depending upon the technology, computing the **exact size** ranges from very cheap to extremely expensive to calculate. For example, consider a system that has to take into account access control limitations of the user. My current opinion is that few applications actually need an exact size, and if they do there may be alternatives (like counting as they iterate over the results). >>>>>> >>>>>> An alternative is to expose an **approximate size**, which is likely to be sufficient for generating display or other pre-computed information such as links or paging details. I think that this is sufficient for most needs, and that even an order of magnitude is sufficient. When the results are known to be small, the system might want to determine the exact size (e.g., by iterating). >>>>>> >>>>>> So one option is to expose both methods, but allow the exact size method to return -1 if the system can’t determine the size or if doing so is very expensive. This allows the system a way out for large/complex queries and flexibility in the implementation technology. The approximate size method probably always needs to return at least some usable value. >>>>>> >>>>>> BTW, computing an exact size by iterating can be expensive unless you can keep all the results in memory. That’s not ideal - a query with large results could fill up available memory. If you don’t keep all results in memory, then if you’re going to allow clients to access the results more than once you have to provide a way to buffer the results. >>>>>> >>>>>> >>>>>>> On Mar 10, 2014, at 7:23 AM, Sanne Grinovero <sanne(a)infinispan.org&gt; wrote: >>>>>>> >>>>>>> Hi all, >>>>>>> we are exposing a nice feature inherited from the Search engine via >>>>>>> the "simple" DSL version, the one which is also available via Hot Rod: >>>>>>> >>>>>>> org.infinispan.query.dsl.Query.getResultSize() >>>>>>> >>>>>>> To be fair I hadn't noticed we do expose this, I just noticed after a >>>>>>> recent PR review and I found it surprising. >>>>>>> >>>>>>> This method returns the size of the full resultset, disregarding >>>>>>> pagination options; you can imagine it fit for situations like: >>>>>>> >>>>>>> "found 6 million matches, these are the top 20: " >>>>>>> >>>>>>> A peculiarity of Hibernate Search is that the total number of matches >>>>>>> is extremely cheap to figure out as it's generally a side effect of >>>>>>> finding the 20 results. Essentially we're just exposing an int value >>>>>>> which was already computed: very cheap, and happens to be useful in >>>>>>> practice. >>>>>>> >>>>>>> This is not the case with a SQL statement, in this case you'd have to >>>>>>> craft 2 different SQL statements, often incurring the cost of 2 round >>>>>>> trips to the database. So this getResultSize() is not available on the >>>>>>> Hibernate ORM Query, only on our FullTextQuery extension. >>>>>>> >>>>>>> Now my doubt is if it is indeed a wise move to expose this method on >>>>>>> the simplified DSL. Of course some people might find it useful, still >>>>>>> I'm wondering how much we'll be swearing at needing to maintain this >>>>>>> feature vs its usefulness when we'll implement alternative execution >>>>>>> engines to run queries, not least on Map/Reduce based filtering, and >>>>>>> ultimately hybrid strategies. >>>>>>> >>>>>>> In case of Map/Reduce I think we'll need to keep track of possible >>>>>>> de-duplication of results, in case of a Teiid integration it might >>>>>>> need a second expensive query; so in this case I'd expect this method >>>>>>> to be lazily evaluated. >>>>>>> >>>>>>> Should we rather remove this functionality? >>>>>>> >>>>>>> Sanne >>>>>>> _______________________________________________ >>>>>>> infinispan-dev mailing list >>>>>>> infinispan-dev(a)lists.jboss.org >>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>> >>>>>> _______________________________________________ >>>>>> infinispan-dev mailing list >>>>>> infinispan-dev(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>> >>>>> _______________________________________________ >>>>> infinispan-dev mailing list >>>>> infinispan-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>>> >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev