Hi, we've found the following problem regarding projection queries when dealing with documents containing multiple fields with the same name. Let's say we add field "foo" with two different values to the same document: luceneOptions.addFieldToDocument("foo", "aaa", document); luceneOptions.addFieldToDocument("foo", "bbb", document); If we now do a projection query on field "foo", one would expect the resultset to contain exactly two results ({"aaa"} and {"bbb"}), but HSearch returns only a single result (the property value of the result is either "aaa" or "bbb", because Document.getFieldable("foo"), which is called in o.h.search.engine.impl.DocumentBuilderHelper, returns the first field that matches the given name). DocumentExtractor.extract() returns a single EntityInfo, but in order for it to properly handle projections as described in the previous paragraph, it should really be modified to return List<EntityInfo>. This sounds pretty reasonable when the query is projecting only a single field. When projecting multiple multi-valued fields, the resultset should actually return a cartesian product. This is one way of doing it. The other way of doing it is if we consider multiple fields with the same name as a single multi-valued field. When projecting such fields, the resultset would contain the same number of results as there are matching documents, with the projected value being a collection of all the values stored in the field. Actually, in CapeDwarf we need the cartesian product, as this is the way Google AppEngine does it. What do you guys think? Marko _______________________________________________ hibernate-dev mailing list hibernate-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev

I don't think it's as simple as you imply. If we go that route the engine atop Hibernate Search would need to know that a given field is multivaluable (it could be a serialized List otherwise) and look into each entry in the Object[] to "cartesianize it". It's doable but it seems that it would be easier if Hibernate Search does the work. But I don't see that as being the default value. Depending on the situation you want: - an array entry with a List in your Object[] representing a row - or you want n entries in your List<Object[]> with duplicated values in the Object[] except for the multivalued element Is the current behavior good for anything? We certainly did not design it with multivalue fields stored in mind. So we might want to allow for setting a given value globally with ways to override that per association. Thoughts? Emmanuel On Thu 2013-01-03 14:42, Sanne Grinovero wrote: > Hi Marko, > this is expected by our typical users, as you only have multiple field > values on tokenized fields, and you won't project these; occasionally > someone uses the _addFieldToDocument_ multiple times to give the > illusion of merging multiple strings to be tokenized in the same > stream, or occasionally even if you are applying an analyzer to the > field you just know for sure the output element is single, so we don't > enforce it. > > Projection on the other hand can't be applied on all fields, it really > is expected on Stored fields only - and typically one stored the field > only once. > > We can discuss how to improve this for your use case but I'd like to > better understand what you're needing: > I don't think you would need to change EntityInfo to List<EntityInfo> > : it still represents a *single* Document which matched your search > criteria, it looks like what you need is that one of the projected > fields is actually a multivalued element; but this would still be an > element of the same and only EntityInfo. > > This implies that, since the return type of a projection is Object[], > there is no need to break any API to implement such a feature: one of > those Object elements could be a Set or an array. > > Also consider there is no way to recover the multiple value in the > same order; it might seem order is maintained at a first glance but > during index reorganization (merges, optimisations) this is not > guaranteed; I'd think carefully before relying on multi-valued field > encodings as you're entering an out-of-scope usage, but if all you > need is return multiple strings that should be doable. > > Sanne > > On 3 January 2013 14:11, Marko Lukša <marko.luksa(a)gmail.com&gt; wrote: >> Hi, >> >> we've found the following problem regarding projection queries when >> dealing with documents containing multiple fields with the same name. >> >> Let's say we add field "foo" with two different values to the same document: >> >> luceneOptions.addFieldToDocument("foo", "aaa", document); >> luceneOptions.addFieldToDocument("foo", "bbb", document); >> >> If we now do a projection query on field "foo", one would expect the >> resultset to contain exactly two results ({"aaa"} and {"bbb"}), but >> HSearch returns only a single result (the property value of the result >> is either "aaa" or "bbb", because Document.getFieldable("foo"), which is >> called in o.h.search.engine.impl.DocumentBuilderHelper, returns the >> first field that matches the given name). >> >> DocumentExtractor.extract() returns a single EntityInfo, but in order >> for it to properly handle projections as described in the previous >> paragraph, it should really be modified to return List<EntityInfo>. >> >> This sounds pretty reasonable when the query is projecting only a single >> field. When projecting multiple multi-valued fields, the resultset >> should actually return a cartesian product. >> >> This is one way of doing it. The other way of doing it is if we >> consider multiple fields with the same name as a single multi-valued >> field. When projecting such fields, the resultset would contain the same >> number of results as there are matching documents, with the projected >> value being a collection of all the values stored in the field. >> >> Actually, in CapeDwarf we need the cartesian product, as this is the way >> Google AppEngine does it. >> >> What do you guys think? >> >> Marko >> _______________________________________________ >> hibernate-dev mailing list >> hibernate-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/hibernate-dev > > _______________________________________________ > hibernate-dev mailing list > hibernate-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hibernate-dev _______________________________________________ hibernate-dev mailing list hibernate-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev

> I think anything handled by not-Lucene is wrong. I'm afraid Lucene won't do it, so we have no option. It's definitely not designed to do this: even a custom Collector can't return more results than Documents in its segment, as all representations work by using int as relative ids.

On Thu 2013-01-03 17:59, Ales Justin wrote: > >> I think anything handled by not-Lucene is wrong. > > > > I'm afraid Lucene won't do it, so we have no option. It's definitely > > not designed to do this: even a custom Collector can't return more > > results than Documents in its segment, as all representations work by > > using int as relative ids. > > Even if we have Document(s) for every possible combination - all elements of cartesian product? > Where current impl basically handles 1 x 1 x 1 x ... x 1. > And new HS impl just needs to take this multi Document approach into account. The problem with this approach is that it will makes things hard for most usecase except the one involving associated collections. We would need to filter out documents that refer to the same entity unless the query ought to be a cartesian one. this seems like a wrong thing to do as it would cost us more index size and speed as well as make the overall code more complex. I'd rather have a transformer cartesianize the end result. That would be much cheaper and code localized.

>>>> I think anything handled by not-Lucene is wrong. >>> >>> I'm afraid Lucene won't do it, so we have no option. It's definitely >>> not designed to do this: even a custom Collector can't return more >>> results than Documents in its segment, as all representations work by >>> using int as relative ids. >> >> Even if we have Document(s) for every possible combination - all elements of cartesian product? >> Where current impl basically handles 1 x 1 x 1 x ... x 1. >> And new HS impl just needs to take this multi Document approach into account. > > The problem with this approach is that it will makes things hard for > most usecase except the one involving associated collections. > We would need to filter out documents that refer to the same entity > unless the query ought to be a cartesian one. > this seems like a wrong thing to do as it would cost us more index size > and speed as well as make the overall code more complex. > I'd rather have a transformer cartesianize the end result. That would be > much cheaper and code localized. But this way you only know the real size once you expand the results into cartesian results. Which, as we discussed, completely beaks any counters and possible lazy handling. Not to mention you do this in-memory, which brings additional problems.