1:26 p.m.
You might remember that running a full-text Query on a field always
required some specific care;
since the beginning of Hibernate Search the user had to make sure the
field was not tokenized, or tokenized but generating a single token.
This was a "soft requirement": if you didn't know, you'd get
inconsistent results but no error would be shown - after all, a Lucene
was typically schema-less.
With Lucene 5, if you didn't map your field specifically for the
sorting purpose, you'll get a runtime exception at query time. By
"specifically for sorting" the requirement is that the *single token*
needs to be stored as a DocValue.
DocValues are useful for other purposes too; for example they are a
more efficient strategy to store our "id" field and I hope we'll soon
use that transparently. It's also a better replacement for all those
use cases which previously would rely on a FieldCache (which by the
way was used by the sorting code under the hood). For example, we
recently migrated the Spatial indexing to use DocValues, and we now
support serializing and writing of DocValues.
What we don't have is a way for end user to explicitly single out
which fields they want to be "DocValue encoded", and this really needs
to be added now as the workaround we have to be able to run Sort
operations without it is killing our performance.
How should such annotations look like?
I don't like to expose such low-level details when for most people
it's just about sorting. Still, being useful for other reasons having
a "@Sortable" (or similar) named annotation would be limiting.
DocValues themselves - as a concept - are fine but even in the Lucene
history the exact name changed several times; wondering if we should
stick to the (current) technical term or abstract a bit from it.
I'm not sure if this should be extending the @Field annotation as
there are special restrictions implied in terms of analysis: are we
going to enforce a specific type of tokenizer, or simply take the
analysis option away?
Any nice suggestion of how this could looke like? This would become a
highly used public API.
The good news is that we'll be able to validate sort definitions.
Thanks,
Sanne
4:11 p.m.
Hi Sanne,
On Wed, Jul 29, 2015 at 1:26 PM, Sanne Grinovero <sanne(a)hibernate.org>
wrote:
I'm not sure if this should be extending the @Field annotation
as
there are special restrictions implied in terms of analysis: are we
going to enforce a specific type of tokenizer, or simply take the
analysis option away?
You can't remove the analysis option away: it's often used to normalize
sorting on strings (lowercase, remove accents, remove special characters
and so on).
FWIW, we use specific fields for sorting each time we need to sort on a
string as we don't want to tokenize the string (but not for numerics and
dates). Maybe @SortFields/@SortField annotations would be in order (I don't
like Sortable as I don't think it's a good idea to use these fields for
search).
--
Guillaume
6 p.m.
Hi Guillaume,
thanks! great input. Some comments inline:
On 4 August 2015 at 15:11, Guillaume Smet <guillaume.smet(a)gmail.com> wrote:
Hi Sanne,
On Wed, Jul 29, 2015 at 1:26 PM, Sanne Grinovero <sanne(a)hibernate.org>
wrote:
>
> I'm not sure if this should be extending the @Field annotation as
> there are special restrictions implied in terms of analysis: are we
> going to enforce a specific type of tokenizer, or simply take the
> analysis option away?
You can't remove the analysis option away: it's often used to normalize
sorting on strings (lowercase, remove accents, remove special characters and
so on).
Right we made this same example in a recent meeting we had on this same subject.
So that's what makes it tricky: we want to allow Analysis, but while
Lucene needs a strong guarantee that it will be unique, we can't
really verify for that unless we take away the liberty to use any
analyzer.
An alternative would be to wrap the Analyzer to monitor and verify it
to be "well-behaved" but I'm not sure if that's doable, or if the
performance would be negligible. I guess we'll just put it into user's
hands to make a sensible choice.. not that we've done better so far on
this aspect.
FWIW, we use specific fields for sorting each time we need to sort on
a
string as we don't want to tokenize the string (but not for numerics and
dates). Maybe @SortFields/@SortField annotations would be in order (I don't
like Sortable as I don't think it's a good idea to use these fields for
search).
I like that name proposal, and +1 to not encourage people to try reuse
the same field for sorting and indexing.
The next action for us is to verify what the performance impact is of
the current approach, which is based on the UninvertingReader from
lucene-misc. Gunnar pointed out that uninverting and loading into a
FieldCache is not very different than what Lucene has been doing so
far, so that might be a good strategy to allow migrating to Lucene 5
incrementally, and provide an incremental improvement in this area
rather than requiring the new mapping.
I'll soon merge this approach, and as usual I'm lacking on real-world
applications to benchmark so if you're interested in helping on that
that would be awesome; we just need to know that the new code won't be
significantly slower than the Lucene 4 based strategies for sorted
queries.
Thanks,
Sanne
11:41 a.m.
Hi,
I think a great solution for that would be to leverage "annotation
composition" as done e.g. in CDI and Bean Validation.
There would be an annotation @DocValuesField which will cause the creation
of a DocValues and which would expose attributes required for its
configuration:
@DocValuesField(name="foo", ... )
private String bar;
Besides using @DocValuesField itself directly, it would be usable as a
meta-annotation on "doc value annotation types":
@DocValuesField
public @interface SortField {
@OverridesAttribute(name="name")
String name();
}
And then its usage:
@SortField
private String bar;
Of course such doc value annotation types (I think @SortField, @Facet and
@DocumentId could be modelled as that) would only expose those degrees of
freedom needed for a specific use case.
Most users would only use these more abstract, easy-to-use specific purpose
annotations. But others could use @DocValuesField directly to create custom
doc values or they could even create their own, domain-specific doc value
annotation type.
Cheers,
--Gunnar
2015-08-04 18:00 GMT+02:00 Sanne Grinovero <sanne(a)hibernate.org>:
Hi Guillaume,
thanks! great input. Some comments inline:
On 4 August 2015 at 15:11, Guillaume Smet <guillaume.smet(a)gmail.com>
wrote:
> Hi Sanne,
>
> On Wed, Jul 29, 2015 at 1:26 PM, Sanne Grinovero <sanne(a)hibernate.org>
> wrote:
>>
>> I'm not sure if this should be extending the @Field annotation as
>> there are special restrictions implied in terms of analysis: are we
>> going to enforce a specific type of tokenizer, or simply take the
>> analysis option away?
>
>
> You can't remove the analysis option away: it's often used to normalize
> sorting on strings (lowercase, remove accents, remove special characters
and
> so on).
Right we made this same example in a recent meeting we had on this same
subject.
So that's what makes it tricky: we want to allow Analysis, but while
Lucene needs a strong guarantee that it will be unique, we can't
really verify for that unless we take away the liberty to use any
analyzer.
An alternative would be to wrap the Analyzer to monitor and verify it
to be "well-behaved" but I'm not sure if that's doable, or if the
performance would be negligible. I guess we'll just put it into user's
hands to make a sensible choice.. not that we've done better so far on
this aspect.
> FWIW, we use specific fields for sorting each time we need to sort on a
> string as we don't want to tokenize the string (but not for numerics and
> dates). Maybe @SortFields/@SortField annotations would be in order (I
don't
> like Sortable as I don't think it's a good idea to use these fields for
> search).
I like that name proposal, and +1 to not encourage people to try reuse
the same field for sorting and indexing.
The next action for us is to verify what the performance impact is of
the current approach, which is based on the UninvertingReader from
lucene-misc. Gunnar pointed out that uninverting and loading into a
FieldCache is not very different than what Lucene has been doing so
far, so that might be a good strategy to allow migrating to Lucene 5
incrementally, and provide an incremental improvement in this area
rather than requiring the new mapping.
I'll soon merge this approach, and as usual I'm lacking on real-world
applications to benchmark so if you're interested in helping on that
that would be awesome; we just need to know that the new code won't be
significantly slower than the Lucene 4 based strategies for sorted
queries.
Thanks,
Sanne
_______________________________________________
hibernate-dev mailing list
hibernate-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hibernate-dev
2:14 p.m.
A quick update on some more exploration on this:
it turns out sorting on a NumericField when this field is also using
an "indexNullAs" token gets the UninvertingReader approach to throw an
exception.
My two conclusions:
- we need to move away from supporting those tokens in NumericField,
especially as stricter schema is coming in next gen Lucene
- yet another reason to clearly separate fields meant for searching vs sorting
-- Sanne
On 5 August 2015 at 10:41, Gunnar Morling <gunnar(a)hibernate.org> wrote:
Hi,
I think a great solution for that would be to leverage "annotation
composition" as done e.g. in CDI and Bean Validation.
There would be an annotation @DocValuesField which will cause the creation
of a DocValues and which would expose attributes required for its
configuration:
@DocValuesField(name="foo", ... )
private String bar;
Besides using @DocValuesField itself directly, it would be usable as a
meta-annotation on "doc value annotation types":
@DocValuesField
public @interface SortField {
@OverridesAttribute(name="name")
String name();
}
And then its usage:
@SortField
private String bar;
Of course such doc value annotation types (I think @SortField, @Facet and
@DocumentId could be modelled as that) would only expose those degrees of
freedom needed for a specific use case.
Most users would only use these more abstract, easy-to-use specific purpose
annotations. But others could use @DocValuesField directly to create custom
doc values or they could even create their own, domain-specific doc value
annotation type.
Cheers,
--Gunnar
2015-08-04 18:00 GMT+02:00 Sanne Grinovero <sanne(a)hibernate.org>:
>
> Hi Guillaume,
> thanks! great input. Some comments inline:
>
> On 4 August 2015 at 15:11, Guillaume Smet <guillaume.smet(a)gmail.com>
> wrote:
> > Hi Sanne,
> >
> > On Wed, Jul 29, 2015 at 1:26 PM, Sanne Grinovero <sanne(a)hibernate.org>
> > wrote:
> >>
> >> I'm not sure if this should be extending the @Field annotation as
> >> there are special restrictions implied in terms of analysis: are we
> >> going to enforce a specific type of tokenizer, or simply take the
> >> analysis option away?
> >
> >
> > You can't remove the analysis option away: it's often used to normalize
> > sorting on strings (lowercase, remove accents, remove special characters
> > and
> > so on).
>
> Right we made this same example in a recent meeting we had on this same
> subject.
> So that's what makes it tricky: we want to allow Analysis, but while
> Lucene needs a strong guarantee that it will be unique, we can't
> really verify for that unless we take away the liberty to use any
> analyzer.
> An alternative would be to wrap the Analyzer to monitor and verify it
> to be "well-behaved" but I'm not sure if that's doable, or if the
> performance would be negligible. I guess we'll just put it into user's
> hands to make a sensible choice.. not that we've done better so far on
> this aspect.
>
> > FWIW, we use specific fields for sorting each time we need to sort on a
> > string as we don't want to tokenize the string (but not for numerics and
> > dates). Maybe @SortFields/@SortField annotations would be in order (I
> > don't
> > like Sortable as I don't think it's a good idea to use these fields for
> > search).
>
> I like that name proposal, and +1 to not encourage people to try reuse
> the same field for sorting and indexing.
>
> The next action for us is to verify what the performance impact is of
> the current approach, which is based on the UninvertingReader from
> lucene-misc. Gunnar pointed out that uninverting and loading into a
> FieldCache is not very different than what Lucene has been doing so
> far, so that might be a good strategy to allow migrating to Lucene 5
> incrementally, and provide an incremental improvement in this area
> rather than requiring the new mapping.
>
> I'll soon merge this approach, and as usual I'm lacking on real-world
> applications to benchmark so if you're interested in helping on that
> that would be awesome; we just need to know that the new code won't be
> significantly slower than the Lucene 4 based strategies for sorted
> queries.
>
> Thanks,
> Sanne
> _______________________________________________
> hibernate-dev mailing list
> hibernate-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hibernate-dev
2:31 p.m.
On Fri, Aug 7, 2015 at 1:14 PM, Sanne Grinovero <sanne(a)hibernate.org> wrote:
A quick update on some more exploration on this:
it turns out sorting on a NumericField when this field is also using
an "indexNullAs" token gets the UninvertingReader approach to throw an
exception.
My two conclusions:
- we need to move away from supporting those tokens in NumericField,
especially as stricter schema is coming in next gen Lucene
- yet another reason to clearly separate fields meant for searching vs
sorting
One possible workaround is to enforce the indexNullAs value to match the
underlying field type, at the
moment it is always a string.
Another possibility is to model "null" as field not present in the lucene
document, instead of a marker token.
Gustavo
12:18 a.m.
On 7 August 2015 at 13:31, Gustavo Fernandes <gustavo(a)infinispan.org> wrote:
On Fri, Aug 7, 2015 at 1:14 PM, Sanne Grinovero
<sanne(a)hibernate.org> wrote:
>
> A quick update on some more exploration on this:
> it turns out sorting on a NumericField when this field is also using
> an "indexNullAs" token gets the UninvertingReader approach to throw an
> exception.
> My two conclusions:
> - we need to move away from supporting those tokens in NumericField,
> especially as stricter schema is coming in next gen Lucene
> - yet another reason to clearly separate fields meant for searching vs
> sorting
One possible workaround is to enforce the indexNullAs value to match the
underlying field type, at the
moment it is always a string.
Interesting idea, but the user would need to provide which "value"
he's ok to give up, as he would need to pick a number to be treated as
NaN.
Since the indexNullAs parameter requires a string, would we expect
people to write a number in there?
Another possibility is to model "null" as field not present
in the lucene
document, instead of a marker token.
Thanks! I've run some tests with this and have a patch working based
on this idea; I was sceptical initially because of scoring, but it
seems it's doable without needing to score all documents on this
negation. I'll send a PR soon; basically with this approach the
indexNullAs attribute will be ignored, but it's ok I think as this
solution is better and there's no drawback (nor needing any user
configuration).
Sanne
10:14 a.m.
On 10 Aug 2015, at 23:18, Sanne Grinovero <sanne(a)hibernate.org>
wrote:
On 7 August 2015 at 13:31, Gustavo Fernandes <gustavo(a)infinispan.org
<mailto:gustavo@infinispan.org>> wrote:
> On Fri, Aug 7, 2015 at 1:14 PM, Sanne Grinovero <sanne(a)hibernate.org> wrote:
>>
>> A quick update on some more exploration on this:
>> it turns out sorting on a NumericField when this field is also using
>> an "indexNullAs" token gets the UninvertingReader approach to throw an
>> exception.
>> My two conclusions:
>> - we need to move away from supporting those tokens in NumericField,
>> especially as stricter schema is coming in next gen Lucene
>> - yet another reason to clearly separate fields meant for searching vs
>> sorting
>
>
> One possible workaround is to enforce the indexNullAs value to match the
> underlying field type, at the
> moment it is always a string.
Interesting idea, but the user would need to provide which "value"
he's ok to give up, as he would need to pick a number to be treated as
NaN.
Since the indexNullAs parameter requires a string, would we expect
people to write a number in there?
I’m not sure if uninverting a field containing a mix of string/numeric values will be
supported again in Lucene.
Maybe it will [1], maybe not, in the meantime, the indexNullAs would need at least to be
parseable to a number by the
underlying field bridge. Many use cases allows for NaN to be a number itself without
problems.
[1] https://issues.apache.org/jira/browse/SOLR-7521
<https://issues.apache.org/jira/browse/SOLR-7521>
> Another possibility is to model "null" as field not present in the lucene
> document, instead of a marker token.
Thanks! I've run some tests with this and have a patch working based
on this idea; I was sceptical initially because of scoring, but it
seems it's doable without needing to score all documents on this
negation. I'll send a PR soon; basically with this approach the
indexNullAs attribute will be ignored, but it's ok I think as this
solution is better and there's no drawback (nor needing any user
configuration).
+1
Gustavo
8:59 a.m.
2015-08-11 0:18 GMT+02:00 Sanne Grinovero <sanne(a)hibernate.org>:
> One possible workaround is to enforce the indexNullAs value to
match the
> underlying field type, at the
> moment it is always a string.
Interesting idea, but the user would need to provide which "value"
he's ok to give up, as he would need to pick a number to be treated as
NaN.
Since the indexNullAs parameter requires a string, would we expect
people to write a number in there?
+1 for that idea.
It'd work very well in the ES case where null tokens are also expected
to be of the correct type, e.g. the number -1 for a numeric field.
Also leaving ES aside, I find it troubling that setting indexNullAs()
transparently alters the type of a field of a numeric property from
IntField etc. to a string field.
> Another possibility is to model "null" as field not present in the lucene
> document, instead of a marker token.
Thanks! I've run some tests with this and have a patch working based
on this idea; I was sceptical initially because of scoring, but it
seems it's doable without needing to score all documents on this
negation. I'll send a PR soon; basically with this approach the
indexNullAs attribute will be ignored, but it's ok I think as this
solution is better and there's no drawback (nor needing any user
configuration).
Isn't that what's happening already atm.? If a value is null (and
indexNullAs() is *not* given), it will not be part of the index as per
my observations. But if it is null and indexNullAs() is given, that
token will be used. That seems like the right thing to do IMO. Esp. I
would not ignore indexNullAs() if given, apparently the user meant to
encode null in that case e.g. to use it in queries.
Sanne
_______________________________________________
hibernate-dev mailing list
hibernate-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hibernate-dev
2:33 p.m.
On Tue, Aug 18, 2015 at 08:59:28AM +0200, Gunnar Morling wrote:
>> One possible workaround is to enforce the indexNullAs value
to match the
>> underlying field type, at the
>> moment it is always a string.
>
> Interesting idea, but the user would need to provide which "value"
> he's ok to give up, as he would need to pick a number to be treated as
> NaN.
> Since the indexNullAs parameter requires a string, would we expect
> people to write a number in there?
+1 for that idea.
+1 as well
Isn't that what's happening already atm.? If a value is null
(and
indexNullAs() is *not* given), it will not be part of the index as per
my observations. But if it is null and indexNullAs() is given, that
token will be used. That seems like the right thing to do IMO. Esp. I
would not ignore indexNullAs() if given, apparently the user meant to
encode null in that case e.g. to use it in queries.
Right, that the intended behavior. If it behaves differently right now
a regression has sneaked in.
--Hardy
3167
days inactive
3187
days old
9 comments
5 participants
participants (5)
-
Guillaume Smet -
Gunnar Morling -
Gustavo Fernandes -
Hardy Ferentschik -
Sanne Grinovero