1:20 a.m.
Hi,
That's a catchy subject, now I'm sure I got Guillaume's (angry) attention :)
Sanne wanted us to think about feature removal... Here's a proposal.
The feature I'd like to deprecate/remove: automatic truncation of
date/calendars to a given resolution. It's used like that:
@Field
@CalendarBridge(resolution = Resolution.DAY)
private Calendar dateSent;
I've been working on date-related tests for the past few hours (well,
months, too), and I'm becoming more and more convinced that this resolution
thing is the wrong solution to an admittedly common issue. The other (in my
opinion right) solution being to use range queries.
When we truncate a date to a given resolution, we must use a particular
timezone, obviously. It makes no sense to truncate to the day when we don't
choose a particular timezone.
Then the truncated date will match only if we search for dates whose
truncation is identical in this timezone.
And guess what? In the current implementation, we use... GMT. Always. See
how org.apache.lucene.document.DateTools.round(long, Resolution) works,
getting the calendar used for truncation from TL_CAL, which uses the GMT
timezone.
What it means is that the truncation currently produces rather unexpected
results as soon as applications use a timezone different from GMT.
So it's likely to be considered as broken:
- for multi-timezone applications
- for single-timezone applications using a different timezone than GMT
For instance, let's index 2016-01-01T00:00:00+01:00 and query
2016-01-01T12:00:00+01:00 with a DAY resolution:
- We'll index 2015-12-31 (2016-01-01T00:00:00+01:00 is
2015-12-31T23:00:00+00:00, which once truncated is 2015-12-31+00:00)
- We'll query 2016-01-01 (2016-01-01T12:00:00+01:00 is
2016-01-01T11:00:00+00:00, which once truncated is 2016-01-01+00:00)
That won't match, but from the point of view of a single-timezone user it
should have.
Of course I'm exaggerating a bit, and the issue probably hasn't been
noticed by many, because:
- there is no problem with the MINUTES, SECONDS and MILISECONDS
resolutions, which are not affected by timezones
- there is no problem with the HOUR resolution in most timezones (there
is a problem, however, when the timezone is something like -11:30)
- the problem is less likely to be detected when the user timezone is
close to GMT, because then only date/times close to midnight (i.e. outside
business hours) will be affected.
But my point is: the current solution produces counter-intuitive, sometimes
even useless results.
It was useful when dates were indexed as strings, but now that we (mostly)
index them as integers, this does not seem relevant. Even the DateTools
javadoc hints that DateTools (including "round") are not relevant for
numeric dates:
Another approach is {@link NumericUtils}, which provides
a sortable binary representation (prefix encoded) of numeric values,
which
date/time are.
For indexing a {@link Date} or {@link Calendar}, just get the unix
timestamp as
<code>long</code> using {@link Date#getTime} or {@link
Calendar#getTimeInMillis} and
index this as a numeric value with {@link LongField}
and use {@link NumericRangeQuery} to query it.
What I'd like to propose is to deprecate or remove date/calendar
resolution, or limit it to STRING encoding, and instead encourage users to
query their dates with range queries whenever they don't want exact matches.
Of course we could also implement a way for users to customize the timezone
to be used when truncating. But range queries have other advantages:
- they fit the multi-timezone use case
- they allow to query the same field with multiple resolutions
At the very least, I'd like us to agree that in the future, we will only
implement automatic truncation with date/time types where the timezone is
either irrelevant or included (for instance ZonedDateTime or LocalDate or
LocalTime, but *not* Instant).
Relevant ticket: https://hibernate.atlassian.net/browse/HSEARCH-2378.
What do you think?
Yoann Rodière <yoann(a)hibernate.org>
Hibernate NoORM Team
8:53 p.m.
Hi,
On Thu, Jan 5, 2017 at 5:20 PM, Yoann Rodiere <yoann(a)hibernate.org> wrote:
That's a catchy subject, now I'm sure I got Guillaume's
(angry) attention
:)
Sanne wanted us to think about feature removal... Here's a proposal.
The feature I'd like to deprecate/remove: automatic truncation of
date/calendars to a given resolution. It's used like that:
Yeah, as you can imagine, I'm pretty reluctant to the idea of removing this
feature. We used it a lot at OW and I'm sure it's pretty widespread out
there (and even when we didn't use HS but Solr, we needed an equivalent
feature). Using range queries is a solution but it's a bit cumbersome for
such a common requirement.
I agree the current behavior might lead to unexpected results. To be
honest, I think we should simply ignore the timezone when truncating. I
mean, when I want to index at a day resolution, I want to index the user
facing day, not the one in the server time zone or in GMT.
So for instance, for 2016-01-01T00:00:00+01:00, I would index 2016-01-01,
not 2015-12-31.
So, as long as the resolution is less precise than day included, what I
would do is in fact remove our call to DateTools.round and have something
like:
- create a GMT GregorianCalendar
- don't set the calendar value using the time in millis but with the
original fields from the calendar passed in parameter (or 0/1 depending on
the resolution)
- index this value
I think this new behavior would fix most of the issues you mention. And if
you really need the behavior when dates in one time zone might be a
different date in a different time zone, then you definitely wouldn't use
the resolution option and stick to the full date with range queries.
So I agree the current behavior is probably not what user expects but I
think it has value.
I would pay to see your face while reading this email :).
--
Guillaume
10:28 p.m.
We used it a lot at OW and I'm sure it's pretty widespread
out there (and
even when we didn't use HS but Solr, we needed an equivalent
feature).
Using range queries is a solution but it's a bit cumbersome for such a
common requirement.
Please note that Hibernate Search queries are now built using a wrapper in
OWSI-Core
<https://github.com/openwide-java/owsi-core-parent/blob/bd6ab1ec1d54c22ed3...
in
most recent applications, and it could easily be adapted to add a Date
match method (using method overloading). In fact, if I remember correctly I
had to do exactly that for one application.
One thing to note about the OW use case, too: I've never seen anything
other than a Date being used in business models of single-timezone
applications at Open Wide. See below why it's relevant.
I think this new behavior would fix most of the issues you mention.
An easier implementation would be what you did with Elasticsearch using the
clone() method. But anyway, this would only fix the issue for Calendar. For
Date, we basically only have a GMT timestamp with no timezone data nor
field-based data, so there's really nothing we can do.
If we want to fix the default behavior for dates, the only solution I can
see is to use the default JVM timezone instead of GMT, maybe...
And if you really need the behavior when dates in one time zone might
be
a different date in a different time zone, then you definitely wouldn't use
the resolution option and stick to the full date with range queries.
All dates in one time zone may be a different date in a different time
zone... In fact, I'm pretty sure all dates in one time zone *are* a
different date in one different time zone. I see what you mean, but I just
wanted to point out how difficult it will be to explain properly in the
documentation. I don't know about you, but I really don't want to explain
in the docs that "this feature should only be used with java.util.Date if
your timezone is close to GMT", and I want even less to explain why...
Anyway, you do understand we're offering a feature that won't work properly
in most time zones, and will simply not work most of the time in time zones
such as, for instance, the one for the US west coast (UTC-8)? (in that
specific case it won't work after 4 PM).
Could we agree the current behavior should change, not only for
java.util.Calendar but also for java.util.Date?
If we do, we'll probably have to wait for 6.0, because users will have to
perform a full reindex after upgrading to a version of Hibernate Search
with the fix, even if they previously weren't affected by the bug.
Also:
At the very least, I'd like us to agree that in the future, we
will only
implement automatic truncation with date/time types where the timezone is
either irrelevant or included (for instance ZonedDateTime or LocalDate or
LocalTime, but *not* Instant).
Do you agree with the statement above?
I would pay to see your face while reading this email :).
Yeah, well, not much surprise here :)
Yoann Rodière <yoann(a)hibernate.org>
Hibernate NoORM Team
On 9 January 2017 at 12:53, Guillaume Smet <guillaume.smet(a)gmail.com> wrote:
> Hi,
>
> On Thu, Jan 5, 2017 at 5:20 PM, Yoann Rodiere <yoann(a)hibernate.org> wrote:
>
>> That's a catchy subject, now I'm sure I got Guillaume's (angry)
attention
>> :)
>>
>> Sanne wanted us to think about feature removal... Here's a proposal.
>>
>> The feature I'd like to deprecate/remove: automatic truncation of
>> date/calendars to a given resolution. It's used like that:
>>
>
> Yeah, as you can imagine, I'm pretty reluctant to the idea of removing
> this feature. We used it a lot at OW and I'm sure it's pretty widespread
> out there (and even when we didn't use HS but Solr, we needed an equivalent
> feature). Using range queries is a solution but it's a bit cumbersome for
> such a common requirement.
>
> I agree the current behavior might lead to unexpected results. To be
> honest, I think we should simply ignore the timezone when truncating. I
> mean, when I want to index at a day resolution, I want to index the user
> facing day, not the one in the server time zone or in GMT.
>
> So for instance, for 2016-01-01T00:00:00+01:00, I would index 2016-01-01,
> not 2015-12-31.
>
> So, as long as the resolution is less precise than day included, what I
> would do is in fact remove our call to DateTools.round and have something
> like:
> - create a GMT GregorianCalendar
> - don't set the calendar value using the time in millis but with the
> original fields from the calendar passed in parameter (or 0/1 depending on
> the resolution)
> - index this value
>
> I think this new behavior would fix most of the issues you mention. And if
> you really need the behavior when dates in one time zone might be a
> different date in a different time zone, then you definitely wouldn't use
> the resolution option and stick to the full date with range queries.
>
> So I agree the current behavior is probably not what user expects but I
> think it has value.
>
I would pay to see your face while reading this email :).
>
> --
> Guillaume
>
2675
days inactive
2679
days old
2 comments
2 participants
participants (2)
-
Guillaume Smet -
Yoann Rodiere