1:47 a.m.
some inline comments:
2008/4/22, Hardy Ferentschik <hibernate(a)ferentschik.de>:
On Tue, 22 Apr 2008 03:29:18 +0200, Sanne Grinovero
<sanne.grinovero(a)gmail.com> wrote:
> A new proposal:
> I got inspired by the "3VL" considerations described in Emmanuel's
> link to wikipedia, and think backwards compatibility is nice:
> add a "@IndexNullMarker" on the property, this will add an additional
> Field to the index for null values:
>
Hmm, interesting idea. It addresses one of the biggest concerns I have
with this null marker thing, namely ambiguities. But would querying look
like in this
case. Wouldn't it become harder? Whenever you want to use this feature you
would have
to combine two fields - foo and fooIsFalse - within a boolean query to get
the expected
result. Something like this: "foo:bar OR fooIsNull:true".
Yes right but I don't see how this is worse than searching for
"foo:bar OR foo:NULL-KEYWORD", just some less ambiguity.
If you just want to search for null fields, fooIsNull:true
Of course it would also mean that the index size grows since we are
adding
more fields.
And the bigger the index, ...
Interesting I wasn't expecting the index to grow as I remove a Field
and replace it with another; I've made a test for this: on 10,000,000
docs having 50% a random text value (chosen from 800 constants to
limit total string tokens) and 50% nulls the index
size grows by 3.5% compared to no null values (same docs and 800 consts).
I wasn't expecting any growth above some bytes, anyway I think 3.5% is
quite good.
> The Field and StringBridge API would remain as-is;
> If you prefer not to add an additional @IndexNullMarker could be
> dropped if you think adding this field is acceptable for all fields.
>
I think it should stay an optional and explicit feature. Adding one
addtional field for
each indexed properties does not seem justified. Especially, since we agree
that
the best solution would be to re-think your design and come up with a
proper
non-null default. So by offering this feature we might end up encouraging
people
to stick with there less optimal design ;-)
Absolutely agree, default is disabled. But I hope it will be possible
to log an explicit error if someone tries to use the field (without
enabling this feature) with the "NullQuery" Emmanuel proposed.
Cheers,
Hardy
Regards,
Sanne
4:19 p.m.
New subject: [Hibernate-JIRA] Commented: (HSEARCH-115) Add a default value for indexing null value
On Tue, 22 Apr 2008 18:47:14 +0200, Sanne Grinovero
<sanne.grinovero(a)gmail.com> wrote:
Yes right but I don't see how this is worse than searching for
"foo:bar OR foo:NULL-KEYWORD", just some less ambiguity.
If you just want to search for null fields, fooIsNull:true
It is not much worse, but as less intuative. One has to know that if
@IndexNullMarker
is added another field fooIsNull is added. It's less intuative. On the
other hand would
actually work without ambiguities. I guess one would have to RTFM.
Interesting I wasn't expecting the index to grow as I remove a
Field
and replace it with another; I've made a test for this: on 10,000,000
docs having 50% a random text value (chosen from 800 constants to
limit total string tokens) and 50% nulls the index
size grows by 3.5% compared to no null values (same docs and 800
consts).
I wasn't expecting any growth above some bytes, anyway I think 3.5% is
quite good.
If you just add one of the fields (foo or fooIsNull) at the time we
are
fine. It could
be more of an issue if we have always a fooIsNull:false for consistency as
you mentioned as well.
Nevertheless, I think your idea is still better than the straight forward
approach. It just comes with
more complexity usage wise.
--Hardy
6:33 p.m.
New subject: [Hibernate-JIRA] Commented: (HSEARCH-115) Add a default value for indexing null value
Hi again :)
One more thing comes to my mind:
On Tue, 22 Apr 2008 18:47:14 +0200, Sanne Grinovero
<sanne.grinovero(a)gmail.com> wrote:
> > The Field and StringBridge API would remain as-is;
I am not so sure about that. Looking at the DocumentBuilder code it would
make sense to
let the FieldBridge handle the null marker. DocumentBuilder just iterates
over the members
of the entity and for each with @Field annotated member calls the
reponsible FieldBridge.
It also passes along additonal annotation values like the boost. It would
make sense to handle the
@IndexNullMarker the same way, right?
--Hardy
8:14 p.m.
New subject: [Hibernate-JIRA] Commented: (HSEARCH-115) Add a default value for indexing null value
Hi,
hoping the rest of list users won't hate me:
IMHO I think it would be preferable to handle this in the DocumentBuilder,
so that people won't have to repeat this complexity in custom
FieldBridge implementations,
and you don't have to update all built-in FieldBridges.
We could make a wrapping FieldBridge that adds this functionality, but
the special null marker Field is actually a constant and could be
added to the metaproperties at DocumentBuilder initialization; also
when building the Document you could test first for null
and avoid passing all other options to the FieldBridge, just adding
the constant Field to the document and skipping all further processing
for the current property.
We could skip this same processing also for properties which don't use
the new feature, effectively speeding it up a bit, so we don't feel
guilty for complicating the DocumentBuilder but are actually doing
some optimization and code cleanup.
Don't know if that is a dangerous optimization for backwards
compatibility... I expect no FieldBridge to generate Fields on a null
value but someone could rely on it..?
Also I'm not sure about adding @IndexNullMarker or adding an option to @Field,
what do others think about this?
Emmanuel said:
If we go that path, we should add a NullQuery class that can be
combined with other >*Query from Lucene and hide the complexity.
This looks brilliant, but how should we instantiate a new NullQuery?
If an @IndexNullMarker option could override the used fieldname we
need a factory class with a reference to the DocumentBuilder or some
other way to discover the special field name,
so the simpleas way is to not have an option to choose the
keyword-fieldname and define it only from the property name. Would it
be acceptable that the user can't override the fieldname?
regards,
Sanne
2008/4/23, Hardy Ferentschik <hibernate(a)ferentschik.de>:
Hi again :)
One more thing comes to my mind:
On Tue, 22 Apr 2008 18:47:14 +0200, Sanne Grinovero
<sanne.grinovero(a)gmail.com> wrote:
> > > The Field and StringBridge API would remain as-is;
>
I am not so sure about that. Looking at the DocumentBuilder code it would
make sense to
let the FieldBridge handle the null marker. DocumentBuilder just iterates
over the members
of the entity and for each with @Field annotated member calls the
reponsible FieldBridge.
It also passes along additonal annotation values like the boost. It would
make sense to handle the
@IndexNullMarker the same way, right?
--Hardy
10:39 p.m.
New subject: [Hibernate-JIRA] Commented: (HSEARCH-115) Add a default value for indexing null value
Wau - this feature greated more interest as expected ;-)
On Wed, 23 Apr 2008 13:14:26 +0200, Sanne Grinovero
<sanne.grinovero(a)gmail.com> wrote:
IMHO I think it would be preferable to handle this in the
DocumentBuilder,
so that people won't have to repeat this complexity in custom
FieldBridge implementations,
and you don't have to update all built-in FieldBridges.
We could make a wrapping FieldBridge that adds this functionality, but
the special null marker Field is actually a constant and could be
added to the metaproperties at DocumentBuilder initialization; also
when building the Document you could test first for null
and avoid passing all other options to the FieldBridge, just adding
the constant Field to the document and skipping all further processing
for the current property.
Can definitely be done this way. The question is really
if it is a good
idea
design wise. So far the DocumentBuilder did not 'inspect' the actual
values.
Now we would have to test at the very least against null.
We could skip this same processing also for properties which
don't use
the new feature, effectively speeding it up a bit, so we don't feel
guilty for complicating the DocumentBuilder but are actually doing
some optimization and code cleanup.
Don't know if that is a dangerous optimization for backwards
compatibility... I expect no FieldBridge to generate Fields on a null
value but someone could rely on it..?
You never know. Some people might have
already implented null indexing via
their custom bridge or rely on for other reason on the fact that null
values
are passed to the FieldBridge. However, I agree that the DocumentBuilder
could
be cleaned up a little ;-)
--Hardy
5858
days inactive
5859
days old
4 comments
2 participants
participants (2)
-
Hardy Ferentschik -
Sanne Grinovero