11:47 p.m.
Hi all,
the default value of norms() in the Field annotation is defined to YES.
I doubt it is always needed, e.g. if you have an Enum field, do you really
need norms for it? The same applies to long/int, boolean, date, OID fields
and very short fields like a code of the country.
I have already asked Sanne on the chat what was the motivation
to set it per default to YES. And he responded that the main reason is
backwards compatibility, i.e. it has always been enabled.
I'm now analyzing our program and I see absolutely no need for norms
(we are in process to switch from Hibernate Search 3.0.1 to the latest one).
But changing it for every field is a really huge task as we have a big application.
To improve the performance and to reduce memory consumption I would
like to change the default value of Field#norms() from YES to NO because
it has impact on the performance and memory consumption.
And users of Hibernate Search should decide per field if they really need
norms or not, e.g. if they want to do advanced queries, then they can activate
the norms for such fields.
I know this would break backwards compatibility. But I think in this case it
is a good thing.
Sanne also suggested an alternative solution. We may introduce a new global
option which would define the default value for Field#norms(). The default
value of this option would be obviously YES. Users like me could change it
to NO if needed. I personally think we have already enough options. But
this solution would be OK for me.
What do you think?
Best regards,
Andrej Golovnin
10:20 a.m.
On May 7, 2012, at 11:47 PM, Andrej Golovnin wrote:
the default value of norms() in the Field annotation is defined to
YES.
I doubt it is always needed, e.g. if you have an Enum field, do you really
need norms for it? The same applies to long/int, boolean, date, OID fields
and very short fields like a code of the country.
It depends. Maybe you want to boost any of these fields ad index time.
I have already asked Sanne on the chat what was the motivation
to set it per default to YES. And he responded that the main reason is
backwards compatibility, i.e. it has always been enabled.
I don't think backwards compatibility is the only reason. IMO it is a sensible
default. Changing it to Norms.NO per default means that @Boost won't work
out of the box anymore. Also looking at org.apache.lucene.document.Field the options
which omit the norm are marked "for expert use".
If you have no needs for index time boosting and need the additional memory you can
gain by not storing norms, you need to explicitly disable the storing of norms.
I'm now analyzing our program and I see absolutely no need for
norms
(we are in process to switch from Hibernate Search 3.0.1 to the latest one).
But changing it for every field is a really huge task as we have a big application.
If you already have the @Field annotations in the code, is it not just a simple
search&replace.
Either via IDE or simple see script?
Sanne also suggested an alternative solution. We may introduce a new
global
option which would define the default value for Field#norms(). The default
value of this option would be obviously YES. Users like me could change it
to NO if needed. I personally think we have already enough options. But
this solution would be OK for me.
-1 I agree that we already have enough options and the proposed changed will
just introduce ambiguities.
--Hardy
11:43 a.m.
Hi,
On Tue, May 8, 2012 at 10:20 AM, Hardy Ferentschik <hardy(a)hibernate.org> wrote:
> Sanne also suggested an alternative solution. We may introduce a
new global
> option which would define the default value for Field#norms(). The default
> value of this option would be obviously YES. Users like me could change it
> to NO if needed. I personally think we have already enough options. But
> this solution would be OK for me.
-1 I agree that we already have enough options and the proposed changed will
just introduce ambiguities.
I kinda agree with Hardy. It's difficult to find a good (and
understandable by a developer who isn't a Lucene expert) explanation
of what norms are, how they are useful and when you should enable
them.
Having the most user friendly configuration and allowing advanced
users to set it per field as it is now is IMHO the best option.
--
Guillaume
12:27 p.m.
On 8 May 2012 10:43, Guillaume Smet <guillaume.smet(a)gmail.com> wrote:
Hi,
On Tue, May 8, 2012 at 10:20 AM, Hardy Ferentschik <hardy(a)hibernate.org> wrote:
>> Sanne also suggested an alternative solution. We may introduce a new global
>> option which would define the default value for Field#norms(). The default
>> value of this option would be obviously YES. Users like me could change it
>> to NO if needed. I personally think we have already enough options. But
>> this solution would be OK for me.
>
> -1 I agree that we already have enough options and the proposed changed will
> just introduce ambiguities.
I kinda agree with Hardy. It's difficult to find a good (and
understandable by a developer who isn't a Lucene expert) explanation
of what norms are, how they are useful and when you should enable
them.
Having the most user friendly configuration and allowing advanced
users to set it per field as it is now is IMHO the best option.
I agree as well, when proposing it in the chat I meant it as "the only
option I see, but I don't like the added complexity". Hardy expressed
it better: ambiguities. Very evil stuff.
It's more like it's the only solution I see to change it without
breaking current behaviour for potential other users relying on it. It
is of course a great question, if it makes sense to rely on it for
some types?
Proposal for next versions: allow @Stereotype - like user annotations?
We would let the user define custom annotations to stack/combine
multiple framework annotations, possibly centralizing what the
"default" is for your app and reduce the amount of code people have to
pile on each getter.
I think it's reasonable to think that many user types are reused
multiple times in a real world application having many entities, and
likely want them to be indexed all in the same way.
As an added benefit this would resolve the concerns raised by some
users about having their domain model need to "depend" on our
annotations at compile time.
Sanne
--
Guillaume
_______________________________________________
hibernate-dev mailing list
hibernate-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hibernate-dev
2:08 p.m.
On Tue, May 8, 2012 at 12:27 PM, Sanne Grinovero <sanne(a)hibernate.org> wrote:
Proposal for next versions: allow @Stereotype - like user
annotations?
We would let the user define custom annotations to stack/combine
multiple framework annotations, possibly centralizing what the
"default" is for your app and reduce the amount of code people have to
pile on each getter.
I think it's reasonable to think that many user types are reused
multiple times in a real world application having many entities, and
likely want them to be indexed all in the same way.
It sure looks like a good idea.
We usually have only 3 differents types of field in our applications:
text, keyword and a specific fieldbridge to index only the id of an
element/collection of elements (it was before @IndexEmbedded was made
clever).
--
Guillaume
2:21 p.m.
Proposal for next versions: allow @Stereotype - like user
annotations?
We would let the user define custom annotations to stack/combine
multiple framework annotations, possibly centralizing what the
"default" is for your app and reduce the amount of code people have to
pile on each getter.
I think it's reasonable to think that many user types are reused
multiple times in a real world application having many entities, and
likely want them to be indexed all in the same way.
That's definitely a different story and worth exploring.
As an added benefit this would resolve the concerns raised by some
users about having their domain model need to "depend" on our
annotations at compile time.
If it were just or this benefit I would give the feature a -1 :-)
I never understood what this dependency problem is supposed to be.
--hardy
12:43 a.m.
Hi Sanne,
Proposal for next versions: allow @Stereotype - like user
annotations?
We would let the user define custom annotations to stack/combine
multiple framework annotations, possibly centralizing what the
"default" is for your app and reduce the amount of code people have to
pile on each getter.
I think it's reasonable to think that many user types are reused
multiple times in a real world application having many entities, and
likely want them to be indexed all in the same way.
+1. This is a cool idea. :-)
As an added benefit this would resolve the concerns raised by some
users about having their domain model need to "depend" on our
annotations at compile time.
We have no problem to depend on your annotations at compile time. :-)
Best regards,
Andrej Golovnin
12:43 a.m.
Hi Hardy,
It depends. Maybe you want to boost any of these fields ad index
time.
OK, what about the document id? It is indexed with NOT_ANALYZED_NO_NORMS.
As far as I understand (please correct me if I'm wrong) to boost fields at index time
norms must be enabled for those fields. And because Hibernate Search uses for document
ids
NOT_ANALYZED_NO_NORMS, adding @Boost to a document id would have no effect.
Although I'm not sure, someone really needs this. But it should be at least
documented.
Also it is also not clear from the Hibernate Search documentation, what will happen
if I have following annotation on a field:
@Field(boost=(a)Boost(1.2f), norms = Norms.NO)
private String description;
Would I see a warning or maybe an error message from Hibernate Search?
Does make such combination sense?
norms may not
be useful for short or non full text fields. I personally consider
Enums, boolean/int/short, date, OIDs as short and non full text fields.
And in my application those fields are majority. We have 130 indexes.
Some of this indexes are shared by multiple entities. So I would guess
that we have ca. 140-150 indexed entities + a lot of IndexEmbedded.
Of course I could do search&replace. But at some point a developer would
forget to add "norms = Norms.NO" to a field and our application would start
consuming more memory. And this is my dilemma. May be I can add an
automatic test to our build process which would force developers to add
"norms = Norms.NO". Alternative solution for would be to have a custom
Hibernate Search version, but I don't like this idea. I will discuss with my team
what is the best way to solve this small problem.
Btw Is it possible that OIDs added to index by IndexedEmbedded have
norms? Luke shows me it in two different indexes. Could someone
else please verify it? If it is the case, it would be nice if Hibernate Search
would have the same behavior for OIDs added to index either
by DocumentId or by IndexedEmbedded, e.g. index them with
NOT_ANALYZED_NO_NORMS.
I understand that for you as a framework provider it is difficult to find a golden path
which would satisfy every framework user. I think Sanne's suggestion
with "@Stereotype - like user annotations" may help us to achieve this goal.
Btw Hibernate Search and Hibernate helped us to create a really cool
product, so kudos to all who have contributed to this great projects!
Best regards,
Andrej Golovnin
12:26 p.m.
On 9 mai 2012, at 00:43, Andrej Golovnin wrote:
Hi Hardy,
> It depends. Maybe you want to boost any of these fields ad index time.
OK, what about the document id? It is indexed with NOT_ANALYZED_NO_NORMS.
As far as I understand (please correct me if I'm wrong) to boost fields at index
time
norms must be enabled for those fields. And because Hibernate Search uses for document
ids
NOT_ANALYZED_NO_NORMS, adding @Boost to a document id would have no effect.
Although I'm not sure, someone really needs this. But it should be at least
documented.
@DocumentId is special as we need to make sure we can lookup by its exact value. So
Hibernate Search takes over the responsibility to set the right options.
Plus it's unlikely that you need to "full-text" search an id field. If you
need to, you can use an extra @Field on it.
Can you open a JIRA issue to clarify this in
http://docs.jboss.org/hibernate/search/4.1/reference/en-US/html_single/#b... ?
Even better, can you propose the change as a GitHub pull request? :)
Also it is also not clear from the Hibernate Search documentation, what will happen
if I have following annotation on a field:
@Field(boost=(a)Boost(1.2f), norms = Norms.NO)
private String description;
Would I see a warning or maybe an error message from Hibernate Search?
Does make such combination sense?
I don't think we raise an exception, we should do something indeed. Do you want to
open the JIRA and possibly provide a patch via a GitHub pull request?
> From http://www.lucidimagination.com/content/scaling-lucene-and-solr#d0e71
norms may not be useful for short or non full text fields. I personally consider
Enums, boolean/int/short, date, OIDs as short and non full text fields.
And in my application those fields are majority. We have 130 indexes.
Some of this indexes are shared by multiple entities. So I would guess
that we have ca. 140-150 indexed entities + a lot of IndexEmbedded.
Of course I could do search&replace. But at some point a developer would
forget to add "norms = Norms.NO" to a field and our application would start
consuming more memory. And this is my dilemma. May be I can add an
automatic test to our build process which would force developers to add
"norms = Norms.NO". Alternative solution for would be to have a custom
Hibernate Search version, but I don't like this idea. I will discuss with my team
what is the best way to solve this small problem.
I think stereotypes would be best. For numbers, I encourage you to use @NumericField. I
don't believe these are normed and they are for sure more optimized.
Btw Is it possible that OIDs added to index by IndexedEmbedded have
norms? Luke shows me it in two different indexes. Could someone
else please verify it? If it is the case, it would be nice if Hibernate Search
would have the same behavior for OIDs added to index either
by DocumentId or by IndexedEmbedded, e.g. index them with
NOT_ANALYZED_NO_NORMS.
I am not following you. What's an OID.
I understand that for you as a framework provider it is difficult to find a golden path
which would satisfy every framework user. I think Sanne's suggestion
with "@Stereotype - like user annotations" may help us to achieve this goal.
Btw Hibernate Search and Hibernate helped us to create a really cool
product, so kudos to all who have contributed to this great projects!
Best regards,
Andrej Golovnin
_______________________________________________
hibernate-dev mailing list
hibernate-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hibernate-dev
10:04 p.m.
Hi Emmanuel,
Can you open a JIRA issue to clarify this in
http://docs.jboss.org/hibernate/search/4.1/reference/en-US/html_single/#b... ?
Even better, can you propose the change as a GitHub pull request? :)
Sure. :-)
>
> @Field(boost=(a)Boost(1.2f), norms = Norms.NO)
> private String description;
>
> Would I see a warning or maybe an error message from Hibernate Search?
> Does make such combination sense?
I don't think we raise an exception, we should do something indeed. Do you want to
open the JIRA and possibly provide a patch via a GitHub pull request?
OK, I will do that. But creating patch will take some time.
> Btw Is it possible that OIDs added to index by IndexedEmbedded
have
> norms? Luke shows me it in two different indexes. Could someone
> else please verify it? If it is the case, it would be nice if Hibernate Search
> would have the same behavior for OIDs added to index either
> by DocumentId or by IndexedEmbedded, e.g. index them with
> NOT_ANALYZED_NO_NORMS.
I am not following you. What's an OID.
Suppose you have following classes:
@Entity
public PostalAddress {
@Id
private String OID;
.....
}
@Entity
@Indexed
public Partner {
@Id
private String OID;
@OneToOne
@IndexEmbedded(depth = 1)
private PostalAddress defaultAddress;
....
}
The Lucene document for Partner would contain the field
OID from Partner indexed with NOT_ANALYZED_NO_NORMS
and it will contain also the field "defaultAddress.OID". But the
field "defaultAddress.OID" would have norms. This is what I see
at least with Luke in my indexes. I think the field "defaultAddress.OID"
should be also indexed with NOT_ANALYZED_NO_NORMS.
Best regards
Andrej Golovnin
11:54 p.m.
On 9 May 2012 21:04, Andrej Golovnin <golovnin(a)gmx.net> wrote:
Hi Emmanuel,
> Can you open a JIRA issue to clarify this in
http://docs.jboss.org/hibernate/search/4.1/reference/en-US/html_single/#b... ?
> Even better, can you propose the change as a GitHub pull request? :)
Sure. :-)
>>
>> @Field(boost=(a)Boost(1.2f), norms = Norms.NO)
>> private String description;
>>
>> Would I see a warning or maybe an error message from Hibernate Search?
>> Does make such combination sense?
>
> I don't think we raise an exception, we should do something indeed. Do you want
to open the JIRA and possibly provide a patch via a GitHub pull request?
OK, I will do that. But creating patch will take some time.
>> Btw Is it possible that OIDs added to index by IndexedEmbedded have
>> norms? Luke shows me it in two different indexes. Could someone
>> else please verify it? If it is the case, it would be nice if Hibernate Search
>> would have the same behavior for OIDs added to index either
>> by DocumentId or by IndexedEmbedded, e.g. index them with
>> NOT_ANALYZED_NO_NORMS.
>
> I am not following you. What's an OID.
Suppose you have following classes:
@Entity
public PostalAddress {
@Id
private String OID;
.....
}
@Entity
@Indexed
public Partner {
@Id
private String OID;
@OneToOne
@IndexEmbedded(depth = 1)
private PostalAddress defaultAddress;
....
}
The Lucene document for Partner would contain the field
OID from Partner indexed with NOT_ANALYZED_NO_NORMS
and it will contain also the field "defaultAddress.OID". But the
field "defaultAddress.OID" would have norms. This is what I see
at least with Luke in my indexes. I think the field "defaultAddress.OID"
should be also indexed with NOT_ANALYZED_NO_NORMS.
You're right. In fact there was a different proposal about not
indexing "defaultAddress.OID" at all or optionally, but while that
caused some discussions and action is still unclear, let's at least
omit norms as they are definitely not needed nor expected for such a
field.. another JIRA please?
Regards,
Sanne
Best regards
Andrej Golovnin
_______________________________________________
hibernate-dev mailing list
hibernate-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hibernate-dev
4621
days inactive
4623
days old
10 comments
5 participants
participants (5)
-
Andrej Golovnin -
Emmanuel Bernard -
Guillaume Smet -
Hardy Ferentschik -
Sanne Grinovero