3:26 p.m.
Hello,
while debugging I just noticed that having a two entities scenario like
@Indexed class Book {
Long id;
@IndexedEmbedded Nation bublishedFirstIn;
}
Nation { //NOT indexed
Long id;
@Field name
}
the index for Book will contain these fields:
_hibernate_class ------------- NOT_ANALYZED_NO_NORMS, STORED
id ----------------------------------- NOT_ANALYZED_NO_NORMS, STORED
bublishedFirstIn.id ------------ NOT_ANALYZED, STORED
bublishedFirstIn.name ------ ok (don't care)
So, doubts:
1) I guess at minimum "bublishedFirstIn.id" should use
NOT_ANALYZED_NO_NORMS as well, saving some precious memory and index
space (see HSEARCH-473)
2) Why is "bublishedFirstIn.id" at all indexed? I can't think of a use
case, likely missing something.
3) (Extreme) why is "id" indexed ? Are we ever supposed to run a full
text query on the id fields? I think we should have an option about
this, and in future default to Store only.
Cheers,
Sanne
4:56 a.m.
Are you talking about JPA entities (with @Entity and @Id or mere mortal POJOs?
On 14 mars 2011, at 21:26, Sanne Grinovero wrote:
Hello,
while debugging I just noticed that having a two entities scenario like
@Indexed class Book {
Long id;
@IndexedEmbedded Nation bublishedFirstIn;
}
Nation { //NOT indexed
Long id;
@Field name
}
the index for Book will contain these fields:
_hibernate_class ------------- NOT_ANALYZED_NO_NORMS, STORED
id ----------------------------------- NOT_ANALYZED_NO_NORMS, STORED
bublishedFirstIn.id ------------ NOT_ANALYZED, STORED
bublishedFirstIn.name ------ ok (don't care)
So, doubts:
1) I guess at minimum "bublishedFirstIn.id" should use
NOT_ANALYZED_NO_NORMS as well, saving some precious memory and index
space (see HSEARCH-473)
2) Why is "bublishedFirstIn.id" at all indexed? I can't think of a use
case, likely missing something.
3) (Extreme) why is "id" indexed ? Are we ever supposed to run a full
text query on the id fields? I think we should have an option about
this, and in future default to Store only.
Cheers,
Sanne
_______________________________________________
hibernate-dev mailing list
hibernate-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hibernate-dev
10:43 a.m.
2011/3/15 Emmanuel Bernard <emmanuel(a)hibernate.org>:
Are you talking about JPA entities (with @Entity and @Id or mere
mortal POJOs?
I got a bit confused for the reasons of your question, so I made some tests.
The index state reported in the previous email was about annotation
mapped entities, I repeated
the same test on "Infinispan Query POJOs", so the root element is
marked only with
"@ProvidedId @Indexed"
And the contained-in objects have no class level annotations.
As the contained-in elements in the collection don't get assigned a
provided-id, the ID FieldBridge is not adding
it to the index as it skips empty values.
If I add a javax.persistence.Id annotated field (even while I'm not
making it an entity), the ID property gets indexed.
In the case in which nothing is added to the index as id for the
embedded value, I can still query fine so it looks like
this shouldn't be indexed.
Also, it would improve a bit on HSEARCH-183 and HSEARCH-659, at least
the "id" field wouldn't be overwritten when people use an empty prefix
for @IndexedEmbedded (which is quite dangerous, as it will result in
apparently randomly selected values to be returned).
I guess we should prevent the indexing of ids in secondary elements?
Sanne
On 14 mars 2011, at 21:26, Sanne Grinovero wrote:
> Hello,
> while debugging I just noticed that having a two entities scenario like
>
> @Indexed class Book {
> Long id;
> @IndexedEmbedded Nation bublishedFirstIn;
> }
> Nation { //NOT indexed
> Long id;
> @Field name
> }
>
> the index for Book will contain these fields:
>
> _hibernate_class ------------- NOT_ANALYZED_NO_NORMS, STORED
> id ----------------------------------- NOT_ANALYZED_NO_NORMS, STORED
> bublishedFirstIn.id ------------ NOT_ANALYZED, STORED
> bublishedFirstIn.name ------ ok (don't care)
>
> So, doubts:
> 1) I guess at minimum "bublishedFirstIn.id" should use
> NOT_ANALYZED_NO_NORMS as well, saving some precious memory and index
> space (see HSEARCH-473)
>
> 2) Why is "bublishedFirstIn.id" at all indexed? I can't think of a use
> case, likely missing something.
>
> 3) (Extreme) why is "id" indexed ? Are we ever supposed to run a full
> text query on the id fields? I think we should have an option about
> this, and in future default to Store only.
>
> Cheers,
> Sanne
> _______________________________________________
> hibernate-dev mailing list
> hibernate-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hibernate-dev
12:32 p.m.
On 15 mars 2011, at 18:24, Emmanuel Bernard wrote:
On 15 mars 2011, at 16:43, Sanne Grinovero wrote:
> I guess we should prevent the indexing of ids in secondary elements?
If the associated element is an entity, it's perfectly valid to index its id and
query by it
//return all books when the author's id is 2
"author.id:2"
So if we find a way to support this use case and yours, let's go.
BTW I agree we should at least not store the norm.
Maybe something like that:
//default to todays' behavior minus the norm
@DocumentId(indexWhenEmbedded=@Field(...))
1 p.m.
Well not being this very urgent I'm going to create two JIRAs, on it,
one for the norms, and one for possibly changing the storage of the
optional field.
just wanted to mention it as while debugging I found it surprising behaviour.
*But* your last sentence got me a weird idea.
Nowadays if you have two related entities, both indexed, and one is
@ContainedIn in the other, we'll create two documents. Actually if you
think about it, we create two documents which contain exactly the same
values, just the fieldnames are different.
So what happens is that we're actually duplicating the size of the
index, or even more depending on the depth of the tree of indexed &&
related objects. And we also analyse the same text twice, using the
same analyser.
If we had something like "virtual fields", something for which we
dynamically map the fieldName to an internal different name, we could
re-route indexing and also queries built using the DSL in clever ways,
having them point the correct fields.
The downside is of course that we take away the option to directly
access the Document in FieldBridges and ClassBridges - nasty, but
we'll likely need that anyway as nothing is serializable anymore, so
we already need to provide some proxy object mimicking the Document,
to be used as a value container to being sent to the backend.
Other smaller advantages:
* We'll be able to intercept what FieldBridges actually write to,
don't remember now but in some cases we where stuck on that (like
detecting duplicates and conflicting fieldnames)
* some better JOIN implementations can use it, as long as the number
of elements is limited, by creating fieldname_1 fieldname_2, etc.. or
just using the same name but controlling the order in which they're
listed.
* Updating index - a single document can be split in parts, so that
when an entity is indexed we don't have to reload/rewrite many related
entities. Don't know the details, but remember having read some
limited form of this is possible.
I'm sure we can come up with more, let's see if it's worth the
complexity and other problems. I'd propose it for version 4, a nice
breaking change :)
Cheers,
Sanne
2011/3/15 Emmanuel Bernard <emmanuel(a)hibernate.org>:
On 15 mars 2011, at 18:24, Emmanuel Bernard wrote:
>
> On 15 mars 2011, at 16:43, Sanne Grinovero wrote:
>
>> I guess we should prevent the indexing of ids in secondary elements?
>
> If the associated element is an entity, it's perfectly valid to index its id and
query by it
>
> //return all books when the author's id is 2
> "author.id:2"
So if we find a way to support this use case and yours, let's go.
BTW I agree we should at least not store the norm.
Maybe something like that:
//default to todays' behavior minus the norm
@DocumentId(indexWhenEmbedded=@Field(...))
3:44 a.m.
New subject: HSEARCH virtual fields (WAS Re: About mapping of @IndexedEmbedded and @DocumentId)
It's an interesting idea but I am not sure it works in that many practical cases.
It's definitely worth exploring though.
Here are a few comments:
- does not work for embedded collections
- => does not work in a bidirectional @ManyToOne ot @*ToMany environment (quite
common)
- work in a unidirectional env where the associated entity has no back dependency
- will need to store _hibernate_class for the associated entities
- mental note to stop talking with you, it generates weird ideas
I've noticed that you're quite concerned about index size generally, do you think
size is a major drawback?
On 15 mars 2011, at 19:00, Sanne Grinovero wrote:
Well not being this very urgent I'm going to create two JIRAs, on
it,
one for the norms, and one for possibly changing the storage of the
optional field.
just wanted to mention it as while debugging I found it surprising behaviour.
*But* your last sentence got me a weird idea.
Nowadays if you have two related entities, both indexed, and one is
@ContainedIn in the other, we'll create two documents. Actually if you
think about it, we create two documents which contain exactly the same
values, just the fieldnames are different.
So what happens is that we're actually duplicating the size of the
index, or even more depending on the depth of the tree of indexed &&
related objects. And we also analyse the same text twice, using the
same analyser.
If we had something like "virtual fields", something for which we
dynamically map the fieldName to an internal different name, we could
re-route indexing and also queries built using the DSL in clever ways,
having them point the correct fields.
The downside is of course that we take away the option to directly
access the Document in FieldBridges and ClassBridges - nasty, but
we'll likely need that anyway as nothing is serializable anymore, so
we already need to provide some proxy object mimicking the Document,
to be used as a value container to being sent to the backend.
Other smaller advantages:
* We'll be able to intercept what FieldBridges actually write to,
don't remember now but in some cases we where stuck on that (like
detecting duplicates and conflicting fieldnames)
* some better JOIN implementations can use it, as long as the number
of elements is limited, by creating fieldname_1 fieldname_2, etc.. or
just using the same name but controlling the order in which they're
listed.
* Updating index - a single document can be split in parts, so that
when an entity is indexed we don't have to reload/rewrite many related
entities. Don't know the details, but remember having read some
limited form of this is possible.
I'm sure we can come up with more, let's see if it's worth the
complexity and other problems. I'd propose it for version 4, a nice
breaking change :)
Cheers,
Sanne
2011/3/15 Emmanuel Bernard <emmanuel(a)hibernate.org>:
>
> On 15 mars 2011, at 18:24, Emmanuel Bernard wrote:
>
>>
>> On 15 mars 2011, at 16:43, Sanne Grinovero wrote:
>>
>>> I guess we should prevent the indexing of ids in secondary elements?
>>
>> If the associated element is an entity, it's perfectly valid to index its id
and query by it
>>
>> //return all books when the author's id is 2
>> "author.id:2"
>
> So if we find a way to support this use case and yours, let's go.
> BTW I agree we should at least not store the norm.
>
> Maybe something like that:
>
> //default to todays' behavior minus the norm
> @DocumentId(indexWhenEmbedded=@Field(...))
>
>
1:49 p.m.
New subject: HSEARCH virtual fields (WAS Re: About mapping of @IndexedEmbedded and @DocumentId)
2011/3/16 Emmanuel Bernard <emmanuel(a)hibernate.org>:
It's an interesting idea but I am not sure it works in that many
practical cases. It's definitely worth exploring though.
Here are a few comments:
- does not work for embedded collections
no, but we don't support them yet ;)
Also embedded collections wouldn't have their own index, so I see no
point in applying such an optimization.
- => does not work in a bidirectional @ManyToOne ot @*ToMany
environment (quite common)
Agree, but in the Lucene community there's talking on
new possible
approaches to map parent/child relations of documents, like
LUCENE-2454,
which would be nice to have, but requires some sort of new
"intelligence" during indexing and querying.
Mentioning it as it looks like related to this, it would be a nice
complement (even tough only one parent/child relation per doc seems
supported by the idea).
- work in a unidirectional env where the associated entity has no
back dependency
- will need to store _hibernate_class for the associated entities
- mental note to stop talking with you, it generates weird ideas
I've noticed that you're quite concerned about index size generally, do you think
size is a major drawback?
Not *major* but when the index is too big, performance gets worse and
scaling gets harder.
I've seen some people cut mercilessly on features to get the index
size down to something that can be cached in memory.
Also an interesting talk at Apache Lucene eurocon 2010 mentioned the
strategy to have multiple copies of the index, with varying grades of
queries they could support, to store the smallest in memory, the
medium on fast SSDs, the biggest on mass storage: analysing usage
patterns he was able to route the biggest amount of queries to the
smallest and quickest index.
Infinispan helps a bit as you might be able to get help from other
nodes instead of going to mass storage, but still you want it all in
memory - if Infinispan has to swap out as well it won't look very
good, so you'll end up again figuring out how to slim down the index.
In conclusion, as a user I wouldn't like it to see a library forcing
me to store metadata in the index unless there where good reasons.
Cheers,
Sanne
On 15 mars 2011, at 19:00, Sanne Grinovero wrote:
> Well not being this very urgent I'm going to create two JIRAs, on it,
> one for the norms, and one for possibly changing the storage of the
> optional field.
> just wanted to mention it as while debugging I found it surprising behaviour.
>
> *But* your last sentence got me a weird idea.
>
> Nowadays if you have two related entities, both indexed, and one is
> @ContainedIn in the other, we'll create two documents. Actually if you
> think about it, we create two documents which contain exactly the same
> values, just the fieldnames are different.
> So what happens is that we're actually duplicating the size of the
> index, or even more depending on the depth of the tree of indexed &&
> related objects. And we also analyse the same text twice, using the
> same analyser.
>
> If we had something like "virtual fields", something for which we
> dynamically map the fieldName to an internal different name, we could
> re-route indexing and also queries built using the DSL in clever ways,
> having them point the correct fields.
>
> The downside is of course that we take away the option to directly
> access the Document in FieldBridges and ClassBridges - nasty, but
> we'll likely need that anyway as nothing is serializable anymore, so
> we already need to provide some proxy object mimicking the Document,
> to be used as a value container to being sent to the backend.
>
> Other smaller advantages:
> * We'll be able to intercept what FieldBridges actually write to,
> don't remember now but in some cases we where stuck on that (like
> detecting duplicates and conflicting fieldnames)
> * some better JOIN implementations can use it, as long as the number
> of elements is limited, by creating fieldname_1 fieldname_2, etc.. or
> just using the same name but controlling the order in which they're
> listed.
> * Updating index - a single document can be split in parts, so that
> when an entity is indexed we don't have to reload/rewrite many related
> entities. Don't know the details, but remember having read some
> limited form of this is possible.
>
> I'm sure we can come up with more, let's see if it's worth the
> complexity and other problems. I'd propose it for version 4, a nice
> breaking change :)
>
> Cheers,
> Sanne
>
>
>
>
> 2011/3/15 Emmanuel Bernard <emmanuel(a)hibernate.org>:
>>
>> On 15 mars 2011, at 18:24, Emmanuel Bernard wrote:
>>
>>>
>>> On 15 mars 2011, at 16:43, Sanne Grinovero wrote:
>>>
>>>> I guess we should prevent the indexing of ids in secondary elements?
>>>
>>> If the associated element is an entity, it's perfectly valid to index its
id and query by it
>>>
>>> //return all books when the author's id is 2
>>> "author.id:2"
>>
>> So if we find a way to support this use case and yours, let's go.
>> BTW I agree we should at least not store the norm.
>>
>> Maybe something like that:
>>
>> //default to todays' behavior minus the norm
>> @DocumentId(indexWhenEmbedded=@Field(...))
>>
>>
12:32 p.m.
2011/3/15 Emmanuel Bernard <emmanuel(a)hibernate.org>:
On 15 mars 2011, at 16:43, Sanne Grinovero wrote:
> I guess we should prevent the indexing of ids in secondary elements?
If the associated element is an entity, it's perfectly valid to index its id and
query by it
//return all books when the author's id is 2
"author.id:2"
No doubts on that being valid, but on the actual usefulness of this
being always enabled.
Hibernate Search doesn't strictly need it as it does with the class
and root-entity-id, and in other cases we have an "opt-in" approach
for fields.
I'd expect that users might not even know that they can write such a
query, as I found it surprising. I also suspect that for most people
it's just a pointless waste of resources so I think it should be an
option.
Even if we decide to keep it, I'd propose to disable norms on this field(s).
Cheers,
Sanne
4986
days inactive
4988
days old
8 comments
2 participants
participants (2)
-
Emmanuel Bernard -
Sanne Grinovero