9:07 a.m.
It seems that a few parameters should not be configurable per
transaction|batch
- use_compound_file
- max_field_length? not sure
- term_index_interval (is that a config influencing IndexReader?)
What do you think?
I am a bit worried to stick them as is in the project / book as they
are right now.
Let's try and sort this out today.
10:17 a.m.
On Tue, 11 Nov 2008 16:07:06 +0100, Emmanuel Bernard
<emmanuel(a)hibernate.org> wrote:
It seems that a few parameters should not be configurable per
transaction|batch
- use_compound_file
- max_field_length? not sure
- term_index_interval (is that a config influencing IndexReader?)
What do you think?
+ 1 for not configuring them per transaction|batch. I am not sure whether
you can actually
really 'harm' the system, but it just seems wrong.
In fact there is already a sanity check in
LuceneIndexingParameters.doSanityChecks() which should
print out a warning when the max_field_length for batch and index differ.
However, there is actually
a bug, batchParams is never used ;-)
I would go even one step further and add eg another paramter/flag to the
IndexWriterSetting enum -
'groupable'. If someone tries to set a paramter via the batch|transaction
syntax and the groupable
paramter is set to false we throw an exception. And yes, groupable is
probably a bad name ;-)
--Hardy
11:53 a.m.
maybe structural setting?
On Nov 11, 2008, at 11:17, Hardy Ferentschik wrote:
On Tue, 11 Nov 2008 16:07:06 +0100, Emmanuel Bernard
<emmanuel(a)hibernate.org
> wrote:
> It seems that a few parameters should not be configurable per
> transaction|batch
> - use_compound_file
> - max_field_length? not sure
> - term_index_interval (is that a config influencing IndexReader?)
>
> What do you think?
+ 1 for not configuring them per transaction|batch. I am not sure
whether you can actually
really 'harm' the system, but it just seems wrong.
In fact there is already a sanity check in
LuceneIndexingParameters.doSanityChecks() which should
print out a warning when the max_field_length for batch and index
differ. However, there is actually
a bug, batchParams is never used ;-)
I would go even one step further and add eg another paramter/flag to
the IndexWriterSetting enum -
'groupable'. If someone tries to set a paramter via the batch|
transaction syntax and the groupable
paramter is set to false we throw an exception. And yes, groupable
is probably a bad name ;-)
--Hardy
10:13 a.m.
what to you mean by structural setting?
I think we should always apply the same values for both transaction
and batch (shouldn't be possible to have them configured differently).
we had a chat about this, and as far as I remember the idea was to log
a warning for the moment.
If you explain me what a structural setting is I might take a look to
implement it (wand fix the bug Hardy has seen)
Sanne
2008/11/12 Hardy Ferentschik <hibernate(a)ferentschik.de>:
On Tue, 11 Nov 2008 18:53:14 +0100, Emmanuel Bernard
<emmanuel(a)hibernate.org> wrote:
> maybe structural setting?
+1 Sounds good to me.
10:24 a.m.
actually "term_index_interval" should have not harm if configured differently,
and can be useful to tune the batch indexing speed in a different way.
The other 2 should be invariant.
2008/11/12 Sanne Grinovero <sanne.grinovero(a)gmail.com>:
what to you mean by structural setting?
I think we should always apply the same values for both transaction
and batch (shouldn't be possible to have them configured differently).
we had a chat about this, and as far as I remember the idea was to log
a warning for the moment.
If you explain me what a structural setting is I might take a look to
implement it (wand fix the bug Hardy has seen)
Sanne
2008/11/12 Hardy Ferentschik <hibernate(a)ferentschik.de>:
> On Tue, 11 Nov 2008 18:53:14 +0100, Emmanuel Bernard
> <emmanuel(a)hibernate.org> wrote:
>
>> maybe structural setting?
>
> +1 Sounds good to me.
>
1:43 p.m.
So where would we put these?
under hibernate.search.[index name]. straight?
eg
hibernate.search.[index name].use_compound_file
On Nov 11, 2008, at 11:17, Hardy Ferentschik wrote:
On Tue, 11 Nov 2008 16:07:06 +0100, Emmanuel Bernard
<emmanuel(a)hibernate.org
> wrote:
> It seems that a few parameters should not be configurable per
> transaction|batch
> - use_compound_file
> - max_field_length? not sure
> - term_index_interval (is that a config influencing IndexReader?)
>
> What do you think?
+ 1 for not configuring them per transaction|batch. I am not sure
whether you can actually
really 'harm' the system, but it just seems wrong.
In fact there is already a sanity check in
LuceneIndexingParameters.doSanityChecks() which should
print out a warning when the max_field_length for batch and index
differ. However, there is actually
a bug, batchParams is never used ;-)
I would go even one step further and add eg another paramter/flag to
the IndexWriterSetting enum -
'groupable'. If someone tries to set a paramter via the batch|
transaction syntax and the groupable
paramter is set to false we throw an exception. And yes, groupable
is probably a bad name ;-)
--Hardy
4:59 p.m.
Actually I sat down and read more about compound file format.
Basically, instead of creating several files representing a single
segment, the compound format creates a single file for the whole
segment.
And you can have a mix of compound and non compound segments for a
given index. So you can set different settings for transactional and
batch in theory. Not sure why though.
Compounds Pro:
- less file handlers
Compounds Con:
- slower at indexing
- takes temporarily more space on disk as the uncompound segment is
converted to a compound segment when done
I don't think this affects the incremental copy so much are segments
are essentially read-only.
I believe this is not the story we tell to people in the doc, so we
need to fix that.
On Nov 11, 2008, at 10:07, Emmanuel Bernard wrote:
It seems that a few parameters should not be configurable per
transaction|batch
- use_compound_file
- max_field_length? not sure
- term_index_interval (is that a config influencing IndexReader?)
What do you think?
I am a bit worried to stick them as is in the project / book as they
are right now.
Let's try and sort this out today.
_______________________________________________
hibernate-dev mailing list
hibernate-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hibernate-dev
5:18 p.m.
Plus it seems the default value is compound = true, just like in
Lucene. This is not what we are saying in the doc.
Am I completely off-base?
On Nov 12, 2008, at 17:59, Emmanuel Bernard wrote:
Actually I sat down and read more about compound file format.
Basically, instead of creating several files representing a single
segment, the compound format creates a single file for the whole
segment.
And you can have a mix of compound and non compound segments for a
given index. So you can set different settings for transactional and
batch in theory. Not sure why though.
Compounds Pro:
- less file handlers
Compounds Con:
- slower at indexing
- takes temporarily more space on disk as the uncompound segment is
converted to a compound segment when done
I don't think this affects the incremental copy so much are segments
are essentially read-only.
I believe this is not the story we tell to people in the doc, so we
need to fix that.
On Nov 11, 2008, at 10:07, Emmanuel Bernard wrote:
> It seems that a few parameters should not be configurable per
> transaction|batch
> - use_compound_file
> - max_field_length? not sure
> - term_index_interval (is that a config influencing IndexReader?)
>
> What do you think?
> I am a bit worried to stick them as is in the project / book as
> they are right now.
>
> Let's try and sort this out today.
> _______________________________________________
> hibernate-dev mailing list
> hibernate-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hibernate-dev
_______________________________________________
hibernate-dev mailing list
hibernate-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hibernate-dev
4:48 a.m.
2 answers inline:
2008/11/13 Emmanuel Bernard <emmanuel(a)hibernate.org>:
Plus it seems the default value is compound = true, just like in
Lucene.
This is not what we are saying in the doc.
Am I completely off-base?
No, you are correct.
2008/11/12 Emmanuel Bernard <emmanuel(a)hibernate.org>:
So where would we put these?
under hibernate.search.[index name]. straight?
eg
hibernate.search.[index name].use_compound_file
this is not how it is working now, you still need to use either
.transaction or .batch.
Also you should use .indexwriter.transaction as omitting the
".indexwriter" is deprecated.
So I would agree with
hibernate.search.[index name].transaction
but this is not currently supported.
Also the current cfg parsing is expecting a tree of options, depth
having proprity, because of all
different configuration rules:
1- batch inheriting from transaction
2- use or not to use .indexwriter (backwards compatibility)
3- inheritance from .default
4- sharded nested configurations... defaulting to upper node.
maybe I forgot some more cases.
Therefore it would be more complex (currently) to NOT accept a
parameter under "transaction" or "batch"
but give priority for the root node for some cases; not so complex to
ALSO accept a parameter under .indexwriter
only (without the transaction or batch part).
Sanne
8:25 a.m.
Oh I totally forgot about .indexwriter.
OK so here is how it will be
we keep hibernate.search.[indexname].indexwriter.transaction/batch for
most with batch inheriting tx
And we support
hibernate.search.[indexname].indexwriter.use_compound_file
hibernate.search.[indexname].indexwriter.max_field_length
At least that's what in the book now :)
On Nov 13, 2008, at 05:48, Sanne Grinovero wrote:
2 answers inline:
2008/11/13 Emmanuel Bernard <emmanuel(a)hibernate.org>:
> Plus it seems the default value is compound = true, just like in
> Lucene.
> This is not what we are saying in the doc.
> Am I completely off-base?
No, you are correct.
2008/11/12 Emmanuel Bernard <emmanuel(a)hibernate.org>:
> So where would we put these?
> under hibernate.search.[index name]. straight?
>
> eg
> hibernate.search.[index name].use_compound_file
>
this is not how it is working now, you still need to use either
.transaction or .batch.
Also you should use .indexwriter.transaction as omitting the
".indexwriter" is deprecated.
So I would agree with
hibernate.search.[index name].transaction
but this is not currently supported.
Also the current cfg parsing is expecting a tree of options, depth
having proprity, because of all
different configuration rules:
1- batch inheriting from transaction
2- use or not to use .indexwriter (backwards compatibility)
3- inheritance from .default
4- sharded nested configurations... defaulting to upper node.
maybe I forgot some more cases.
Therefore it would be more complex (currently) to NOT accept a
parameter under "transaction" or "batch"
but give priority for the root node for some cases; not so complex to
ALSO accept a parameter under .indexwriter
only (without the transaction or batch part).
Sanne
5906
days inactive
5908
days old
10 comments
3 participants
participants (3)
-
Emmanuel Bernard -
Hardy Ferentschik -
Sanne Grinovero