12:09 p.m.
Hi all,
Yesterday I had a discussion with Sanne on irc [3] about the new api to
access index readers in HS4.0. We couldn't complete our discussion
yesterday, so let's continue here.
As explained in the forum [1], there is currently no good solution for
getting a reader with a subset of the indexes in a sharded environment.
Currently two basic ideas came to mind:
A - Have a SearchFactory.openIndexReader(Class<?> c,
FullTextFilterImplementor...): This is similar to how the IndexManager's
are gathered at query time, and is probably therefore easy to understand
B - (to be further reviewed) Have something like
searchFactory.indexReaders().withShardingOptions( X, Y
).includeType(Class<?> z).openIndexReader(). This also adds the ability
to get an IndexReader for multiple classes. But we need to think about
the .withShardingOptions (or something similar), what input should we
support here? Sharding properties are mostly based on some entity
property(/ies), probably easy to be encode as String. The (custom)
sharding strategy may use such String to select the proper index
managers. Using a String object for identifying which index managers to
use looks fine to me. It will be compatible with current implementation
of custom sharding strategies where one might use the Lucene document at
addition time, or if an entity instance will also be passed (see
discussion [2]), the properties of that entity can probably encoded to
some String. And if HS will cover the mapping/have support for Strings
as identifiers for sharding instead of a user defined mapping of the
index (integer) in the array of IndexManagers, that would be awesome :)
(Relieves the pain of having some mapping that should be stored
somewhere, which I currently do).
Still, we need to know the use cases there might be, i.e. which
flexibility the API should offer.
As is also mentioned in [1], there is currently no direct access to the
index managers, so getting a FSDirectory is currently not possible in
4.0alpha1. I think HS should support this to offer the flexibility to
work on the Lucene indexes directly (for example, to build an auto
completion/spell check index from an existing index)
Let's start by setting up some requirements?
---------
*1 Have access to IndexReader for one class
*2 Have access to IndexReader with a subset of IndexManagers based on
sharding strategy. Sharding strategies are mostly based on some
propert(y/ies) of an entity instance, which can likely be encoded to
some String.
*3 Have access to index directories (FSDirectory/...). Unlike previous
versions (< HS4.0) it would be nice if this uses the ShardingStrategy
instance in use, so mapping is completely and exclusively done in a
ShardingStrategy
* ...
---------
Please extend/modify the list of requirements if you think something is
missing/incorrect and drop your ideas/thoughts about the mentioned
ideas.
Elmer
[1] https://forum.hibernate.org/viewtopic.php?p=2448000#p2448000
[2]
http://www.mailinglistarchive.com/html/hibernate-dev@lists.jboss.org/2011...
[3] IRC log:
<elmervc> sannegrinovero, have you read/did you have time to think about
https://forum.hibernate.org/viewtopic.php?p=2448000#p2448000
<sannegrinovero> hi elmervc , yes I've read it. my next thing on the
todo is to make some prototype, as I'm not happy with the current ideas:
<sannegrinovero> elmervc, are you blocked by this? the workaround is
very simple
<sannegrinovero> generally, I'm wondering if we can avoid having to
expose the DirectoryProviders. I would want them gone from the public
API, but of course limitations like this are not acceptable.
<elmervc> sannegrinovero, I'm branching this migration, so it's not
really blocking. But I would like to try the new H core/search, so for
that to work I need access to the subset of indices
<elmervc> What workaround were you thinking about ?
<elmervc> Just construct an index reader/FSDirs myself using 'hardcoded'
paths ?
<sannegrinovero> nono that's ugly..
<sannegrinovero> elmervc, all logic to open this IR is in
org.hibernate.search.impl.ImmutableSearchFactory.openIndexReader(Class<?>...)
<sannegrinovero> elmervc, and it's just a couple of lines to change ;)
<sannegrinovero> the problem is more how to make it easy to consume
<elmervc> Ok, I'll look into that :)
<elmervc> Using filters is not a good idea?
<sannegrinovero> yes I liked your suggestion. but is it enough ?
<sannegrinovero> and how would the methods look like?
<sannegrinovero> (i.e. the signature)
<elmervc> SearchFactory.openIndexReader(Class<?> c,
FullTextFilterImplementor[] filters) , or what do you mean?
<sannegrinovero> I'd prefer SearchFactory.openIndexReader(Class<?> c,
FullTextFilterImplementor... filters)
<elmervc> But I'm not sure if this covers all use cases of sharding
<sannegrinovero> elmervc, the methods don't need necessarily be defined
on the SearchFactory. We can think of something like
searchFactory.indexReaders().withShardingOptions( X, Y
).includeType(Class<?> z).openIndexReader() .. how does that look like?
<sannegrinovero> I'm just tossing out some ideas, but then we should
bring this up to the mailing list.
<elmervc> the .includeType , do you mean that multiple classes can be
included?
<sannegrinovero> yes
<sannegrinovero> basically the indexReaders() method would open a
context, private to this invocation chain only. (i.e. not affecting
other threads invoking .indexReaders() )
<elmervc> Sounds cool. But then we need to think about
the .withShardingOptions, or something similar. For transparancy it's
best to have something similar to the methods in the ShardingStrategy
interface
<elmervc> Or something similar to what is done @ querytime, i.e.
FullTextFilterImplementors
<elmervc> The point is, we need to know what other use cases one might
have
<elmervc> That's related to how sharding is done, i.e. ... might be a
field in the doc , full text filter, ...
<elmervc> (doc = doc to be added)
<sannegrinovero> yes exactly I need use cases to understand this, that's
why your feedback is very much appreciated :)
<elmervc> sannegrinovero, For example, our sharding strategy is based on
some field in an entity that is added to the Lucene Document (actually,
it has a @Field anno, and this field is removed from the Lucene Document
in the shardingstrategy.getDirectoryProviderForAddition(...)
<sannegrinovero> elmervc, lol that prooves another discussion I had
recently in proposing that we should pass the entity instance and not
the document to the sharding strategy.
<elmervc> It might be usefull indeed, but in our case it's easier to use
a Field in the doc, because that field will always have the same name,
i.e. we can reuse the same sharding strategy.
<sannegrinovero> elmervc, this discussion is very interesting but I'm
busy in other chats now which I can't postpone. Could you please
synthesize this and send a mail to the developer list?
1:26 p.m.
On 25 août 2011, at 12:09, Elmer van Chastelet wrote:
As is also mentioned in [1], there is currently no direct access to
the
index managers, so getting a FSDirectory is currently not possible in
4.0alpha1. I think HS should support this to offer the flexibility to
work on the Lucene indexes directly (for example, to build an auto
completion/spell check index from an existing index)
Do you have other uses cases. We always had in mind to support dictionary based use case
in HSearch, so I don't see this as a motivating enough use case..
1:27 p.m.
On Thu, 25 Aug 2011 12:09:50 +0200, Elmer van Chastelet
<evanchastelet(a)gmail.com> wrote:
Currently two basic ideas came to mind:
A - Have a SearchFactory.openIndexReader(Class<?> c,
FullTextFilterImplementor...): This is similar to how the IndexManager's
are gathered at query time, and is probably therefore easy to understand
I favor this version. I find the reason why you can and in your case even
have
to specify FullTextFilterImplementor instances not so obvious. It requires
some deeper understanding on how things hang together.
B - (to be further reviewed) Have something like
searchFactory.indexReaders().withShardingOptions( X, Y
).includeType(Class<?> z).openIndexReader(). This also adds the ability
to get an IndexReader for multiple classes.
I find 'withShardingOptions' too vague. How exactly does this relate to A
and solved its usecase? Also I find the API a little too fluent.
As is also mentioned in [1], there is currently no direct access to
the
index managers, so getting a FSDirectory is currently not possible in
4.0alpha1. I think HS should support this to offer the flexibility to
work on the Lucene indexes directly (for example, to build an auto
completion/spell check index from an existing index)
That's a question of moving 'getAllIndexesManager' from
SearchFactoryImplementor
higher up the hierarchy, right?
--Hardy
1:37 p.m.
2011/8/25 Elmer van Chastelet <evanchastelet(a)gmail.com>:
Hi all,
Yesterday I had a discussion with Sanne on irc [3] about the new api to
access index readers in HS4.0. We couldn't complete our discussion
yesterday, so let's continue here.
As explained in the forum [1], there is currently no good solution for
getting a reader with a subset of the indexes in a sharded environment.
Currently two basic ideas came to mind:
A - Have a SearchFactory.openIndexReader(Class<?> c,
FullTextFilterImplementor...): This is similar to how the IndexManager's
are gathered at query time, and is probably therefore easy to understand
Current signature is not accepting the FullTextFilterImplementor, but
accepts multiple classes:
SearchFactory.openIndexReader(Class<?>... entities);
Since we can't use two varargs on the same method, this won't work
unless you're suggesting that we should support a single type only.
B - (to be further reviewed) Have something like
searchFactory.indexReaders().withShardingOptions( X, Y
).includeType(Class<?> z).openIndexReader(). This also adds the ability
to get an IndexReader for multiple classes. But we need to think about
the .withShardingOptions (or something similar), what input should we
support here? Sharding properties are mostly based on some entity
property(/ies), probably easy to be encode as String. The (custom)
sharding strategy may use such String to select the proper index
managers. Using a String object for identifying which index managers to
use looks fine to me. It will be compatible with current implementation
of custom sharding strategies where one might use the Lucene document at
addition time, or if an entity instance will also be passed (see
discussion [2]), the properties of that entity can probably encoded to
some String. And if HS will cover the mapping/have support for Strings
as identifiers for sharding instead of a user defined mapping of the
index (integer) in the array of IndexManagers, that would be awesome :)
(Relieves the pain of having some mapping that should be stored
somewhere, which I currently do).
FullTextFilterImplementor could work as a withShardingOptions
parameter, but while it's true that in the end it's going to filter
some values, I wonder if the concept of Filters is misleading in this
case.
Still, we need to know the use cases there might be, i.e. which
flexibility the API should offer.
I'd add another option:
C - SearchFactory.openIndexReader(String... indexName);
This is simple, but it is in no way delegating to the ShardingStrategy
to make the index names choice which I think would be way more usable.
As is also mentioned in [1], there is currently no direct access to
the
index managers, so getting a FSDirectory is currently not possible in
4.0alpha1. I think HS should support this to offer the flexibility to
work on the Lucene indexes directly (for example, to build an auto
completion/spell check index from an existing index)
Why would you need direct access to a Directory? isn't it enough to
provide access to the IndexReader ?
Let's start by setting up some requirements?
---------
*1 Have access to IndexReader for one class
*2 Have access to IndexReader with a subset of IndexManagers based on
sharding strategy. Sharding strategies are mostly based on some
propert(y/ies) of an entity instance, which can likely be encoded to
some String.
*3 Have access to index directories (FSDirectory/...). Unlike previous
versions (< HS4.0) it would be nice if this uses the ShardingStrategy
instance in use, so mapping is completely and exclusively done in a
ShardingStrategy
We can't provide access to a "virtual" Directory exposing the contents
of multiple Directories, that's possible with an IndexReader only.
* ...
---------
Please extend/modify the list of requirements if you think something is
missing/incorrect and drop your ideas/thoughts about the mentioned
ideas.
Elmer
[1] https://forum.hibernate.org/viewtopic.php?p=2448000#p2448000
[2]
http://www.mailinglistarchive.com/html/hibernate-dev@lists.jboss.org/2011...
[3] IRC log:
<elmervc> sannegrinovero, have you read/did you have time to think about
https://forum.hibernate.org/viewtopic.php?p=2448000#p2448000
<sannegrinovero> hi elmervc , yes I've read it. my next thing on the
todo is to make some prototype, as I'm not happy with the current ideas:
<sannegrinovero> elmervc, are you blocked by this? the workaround is
very simple
<sannegrinovero> generally, I'm wondering if we can avoid having to
expose the DirectoryProviders. I would want them gone from the public
API, but of course limitations like this are not acceptable.
<elmervc> sannegrinovero, I'm branching this migration, so it's not
really blocking. But I would like to try the new H core/search, so for
that to work I need access to the subset of indices
<elmervc> What workaround were you thinking about ?
<elmervc> Just construct an index reader/FSDirs myself using 'hardcoded'
paths ?
<sannegrinovero> nono that's ugly..
<sannegrinovero> elmervc, all logic to open this IR is in
org.hibernate.search.impl.ImmutableSearchFactory.openIndexReader(Class<?>...)
<sannegrinovero> elmervc, and it's just a couple of lines to change ;)
<sannegrinovero> the problem is more how to make it easy to consume
<elmervc> Ok, I'll look into that :)
<elmervc> Using filters is not a good idea?
<sannegrinovero> yes I liked your suggestion. but is it enough ?
<sannegrinovero> and how would the methods look like?
<sannegrinovero> (i.e. the signature)
<elmervc> SearchFactory.openIndexReader(Class<?> c,
FullTextFilterImplementor[] filters) , or what do you mean?
<sannegrinovero> I'd prefer SearchFactory.openIndexReader(Class<?> c,
FullTextFilterImplementor... filters)
<elmervc> But I'm not sure if this covers all use cases of sharding
<sannegrinovero> elmervc, the methods don't need necessarily be defined
on the SearchFactory. We can think of something like
searchFactory.indexReaders().withShardingOptions( X, Y
).includeType(Class<?> z).openIndexReader() .. how does that look like?
<sannegrinovero> I'm just tossing out some ideas, but then we should
bring this up to the mailing list.
<elmervc> the .includeType , do you mean that multiple classes can be
included?
<sannegrinovero> yes
<sannegrinovero> basically the indexReaders() method would open a
context, private to this invocation chain only. (i.e. not affecting
other threads invoking .indexReaders() )
<elmervc> Sounds cool. But then we need to think about
the .withShardingOptions, or something similar. For transparancy it's
best to have something similar to the methods in the ShardingStrategy
interface
<elmervc> Or something similar to what is done @ querytime, i.e.
FullTextFilterImplementors
<elmervc> The point is, we need to know what other use cases one might
have
<elmervc> That's related to how sharding is done, i.e. ... might be a
field in the doc , full text filter, ...
<elmervc> (doc = doc to be added)
<sannegrinovero> yes exactly I need use cases to understand this, that's
why your feedback is very much appreciated :)
<elmervc> sannegrinovero, For example, our sharding strategy is based on
some field in an entity that is added to the Lucene Document (actually,
it has a @Field anno, and this field is removed from the Lucene Document
in the shardingstrategy.getDirectoryProviderForAddition(...)
<sannegrinovero> elmervc, lol that prooves another discussion I had
recently in proposing that we should pass the entity instance and not
the document to the sharding strategy.
<elmervc> It might be usefull indeed, but in our case it's easier to use
a Field in the doc, because that field will always have the same name,
i.e. we can reuse the same sharding strategy.
<sannegrinovero> elmervc, this discussion is very interesting but I'm
busy in other chats now which I can't postpone. Could you please
synthesize this and send a mail to the developer list?
_______________________________________________
hibernate-dev mailing list
hibernate-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hibernate-dev
2:13 p.m.
On Thu, 2011-08-25 at 13:26 +0200, Emmanuel Bernard wrote:
On 25 août 2011, at 12:09, Elmer van Chastelet wrote:
> As is also mentioned in [1], there is currently no direct access to the
> index managers, so getting a FSDirectory is currently not possible in
> 4.0alpha1. I think HS should support this to offer the flexibility to
> work on the Lucene indexes directly (for example, to build an auto
> completion/spell check index from an existing index)
Do you have other uses cases. We always had in mind to support dictionary based use case
in HSearch, so I don't see this as a motivating enough use case..
Sorry, my
mistake. Access to FSDirectory is only needed for indexes
created externally, not used by Hibernate Search. I don't have valid use
cases for that anymore :)
2:25 p.m.
C - SearchFactory.openIndexReader(String... indexName);
This is simple
I quite like it. And even though you are not going through the
but it is in no way delegating to the ShardingStrategy
to make the index names choice which I think would be way more usable.
but you can still implement the original usecase we had. I think
this solution could be a reasonable trade-off.
2:48 p.m.
Current signature is not accepting the FullTextFilterImplementor,
but
accepts multiple classes:
SearchFactory.openIndexReader(Class<?>... entities);
Since we can't use two varargs on the same method, this won't work
unless you're suggesting that we should support a single type only.
No,
that's not what I meant. Maybe a ShardingOptions arg may become
usefull?
And that's actually another requirement, access to IndexReader for
multiple classes/indexnames applying the same 'sharding options' for all
of them.
I'd add another option:
C - SearchFactory.openIndexReader(String... indexName);
This is simple, but it is in no way delegating to the ShardingStrategy
to make the index names choice which I think would be way more usable.
+1. Q: Is there a difference between the readers returned by:
SearchFactory.openIndexReader("A");
SearchFactory.openIndexReader(SubA.class);
if A and SubA (subclass of A) both have the bare @Indexed annotation,
thus sharing the same index name. That is, will
SearchFactory.openIndexReader(SubA.class) perform some filtering to only
return docs from SubA entities?
Why would you need direct access to a Directory? isn't it enough
to
provide access to the IndexReader ?
No need for that, my mistake.
So requirements become:
---------
*1 Have access to IndexReader for one class
*2 Have access to IndexReader with a subset of IndexManagers based on
*3 Have access to IndexReader for multiple classes/indexnames applying
the same 'sharding options' for all of them.
---------
As Emmanuel mentioned, can we think of use cases where we would like to
have access to Lucene Directories (/IndexManagers), which is currently
mentioned in the docs:
http://docs.jboss.org/hibernate/search/4.0/reference/en-US/html_single/#d...
?
Elmer
On Thu, 2011-08-25 at 13:37 +0200, Sanne Grinovero wrote:
> 2011/8/25 Elmer van Chastelet <evanchastelet(a)gmail.com>:
> > Hi all,
> >
> > Yesterday I had a discussion with Sanne on irc [3] about the new api to
> > access index readers in HS4.0. We couldn't complete our discussion
> > yesterday, so let's continue here.
> > As explained in the forum [1], there is currently no good solution for
> > getting a reader with a subset of the indexes in a sharded environment.
> >
> > Currently two basic ideas came to mind:
> > A - Have a SearchFactory.openIndexReader(Class<?> c,
> > FullTextFilterImplementor...): This is similar to how the IndexManager's
> > are gathered at query time, and is probably therefore easy to understand
>
> Current signature is not accepting the FullTextFilterImplementor, but
> accepts multiple classes:
> SearchFactory.openIndexReader(Class<?>... entities);
>
> Since we can't use two varargs on the same method, this won't work
> unless you're suggesting that we should support a single type only.
>
> >
> > B - (to be further reviewed) Have something like
> > searchFactory.indexReaders().withShardingOptions( X, Y
> > ).includeType(Class<?> z).openIndexReader(). This also adds the ability
> > to get an IndexReader for multiple classes. But we need to think about
> > the .withShardingOptions (or something similar), what input should we
> > support here? Sharding properties are mostly based on some entity
> > property(/ies), probably easy to be encode as String. The (custom)
> > sharding strategy may use such String to select the proper index
> > managers. Using a String object for identifying which index managers to
> > use looks fine to me. It will be compatible with current implementation
> > of custom sharding strategies where one might use the Lucene document at
> > addition time, or if an entity instance will also be passed (see
> > discussion [2]), the properties of that entity can probably encoded to
> > some String. And if HS will cover the mapping/have support for Strings
> > as identifiers for sharding instead of a user defined mapping of the
> > index (integer) in the array of IndexManagers, that would be awesome :)
> > (Relieves the pain of having some mapping that should be stored
> > somewhere, which I currently do).
>
> FullTextFilterImplementor could work as a withShardingOptions
> parameter, but while it's true that in the end it's going to filter
> some values, I wonder if the concept of Filters is misleading in this
> case.
>
> > Still, we need to know the use cases there might be, i.e. which
> > flexibility the API should offer.
>
I'd add another option:
C - SearchFactory.openIndexReader(String... indexName);
This is simple, but it is in no way delegating to the ShardingStrategy
to make the index names choice which I think would be way more usable.
>
> > As is also mentioned in [1], there is currently no direct access to the
> > index managers, so getting a FSDirectory is currently not possible in
> > 4.0alpha1. I think HS should support this to offer the flexibility to
> > work on the Lucene indexes directly (for example, to build an auto
> > completion/spell check index from an existing index)
>
Why would you need direct access to a Directory? isn't it enough
to
provide access to the IndexReader ?
>
> >
> > Let's start by setting up some requirements?
> > ---------
> > *1 Have access to IndexReader for one class
> > *2 Have access to IndexReader with a subset of IndexManagers based on
> > sharding strategy. Sharding strategies are mostly based on some
> > propert(y/ies) of an entity instance, which can likely be encoded to
> > some String.
> > *3 Have access to index directories (FSDirectory/...). Unlike previous
> > versions (< HS4.0) it would be nice if this uses the ShardingStrategy
> > instance in use, so mapping is completely and exclusively done in a
> > ShardingStrategy
>
> We can't provide access to a "virtual" Directory exposing the contents
> of multiple Directories, that's possible with an IndexReader only.
>
> > * ...
> > ---------
> >
> > Please extend/modify the list of requirements if you think something is
> > missing/incorrect and drop your ideas/thoughts about the mentioned
> > ideas.
> >
> >
> > Elmer
> >
> >
> >
> > [1] https://forum.hibernate.org/viewtopic.php?p=2448000#p2448000
> > [2]
> >
http://www.mailinglistarchive.com/html/hibernate-dev@lists.jboss.org/2011...
> > [3] IRC log:
> >
> > <elmervc> sannegrinovero, have you read/did you have time to think about
> > https://forum.hibernate.org/viewtopic.php?p=2448000#p2448000
> > <sannegrinovero> hi elmervc , yes I've read it. my next thing on the
> > todo is to make some prototype, as I'm not happy with the current ideas:
> > <sannegrinovero> elmervc, are you blocked by this? the workaround is
> > very simple
> > <sannegrinovero> generally, I'm wondering if we can avoid having to
> > expose the DirectoryProviders. I would want them gone from the public
> > API, but of course limitations like this are not acceptable.
> > <elmervc> sannegrinovero, I'm branching this migration, so it's
not
> > really blocking. But I would like to try the new H core/search, so for
> > that to work I need access to the subset of indices
> > <elmervc> What workaround were you thinking about ?
> > <elmervc> Just construct an index reader/FSDirs myself using
'hardcoded'
> > paths ?
> > <sannegrinovero> nono that's ugly..
> > <sannegrinovero> elmervc, all logic to open this IR is in
> >
org.hibernate.search.impl.ImmutableSearchFactory.openIndexReader(Class<?>...)
> > <sannegrinovero> elmervc, and it's just a couple of lines to change
;)
> > <sannegrinovero> the problem is more how to make it easy to consume
> > <elmervc> Ok, I'll look into that :)
> > <elmervc> Using filters is not a good idea?
> > <sannegrinovero> yes I liked your suggestion. but is it enough ?
> > <sannegrinovero> and how would the methods look like?
> > <sannegrinovero> (i.e. the signature)
> > <elmervc> SearchFactory.openIndexReader(Class<?> c,
> > FullTextFilterImplementor[] filters) , or what do you mean?
> > <sannegrinovero> I'd prefer
SearchFactory.openIndexReader(Class<?> c,
> > FullTextFilterImplementor... filters)
> > <elmervc> But I'm not sure if this covers all use cases of sharding
> > <sannegrinovero> elmervc, the methods don't need necessarily be
defined
> > on the SearchFactory. We can think of something like
> > searchFactory.indexReaders().withShardingOptions( X, Y
> > ).includeType(Class<?> z).openIndexReader() .. how does that look like?
> > <sannegrinovero> I'm just tossing out some ideas, but then we should
> > bring this up to the mailing list.
> > <elmervc> the .includeType , do you mean that multiple classes can be
> > included?
> > <sannegrinovero> yes
> > <sannegrinovero> basically the indexReaders() method would open a
> > context, private to this invocation chain only. (i.e. not affecting
> > other threads invoking .indexReaders() )
> > <elmervc> Sounds cool. But then we need to think about
> > the .withShardingOptions, or something similar. For transparancy it's
> > best to have something similar to the methods in the ShardingStrategy
> > interface
> > <elmervc> Or something similar to what is done @ querytime, i.e.
> > FullTextFilterImplementors
> > <elmervc> The point is, we need to know what other use cases one might
> > have
> > <elmervc> That's related to how sharding is done, i.e. ... might be a
> > field in the doc , full text filter, ...
> > <elmervc> (doc = doc to be added)
> > <sannegrinovero> yes exactly I need use cases to understand this,
that's
> > why your feedback is very much appreciated :)
> > <elmervc> sannegrinovero, For example, our sharding strategy is based on
> > some field in an entity that is added to the Lucene Document (actually,
> > it has a @Field anno, and this field is removed from the Lucene Document
> > in the shardingstrategy.getDirectoryProviderForAddition(...)
> > <sannegrinovero> elmervc, lol that prooves another discussion I had
> > recently in proposing that we should pass the entity instance and not
> > the document to the sharding strategy.
> > <elmervc> It might be usefull indeed, but in our case it's easier to
use
> > a Field in the doc, because that field will always have the same name,
> > i.e. we can reuse the same sharding strategy.
> > <sannegrinovero> elmervc, this discussion is very interesting but I'm
> > busy in other chats now which I can't postpone. Could you please
> > synthesize this and send a mail to the developer list?
> >
> > _______________________________________________
> > hibernate-dev mailing list
> > hibernate-dev(a)lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/hibernate-dev
> >
5:56 p.m.
As Emmanuel mentioned, can we think of use cases where we would like
to
have access to Lucene Directories (/IndexManagers), which is currently
mentioned in the docs:
http://docs.jboss.org/hibernate/search/4.0/reference/en-US/html_single/#d...
?
Elmer
Yes that's an important question I would like we could answer before
going to betas;
Ideally I would like to remove "Directory" as a concept of Hibernate
Search users, for various reasons; the whole concept to "identify an
index" and then apply options / operations to it should go through an
IndexManager, so that an application can replace the IndexManager
implementation with one supporting writes to a remote cluster without
having to change their code.
Are there really use cases left in which we need direct access to a
DirectoryProvider ?
We won't of course prevent people from casting their IndexManager to a
DirectoryBasedIndexManager, which exposes a getDirectoryProvider()
mainly for testing reasons, but then I'd expect you know what you're
doing.
Getting back on topic; I had started another thread about Sharding, in
which I suggested that nowadays sharding is coupled to an index name -
at least as far as how it's defined in the configuration - while it
should be coupled to an entity type.
So I wouldn't say that we apply sharing to the index named
"com.mystuff.indexName", but that I want to shard the entity
"com.mystuff.Person".
If we do, the IndexShardingStrategy interface would receive entity
instances instead of o.a.l.Document instances, and it could be
typesafe.
I had no time to make a proof of concept, but I'd try to go in the
following direction:
IndexReader ir = searchFactory.indexReaders()
.forEntity(Car.class)
.enableFullTextFilter("colorsFilter") //must be a
ShardSensitiveOnlyFilter
.setParameter("color","blue")
.forEntity(Animals.class)
.openIndexReader();
searchFactory.indexReaders().closeIndexReader( ir );
I'm not totally happy with this idea, as
1) we could enforce the fact it's a ShardSensitiveOnlyFilter with some
typesafety
2) The IndexShardingStrategy implementor has quite some work to do to
collect all parameters back from the filters - I might like it more to
return the IndexShardingStrategy and allow people to set options to it
directly, but that's not an option as the strategy is used
concurrently by other components.
Proposal D - reusing the FullTextQuery as a proper context to define
the target indexes
FullTextQuery ftQuery = fullTextSession.createFullTextQuery( query,
Driver.class, Animals.class );
ftQuery.enableFullTextFilter( "bestDriver" );
ftQuery.openIndexReader();
Where to close it? I don't feel to like adding a closeIndexReader to
FullTextQuery. We could better return a new kind of MultiReader which
is able to close itself properly when invoking the close() method on
the IndexReader itself.
Sanne
1:05 p.m.
We need to take a decision regarding API changes to unlock the beta releases;
so we should take a decision on this subject.
Summarizing the last proposals:
1) Use a builder API, which defines
- targeted entities
- enabled shard-aware filters
-- parameters on these filters
2) searchFactory.openReader(String ... index names)
pro: simple
con: hard to use: user has to duplicate logic of sharding strategies
and be aware of index names
3) searchFactory.openReader(Class[] entities,
ShardSensitiveOnlyFilter[] filterInstances)
pro: relatively simple
cons:
- user has to initialize arrays
- where does he get the filterInstances from? I guess by using
their constructor directly.
4) provide a ".openReader()" on the Query, basically reusing the query
object as the builder root and inheriting selected entities and
enabled filters.
the open question I had on this is where does it make more sense to
have it closed. My proposal was to allow invoking "close()" on the
returned reader itself.
My opinion:
in the long term (for Hibernate Search 4.1?) provide both #1 and #2,
or both #4 and #2, as #2 provides more flexibility and (#1 || #4) are
more practical.
In the short term I'd provide #2 only for 4.0: IMHO the API makes
sense enough to be exposed to the power user and makes sure nobody is
prevented to use it while we have more time to think on the usability
details of #1 or #4.
(I don't like the signature of #3 but I'm not against it's simplicity
if somebody can rethink the method parameters)
what's your opinion?
Sanne
On 25 August 2011 17:56, Sanne Grinovero <sanne(a)hibernate.org> wrote:
> As Emmanuel mentioned, can we think of use cases where we would
like to
> have access to Lucene Directories (/IndexManagers), which is currently
> mentioned in the docs:
> http://docs.jboss.org/hibernate/search/4.0/reference/en-US/html_single/#d...
> ?
>
> Elmer
Yes that's an important question I would like we could answer before
going to betas;
Ideally I would like to remove "Directory" as a concept of Hibernate
Search users, for various reasons; the whole concept to "identify an
index" and then apply options / operations to it should go through an
IndexManager, so that an application can replace the IndexManager
implementation with one supporting writes to a remote cluster without
having to change their code.
Are there really use cases left in which we need direct access to a
DirectoryProvider ?
We won't of course prevent people from casting their IndexManager to a
DirectoryBasedIndexManager, which exposes a getDirectoryProvider()
mainly for testing reasons, but then I'd expect you know what you're
doing.
Getting back on topic; I had started another thread about Sharding, in
which I suggested that nowadays sharding is coupled to an index name -
at least as far as how it's defined in the configuration - while it
should be coupled to an entity type.
So I wouldn't say that we apply sharing to the index named
"com.mystuff.indexName", but that I want to shard the entity
"com.mystuff.Person".
If we do, the IndexShardingStrategy interface would receive entity
instances instead of o.a.l.Document instances, and it could be
typesafe.
I had no time to make a proof of concept, but I'd try to go in the
following direction:
IndexReader ir = searchFactory.indexReaders()
.forEntity(Car.class)
.enableFullTextFilter("colorsFilter") //must be a
ShardSensitiveOnlyFilter
.setParameter("color","blue")
.forEntity(Animals.class)
.openIndexReader();
searchFactory.indexReaders().closeIndexReader( ir );
I'm not totally happy with this idea, as
1) we could enforce the fact it's a ShardSensitiveOnlyFilter with some
typesafety
2) The IndexShardingStrategy implementor has quite some work to do to
collect all parameters back from the filters - I might like it more to
return the IndexShardingStrategy and allow people to set options to it
directly, but that's not an option as the strategy is used
concurrently by other components.
Proposal D - reusing the FullTextQuery as a proper context to define
the target indexes
FullTextQuery ftQuery = fullTextSession.createFullTextQuery( query,
Driver.class, Animals.class );
ftQuery.enableFullTextFilter( "bestDriver" );
ftQuery.openIndexReader();
Where to close it? I don't feel to like adding a closeIndexReader to
FullTextQuery. We could better return a new kind of MultiReader which
is able to close itself properly when invoking the close() method on
the IndexReader itself.
Sanne
1:56 p.m.
+1 for #2 for 4.0
in the long run I like the simplicity of #3. I found #1 a bit of an
overkill at least
what I saw last in the discussion around this.
--Hardy
On Tue, 06 Sep 2011 13:05:01 +0200, Sanne Grinovero <sanne(a)hibernate.org>
wrote:
We need to take a decision regarding API changes to unlock the beta
releases;
so we should take a decision on this subject.
Summarizing the last proposals:
1) Use a builder API, which defines
- targeted entities
- enabled shard-aware filters
-- parameters on these filters
2) searchFactory.openReader(String ... index names)
pro: simple
con: hard to use: user has to duplicate logic of sharding strategies
and be aware of index names
3) searchFactory.openReader(Class[] entities,
ShardSensitiveOnlyFilter[] filterInstances)
pro: relatively simple
cons:
- user has to initialize arrays
- where does he get the filterInstances from? I guess by using
their constructor directly.
4) provide a ".openReader()" on the Query, basically reusing the query
object as the builder root and inheriting selected entities and
enabled filters.
the open question I had on this is where does it make more sense to
have it closed. My proposal was to allow invoking "close()" on the
returned reader itself.
My opinion:
in the long term (for Hibernate Search 4.1?) provide both #1 and #2,
or both #4 and #2, as #2 provides more flexibility and (#1 || #4) are
more practical.
In the short term I'd provide #2 only for 4.0: IMHO the API makes
sense enough to be exposed to the power user and makes sure nobody is
prevented to use it while we have more time to think on the usability
details of #1 or #4.
(I don't like the signature of #3 but I'm not against it's simplicity
if somebody can rethink the method parameters)
what's your opinion?
Sanne
On 25 August 2011 17:56, Sanne Grinovero <sanne(a)hibernate.org> wrote:
>> As Emmanuel mentioned, can we think of use cases where we would like to
>> have access to Lucene Directories (/IndexManagers), which is currently
>> mentioned in the docs:
>> http://docs.jboss.org/hibernate/search/4.0/reference/en-US/html_single/#d...
>> ?
>>
>> Elmer
>
> Yes that's an important question I would like we could answer before
> going to betas;
> Ideally I would like to remove "Directory" as a concept of Hibernate
> Search users, for various reasons; the whole concept to "identify an
> index" and then apply options / operations to it should go through an
> IndexManager, so that an application can replace the IndexManager
> implementation with one supporting writes to a remote cluster without
> having to change their code.
>
> Are there really use cases left in which we need direct access to a
> DirectoryProvider ?
> We won't of course prevent people from casting their IndexManager to a
> DirectoryBasedIndexManager, which exposes a getDirectoryProvider()
> mainly for testing reasons, but then I'd expect you know what you're
> doing.
>
>
> Getting back on topic; I had started another thread about Sharding, in
> which I suggested that nowadays sharding is coupled to an index name -
> at least as far as how it's defined in the configuration - while it
> should be coupled to an entity type.
> So I wouldn't say that we apply sharing to the index named
> "com.mystuff.indexName", but that I want to shard the entity
> "com.mystuff.Person".
>
> If we do, the IndexShardingStrategy interface would receive entity
> instances instead of o.a.l.Document instances, and it could be
> typesafe.
>
> I had no time to make a proof of concept, but I'd try to go in the
> following direction:
>
> IndexReader ir = searchFactory.indexReaders()
> .forEntity(Car.class)
> .enableFullTextFilter("colorsFilter") //must be a
> ShardSensitiveOnlyFilter
> .setParameter("color","blue")
> .forEntity(Animals.class)
> .openIndexReader();
>
> searchFactory.indexReaders().closeIndexReader( ir );
>
> I'm not totally happy with this idea, as
> 1) we could enforce the fact it's a ShardSensitiveOnlyFilter with some
> typesafety
> 2) The IndexShardingStrategy implementor has quite some work to do to
> collect all parameters back from the filters - I might like it more to
> return the IndexShardingStrategy and allow people to set options to it
> directly, but that's not an option as the strategy is used
> concurrently by other components.
>
> Proposal D - reusing the FullTextQuery as a proper context to define
> the target indexes
>
> FullTextQuery ftQuery = fullTextSession.createFullTextQuery( query,
> Driver.class, Animals.class );
> ftQuery.enableFullTextFilter( "bestDriver" );
> ftQuery.openIndexReader();
>
> Where to close it? I don't feel to like adding a closeIndexReader to
> FullTextQuery. We could better return a new kind of MultiReader which
> is able to close itself properly when invoking the close() method on
> the IndexReader itself.
>
> Sanne
4615
days inactive
4627
days old
10 comments
4 participants
participants (4)
-
Elmer van Chastelet -
Emmanuel Bernard -
Hardy Ferentschik -
Sanne Grinovero