[hibernate-dev] HSearch: Using sharding and avoiding query on multiple shards

[Emmanuel Bernard]

On  Aug 1, 2008, at 13:42, Sanne Grinovero wrote:

> Hello Emmanuel,
>
> 2008/7/31 Emmanuel Bernard <emmanuel at hibernate.org>:
>> On  Jul 31, 2008, at 09:22, Sanne Grinovero wrote:
>>> about the API, wouldn't it make more sense to have it look like a  
>>> filter?
>> can you give more details?
>
> I was just thinking about the name  
> "fullTextQuery.setShardHint("Sony");":
> I wouldn't call it a "hint", but a filter as it could affect the  
> results;
> A "hint" sounds like you are trying to improve the performance in
> a way that shouldn't change the result, so:
>
> fullTextQuery.enableFullTextFilter("Sony")
>
> and it could differ from a normal FullTextFilter only by it's concrete
> implementation.
> Just my 2cents, as I think the effect is the same.

Interesting concept and much more transparent. Not sure how easy it is  
to do that though. A typical filter is cached per IndexReader. We  
cannot do that for the "special" filter as opening the index defeats  
the purpose. Lucene filters are applied per IndexReader so too late in  
the game.

>
> the feature looks great, but in my case I would need the ability in
> the ShardingStrategy to create new
> indexes; what do you think about that? I mean the size of the arrays
> could need to grow.

Yes that's a feature I thought about but it means we will run into a  
lot of concurrency issues (the HSearch config is all done at init time  
today). If we do that this needs to be well thought and I am not sure  
how feasible it is.

>
> Basically all my content is "clustered" in some macrocategories, and
> usually the search is done after
> having selected the category: so it would be perfect to have actually
> different indexes per cat.,
> but eventually someone could need to add a new category, the
> shardingStrategy would need to write
> a new empty index.
> I would like also the possibility to move away from array-indexes to
> some other identifier for the shards;

I am not sure what you gain from that. In any ways, your  
ShardingStrategy can do the conversion from your cat name to the shard  
index.

>
> in my specific case I would love to use something like the PK of the
> category: this could enable
> an easy filter selection (category could be the parameter of the
> filter) and enable something like
> "Cascade delete the index" on category removal.
> This could become a special implementation of ShardingStrategy, to be
> mandatory when using this kind of filtering?
>
> btw, I've committed some more fixes for HSEARCH-241
>
> kind regards,
> Sanne