<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><div><div></div><div><br><div><div>On Jun 10, 2009, at 16:38, Chase Seibert wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">Emmanuel,<br><br>Here is the patch with the new changes:<br><ul><li>If getDirectoryProvidersForQuery() returns zero providers, the search will not execute.<br></li><li>Added methods String getName(), Map<String, Object> getParameters() and isPreSearchShardFilter() to FullTextFilter.</li> <ul><li>Not sure what the value of a FullTextFilterImplementor would be, but I can add that if you want.</li></ul><li>Removed casting to FullTextFilterImpl.</li><li>CustomerShardingStrategy returns all shards for getDirectoryProvidersForDeletion().<br> </li><li>In CustomerShardingStrategy.getFilter(), changed filter name comparison to be case sensitive.</li><li>Added a new Filter sub-class, ShardFilterImpl, where isPreSearchShardFilter() == true.</li><li>FullTextQueryImpl.buildFilters() only get isPreSearchShardFilter() == false filters.</li> <li>FullTextQueryImpl only passes isPreSearchShardFilter() == true filters to getDirectoryProvidersForQuery().</li></ul><br> -Chase<br><br><br><div class="gmail_quote">On Wed, Jun 10, 2009 at 3:38 PM, Emmanuel Bernard <span dir="ltr"><<a href="mailto:emmanuel@hibernate.org">emmanuel@hibernate.org</a>></span> wrote:<br> <blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div style=""><div>I like the patch, we can apply a slightly modified version of it.</div><div> <br></div><div>A few comments:</div><div><br></div><div>* In private DirectoryProvider[] getDirectoryProviders(DocumentBuilderIndexedEntity builder)</div><div><br></div><div><div>if ( directoryProviders != null && directoryProviders.length > 0 ) return directoryProviders;</div> <div><br></div><div>What's the reasoning for returning all shards if this method returns 0 providers. It seems to me that in this situation you want to avoid executing the query altogether.</div><div><br></div><div>* In the testcase, CustomerShardingStrategy does cast FullTextFilter into FullTextFilterImpl</div> <div>That sucks. </div><div>We need to introduce a FullTextFilterImplementor interface with getName() ( and maybe Map<String, Object> getParameters() - we can add this one later);</div><div>We would then get </div><div> public DirectoryProvider<?>[] getDirectoryProvidersForQuery(FullTextFilterImplementor[] filters) {</div><div><br></div><div>* not sure why the CustomerShardingStrategy does not return all shards for getDirectoryProvidersForDeletion</div> <div><br></div><div>* in CustomerShardingStrategy.getFilter</div><div>equalsIgnoreCase is wrong. Filter names are case sensitive.</div><div><br></div><div>* as I said in my reply to Sanne, I would like to get a way to declare a filter to be shard sensitive only. Either by providing a special class or by providing a flag, not sure.</div> <div><br></div><div>@FullTextFilterDef(name="customer", impl=ShardSensitiveOnly.class)</div><div><br></div><div>Because I think we raise an exception if an unknown filter is requested.</div><div><br></div><div>In this case, the ShardSensitiveOnly impl can be ignored by the filter chaining we do otherwise.</div> <div><br></div><div>Emmanuel</div></div><br><div><div><div></div><div class="h5"><div>On Jun 3, 2009, at 16:08, Chase Seibert wrote:</div><br></div></div><blockquote type="cite"><div><div></div><div class="h5">Sanne,<br> <br>I have implemented your suggestion for IndexShardingStrategy to optionally provide a set of DirectoryProviders BEFORE the search based on one or more FullTextFilters. Using this change, I was able to optimize my specific case to search only hitting the relevant shards.<br> <br>I have not yet implemented your labeled shard idea, nor your shard on enum idea. If we can agree on this change first, I think I can implement those on top of this.<br><br>Please see attached svn .patch (diff) file. I have tested the patch on 3.1.1 and 3.2.0. Any feedback is welcome.<br> <br> -Chase<br><br><br><div class="gmail_quote">On Wed, Jun 3, 2009 at 1:27 PM, Sanne Grinovero <span dir="ltr"><<a href="mailto:sanne.grinovero@gmail.com" target="_blank">sanne.grinovero@gmail.com</a>></span> wrote:<br> <blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> I am having a similar need in these days; this should be a very useful<br> feature, but I'd like more something I could use with the existing API<br> like<br> <br> enableFullTextFilter( "MyShardsSelectionStrategy" ).setParameter( ... )<br> <br> a practical example:<br> enableFullTextFilter( "LanguageFilter" ).setParameter( "IT-it" )<br> <br> The existing IndexShardingStrategy should be able to be smarter and<br> have something like<br> <br> DirectoryProvider<?>[] getDirectoryProvidersForQuery( filters && options )<br> <br> So a smart ShardingStrategy could do some selections considering this.<br> <br> I'm currently using sharding to shard my index on 25 different<br> languages (using per-language stemmers), so this would<br> be useful but I'd especially need to be able to "label" my different<br> DirectoryProviders using String identifiers,<br> I'd suggest to add a getName() to the DirectoryProvider interface: I<br> would use that to store countrycodes and<br> keep a map<String,DirectoryProvider> in my ShardingStrategy, so I can<br> easily select the right DP when<br> the LanguageFilter is enabled.<br> <br> Another usage would be to shard an entity on an Enumerated property:<br> in this case an appropriate ShardingStrategy<br> could be provided by Search and auto-configured by reading the<br> possible enum values: that would be a very easy way<br> to enable sharding on an entity.<br> <br> Sanne<br> <br> 2009/6/3 Emmanuel Bernard <<a href="mailto:emmanuel@hibernate.org" target="_blank">emmanuel@hibernate.org</a>>:<br> ><br> ><br> > Begin forwarded message:<br> ><br> > From: <a href="mailto:chase.seibert%2Bopensubscriber@gmail.com" target="_blank">chase.seibert+opensubscriber@gmail.com</a><br> > Date: June 3, 2009 09:21:21 PDT<br> > To: <a href="mailto:emmanuel@hibernate.org" target="_blank">emmanuel@hibernate.org</a><br> > Subject: Re: Re: [hibernate-dev] HSearch: Using sharding and avoiding query<br> > on multiple shards<br> > Reply-To: <a href="mailto:chase.seibert%2Bopensubscriber@gmail.com" target="_blank">chase.seibert+opensubscriber@gmail.com</a><br> > Emmanuel,<br> ><br> > Regarding HSEARCH-251, and<br> > <a href="http://www.opensubscriber.com/message/hibernate-dev@lists.jboss.org/9770383.html" target="_blank">http://www.opensubscriber.com/message/hibernate-dev@lists.jboss.org/9770383.html</a><br> ><br> > Being able to query just a single shard or subset of shards would be<br> > awesome. I was thinking of a similar API:<br> ><br> > IndexShardingStrategy:<br> > public DirectoryProvider<?>[]<br> > getDirectoryProviderForShard(int shardNum);<br> ><br> > FullTextQuery:<br> > public void enableShardFilter(int shardNum);<br> > public void enableShardFilters(int[] shardNums);<br> ><br> > FullTextQuery.buildSearcher() would need to be modified to call<br> > getDirectoryProviderForShard() for each shardNum if shardNums are set,<br> > otherwise it should continue to use getDirectoryProvidersForAllShards();<br> ><br> > Calling this API from a consumer's stand-point would look like:<br> > FullTextQuery fullTextQuery =<br> > fullTextSession.createFullTextQuery(luceneQuery, entityClass);<br> > fullTextQuery.enableShardFilter(5);<br> > fullTextQuery.list();<br> ><br> > This could be changed to pass named shards easily. I could prototype this<br> > and submit a .patch if you are interested.<br> ><br> > -Chase<br> ><br> > --<br> > This message was sent on behalf of <a href="mailto:chase.seibert%2Bopensubscriber@gmail.com" target="_blank">chase.seibert+opensubscriber@gmail.com</a> at<br> > openSubscriber.com<br> > <a href="http://www.opensubscriber.com/message/hibernate-dev@lists.jboss.org/9800518.html" target="_blank">http://www.opensubscriber.com/message/hibernate-dev@lists.jboss.org/9800518.html</a><br> ><br> ><br> > _______________________________________________<br> > hibernate-dev mailing list<br> > <a href="mailto:hibernate-dev@lists.jboss.org" target="_blank">hibernate-dev@lists.jboss.org</a><br> > <a href="https://lists.jboss.org/mailman/listinfo/hibernate-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/hibernate-dev</a><br> ><br> ><br> </blockquote></div><br> </div></div><span><getDirectoryProvidersForQuery.patch></span></blockquote></div><br></div></blockquote></div><br> <span><all.patch></span></blockquote></div><br></div></div></div></body></html>