On 25 Aug 2009, at 10:13, Navin Surtani wrote: Looks good. But I think we should also support Lucene queries directly, as they provide a richer set of searchable terms and constraints. Or perhaps impls like BooleanQuery to combine queries. E.g.: Query luceneQuery = createComplexLuceneQuery(); CacheQuery cq = new QueryFactory(cache).getQuery(luceneQuery); See: http://lucene.apache.org/java/2_4_1/api/core/org/apache/lucene/search/Que... Yeah this needs thought, and is a more generic question than something specific to querying. Essentially, how does module X get its configuration from XML, where the XML parsing and schema logic is in the core module? Vladimir, any thoughts? Perhaps what we need is some form of module bootstrap, or a callback of sorts? I suppose there are 2 parts to this: the first being a bean to parse and hold configuration specifics, so that the parser can instantiate this bean, e.g., QueryConfiguration, and then an actual bootstrap to bind module-specific components to the ComponentRegistry so it participates in various lifecycle phases of the cache. -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org

On 25 Aug 2009, at 11:09, Emmanuel Bernard wrote: Aren't there standards around this already? -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org

I agree with ou it's annoying (though you don't need a QP necessarily). I do think the DSL approach will work best. Well but then how do you pass each side which are Lucene queries themselves.

On 25 août 09, at 14:27, Manik Surtani wrote: Yes I was thinking about a DSL used for Hibernate Search and maybe all of Lucene if the HS integration benefits offer no value towards simplicity (but I think i can offer value).

I've been thinking about a DSL to build Lucene queries in the last day. What do you think of this proposal? A few remarks: - it asks the analyzer so that we correctly apply the analyzer on terms - it has a few query factory methods - it contains a few orthogonal operations - I am not quite satisfied with how boolean is handled, any idea? Examples SealedQueryBuilder qb = searchFactory.withEntityAnalyzer(Address.class); Query luceneQuery = qb.must(Occurs.MUST) .add( qb.boolean(Occurs.Should) .add( qb.term("city", "Atlanta").boostedTo(4).createQuery() ) .add( qb.term("address1", "Peachtree").fuzzy().createQuery() ) ) .add( qb.from("movingDate", "200604").to("201201").exclusive().createQuery() ) .createQuery(); Analyzer choice queryBuilder.withAnalyzer(Analyzer) queryBuilder.withEntityAnalyzer(Class<?>) queryBuilder.basedOnEntityAnalyzer(Class<?>) .overridesForField(String field, Analyzer) .overridesForField(String field, Analyzer) .build() //sucky name returns a SealedQueryBuilder //sucky name SealedQueryBuilder contains the factory methods Factory methods Hosted onSealedQueryBuilder .term(String field, String text) //define a new query .term(String field, String text) //define a new query .ignoreAnalyzer() //ignore the analyzer, optional .fuzzy() //API prevent wildcard calls, optional .threshold() //optional .prefixLengh() //optional .term(String field, String value) .wildcard() //API prevent fuzzy calls, optional //range query .from(String field, String text) .exclusive() //optional .to(String text) .exclusive() //optional .constantScore() //optional, due to constantScoreRangeQuery but in practice inherited from the common operations //match all docs .all() //phrase query .phrase(String field) .ignoreAnalyzer() //ignore the analyzer, optional .addWord(String text) //at least one .addWord(String text) .sentence(String text) //do we need that? .slop() //optional //search multiple fields for same value .searchInMultipleFields() .onField(String field) .boostedTo(float) //optional .ignoreAnalyzer() //optional .onField(String field) .forWords(String) //do we need that? .forWord(String) Boolean operations SealedQueryBuilder contains the boolean methods .boolean(Occurs occurs) .add( qb.from().to() ) .add( ... ) Works on all queries .boostedTo() .constantScore() .filter(Filter) //filter the current query .scoreMultipliedByField(field) //FieldScoreQuery + FunctionQuery?? //Not backed .createQuery() Todo Span*Queries MultiPhraseQuery - needs to fillup all accepted terms FieldScoreQuery ValueSourceQuery FuzzyLikeThis MoreLikeThis On 25 août 09, at 16:43, Manik Surtani wrote:

On 26 août 09, at 22:08, Hardy Ferentschik wrote: Remember, Hibernate Search's mission is to make full-text search as easy to use as possible to increase the overall technology adoption. There are several advantages to the DSL API listed below, but let's compare my example and the Lucene equivalent and see if you can still claim the API to be useless with a straight face. SealedQueryBuilder qb = searchFactory.withEntityAnalyzer(Address.class); Query luceneQuery = qb.must(Occurs.MUST) .add( qb.boolean(Occurs.Should) .add( qb.term("city", "Atlanta").boostedTo(4).createQuery() ) .add( qb.term("address1", "Peachtree").fuzzy().threshold(. 7).createQuery() ) ) .add( qb.from("movingDate", "200604").to("201201").exclusive().createQuery() ) .createQuery(); vs BooleanQuery luceneQuery = new BooleanQuery(); BooleanQuery addressLocationQuery = new BooleanQuery(); Query city = new TermQuery( new Term("city", "Atlanta") ); city.setBoost(4f); addressLocationQuery.add(BooleanClause.Occur.Should, city); Query address1 = new FuzzyQuery( new Term("address1", "Peachtree"), . 7 ); addressLocationQuery.add(BooleanClause.Occur.Should, address1); luceneQuery.add(BooleanClause.Occur.Must, addressLocationQuery); Query range = new RangeQuery( new Term("movingDate", "200604"), new Term("movingDate", "201201", false); luceneQuery.add(BooleanClause.Occur.Must, range); Advantages: - the query is readable and understandable even to new Lucene users. BTW the example is a quite simple one, it does not involve filter, search in multiple fields, query negation etc. - I have normalized some operations that require knowledge of the lucene query hierarchy (eg. ConstantScoreQuery, ConstantScorePrefixQuery, ConstrantScoreRangeQuery or PrefixQuery vs WildcardQuery) - the API shows available options right away using IDE auto- completion, not by looking at the Query hierarchy and its implementations - the API does take the analyzer into account which means that I can take my input and use it without thinking much about the underlying analyzer used at indexing time. In the example, my plain Lucene rewrite of the query will very likely fail because "Atlanta" and "Peachtree" should really be "atlanta" and "peachtree". In the API, we have the analyzer and can take that into account. Likewise for synonyms, phonetic approximation etc. Even worse, trying to search a user query containing several words in different fields is quite difficult in plain Lucene. In the new API it could look like: String search = "harry potter"; SealedQueryBuilder qb = searchFactory.withEntityAnalyzer(Book.class); Query luceneQuery = qb.searchInMultipleFields() .onField("title").boostedTo(4) .onField("title_ngram") .onField("description") .onField("description_ngram").boostedTo(.25) .forWords(search); vs String search = "harry potter"; Analyzer analyzer = searchFactory.getAnalyzer(Book.class); Map<String,Float> boostPerField = new HashMap<String,Float>(2); // boost factors boostPerField.put( "title", (float) 4); boostPerField.put( "title_ngram", (float) 1); boostPerField.put( "description", (float) 1); boostPerField.put( "description_ngram", (float) .25); BooleanQuery luceneQuery = new BooleanQuery(); for ( Map.Entry<String, Float> entry : boostPerField.entrySet() ) { final String fieldName = entry.getKey(); final Float boost = entry.getValue(); List<String> terms = new ArrayList<String>(); try { Reader reader = new StringReader(search); TokenStream stream = analyzer.tokenStream( fieldName, reader); Token token = new Token(); token = stream.next(token); while (token != null) { if (token.termLength() != 0) { String term = new String(token.termBuffer(), 0, token.termLength()); terms.add( term ); } token = stream.next(token); } } catch ( IOException e ) { throw new RuntimeException("IO exception while reading String stream??", e); } for (String term : terms) { TermQuery termQuery = new TermQuery( new Term( fieldName, term ) ); termQuery.setBoost( boost ); luceneQuery.add( termQuery, BooleanClause.Occur.SHOULD ); } } Did I make my case?

Just wanted to get this topic re-started again. Essentially what I think this project/DSL/module/thingy-bob is thought to become: - A simple package where a user can build Lucene queries without having to know too much about Lucene itself. If I'm headed down the wrong thought path then just thwack me. On 26 Aug 2009, at 21:08, Hardy Ferentschik wrote: What's gained I believe is the fact that people can build complex lucene queries easier. Currently, it's a bit clunky imo so if we provide a cleaner way to build them it can prove beneficial to any lucene user (myself included for querying on Infinispan). Any other thoughts? Navin Surtani Intern Infinispan Intern JBoss Cache Searchable

Very elegant. I'm generally a big fan of 'builder' patterns like this, but this really isn't a DSL, is it? :) When you first mentioned a DSL I had visions of defining a new grammar and an ANTLR parser, etc. But that is overkill. This approach certainly works, and will almost certainly perform better too. One question: for the sake of brevity, why SealedQueryBuilder instead of QueryBuilder ? :) Also, I still think that if this is a generic helper factory that helps you build Lucene queries - and has no knowledge of how and where the query is used (why should it?) - then this should be something people can use outside of HS or Infinispan. E.g., directly with Lucene. On 26 Aug 2009, at 12:39, Emmanuel Bernard wrote: -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org

On 27 Aug 2009, at 13:06, Emmanuel Bernard wrote: I thought this stuff you mentioned made sense: Perhaps rename the static factory methods to something like: QueryBuilder.getQueryBuilder(Analyzer) QueryBuilder.getQueryBuilder(Class<?>) and QueryBuilder instances have overrideAnalyzerForField(String, Analyzer). Why do you need the build() method at the end? Not it doesn't have to be either ASL or even hosted at Apache. I guess what I am suggesting is perhaps even a separate project - LuceneQueryBuilder or something - which plain-old-Lucene users could use as well. Doesn't matter where it's hosted or what the license is - as long as its ASL or LGPL :) -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org

On 27 Aug 2009, at 16:10, Emmanuel Bernard wrote: Ok. Just make sure we use a different maven module or something so that there are no dependencies on the rest of HS or Hibernate. Otherwise spinning out will be a PITA. Lucene should be the only dependencies of this code. Cheers -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org

On 25 août 09, at 12:46, Manik Surtani wrote: None that I am aware of.