[hibernate-commits] Hibernate SVN: r19171 - in search/trunk/hibernate-search/src: main/java/org/hibernate/search/annotations and 3 other directories.

Author: sannegrinovero Date: 2010-04-05 11:21:34 -0400 (Mon, 05 Apr 2010) New Revision: 19171 Added: search/trunk/hibernate-search/src/main/java/org/hibernate/search/annotations/CharFilterDef.java search/trunk/hibernate-search/src/test/resources/org/hibernate/search/test/analyzer/solr/mapping-chars.properties Modified: search/trunk/hibernate-search/src/main/docbook/en-US/modules/mapping.xml search/trunk/hibernate-search/src/main/java/org/hibernate/search/annotations/AnalyzerDef.java search/trunk/hibernate-search/src/main/java/org/hibernate/search/impl/SolrAnalyzerBuilder.java search/trunk/hibernate-search/src/test/java/org/hibernate/search/test/analyzer/solr/SolrAnalyzerTest.java search/trunk/hibernate-search/src/test/java/org/hibernate/search/test/analyzer/solr/Team.java Log: HSEARCH-477 Support for the new Solr's character filters (Gustavo Fernandes) Modified: search/trunk/hibernate-search/src/main/docbook/en-US/modules/mapping.xml =================================================================== --- search/trunk/hibernate-search/src/main/docbook/en-US/modules/mapping.xml 2010-04-05 14:08:35 UTC (rev 19170) +++ search/trunk/hibernate-search/src/main/docbook/en-US/modules/mapping.xml 2010-04-05 15:21:34 UTC (rev 19171) @@ -693,6 +693,12 @@ </listitem> <listitem> + <para>a list of char filters: each char filter is responsible to + pre-process input characters before the tokenization. Char filters can add, + change or remove characters; one common usage is for characters normalization</para> + </listitem> + + <listitem> <para>a tokenizer: responsible for tokenizing the input stream into individual words</para> </listitem> @@ -704,16 +710,17 @@ </listitem> </itemizedlist> - <para>This separation of tasks - a tokenizer followed by a list of + <para>This separation of tasks - a list of char filters, and a tokenizer followed by a list of filters - allows for easy reuse of each individual component and let you build your customized analyzer in a very flexible way (just like - Lego). Generally speaking the <classname>Tokenizer</classname> starts - the analysis process by turning the character input into tokens which + Lego). Generally speaking the <classname>char filters</classname> do some + pre-processing in the character input, then the <classname>Tokenizer</classname> starts + the tokenizing process by turning the character input into tokens which are then further processed by the <classname>TokenFilter</classname>s. Hibernate Search supports this infrastructure by utilizing the Solr analyzer framework. Make sure to add<filename> solr-core.jar and - </filename><filename>solr-common.jar</filename> to your classpath to - use analyzer definitions. In case you also want to utilizing a + </filename><filename>solr-solrj.jar</filename> to your classpath to + use analyzer definitions. In case you also want to use the snowball stemmer also include the <filename>lucene-snowball.jar.</filename> Other Solr analyzers might depend on more libraries. For example, the @@ -727,6 +734,11 @@ framework</title> <programlisting>@AnalyzerDef(name="customanalyzer", + charFilters = { + @CharFilterDef(factory = MappingCharFilterFactory.class, params = { + @Parameter(name = "mapping", value = "org/hibernate/search/test/analyzer/solr/mapping-chars.properties") + }) + }, tokenizer = @TokenizerDef(factory = StandardTokenizerFactory.class), filters = { @TokenFilterDef(factory = ISOLatin1AccentFilterFactory.class), @@ -741,9 +753,12 @@ }</programlisting> </example> - <para>A tokenizer is defined by its factory which is responsible for - building the tokenizer and using the optional list of parameters. This - example use the standard tokenizer. A filter is defined by its factory + <para>A char filter is defined by its factory which is responsible for + building the char filter and using the optional list of parameters. + In our example, a mapping char filter is used, and will replace + characters in the input based on the rules specified in the mapping + file. A tokenizer is also defined by its factory. + This example use the standard tokenizer. A filter is defined by its factory which is responsible for creating the filter instance using the optional parameters. In our example, the StopFilter filter is built reading the dedicated words property file and is expected to ignore @@ -751,7 +766,7 @@ factory.</para> <warning> - <para>Filters are applied in the order they are defined in the + <para>Filters and char filters are applied in the order they are defined in the <classname>@AnalyzerDef</classname> annotation. Make sure to think twice about this order.</para> </warning> @@ -800,13 +815,53 @@ <section> <title>Available analyzers</title> - <para>Solr and Lucene come with a lot of useful default tokenizers and - filters. You can find a complete list of tokenizer factories and + <para>Solr and Lucene come with a lot of useful default char filters, tokenizers and + filters. You can find a complete list of char filter factories, tokenizer factories and filter factories at <ulink url="http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters&quo...;. Let check a few of them.</para> <table> + <title>Some of the available char filters</title> + <tgroup cols="3"> + <thead> + <row> + <entry align="center">Factory</entry> + + <entry align="center">Description</entry> + + <entry align="center">parameters</entry> + </row> + </thead> + <tbody> + <row> + <entry>MappingCharFilterFactory</entry> + + <entry>Replaces one or more characters with one or more characters, based on mappings + specified in the resource file</entry> + + <entry><para><literal>mapping</literal>: points to a resource file containing the mappings + using the format: + <literallayout> + "á" => "a" + "ñ" => "n" + "ø" => "o" + </literallayout> + </para></entry> + </row> + <row> + <entry>HTMLStripCharFilterFactory</entry> + + <entry>Remove HTML standard tags, keeping the text</entry> + + <entry>none</entry> + </row> + </tbody> + </tgroup> + + </table> + + <table> <title>Some of the available tokenizers</title> <tgroup cols="3"> @@ -833,7 +888,7 @@ <entry>HTMLStripStandardTokenizerFactory</entry> <entry>Remove HTML tags, keep the text and pass it to a - StandardTokenizer</entry> + StandardTokenizer. @Deprecated, use the HTMLStripCharFilterFactory instead</entry> <entry>none</entry> </row> Modified: search/trunk/hibernate-search/src/main/java/org/hibernate/search/annotations/AnalyzerDef.java =================================================================== --- search/trunk/hibernate-search/src/main/java/org/hibernate/search/annotations/AnalyzerDef.java 2010-04-05 14:08:35 UTC (rev 19170) +++ search/trunk/hibernate-search/src/main/java/org/hibernate/search/annotations/AnalyzerDef.java 2010-04-05 15:21:34 UTC (rev 19171) @@ -53,6 +53,11 @@ String name(); /** + * @return CharFilters used. The filters are applied in the defined order + */ + CharFilterDef[] charFilters() default { }; + + /** * @return Tokenizer used. */ TokenizerDef tokenizer(); Added: search/trunk/hibernate-search/src/main/java/org/hibernate/search/annotations/CharFilterDef.java =================================================================== --- search/trunk/hibernate-search/src/main/java/org/hibernate/search/annotations/CharFilterDef.java (rev 0) +++ search/trunk/hibernate-search/src/main/java/org/hibernate/search/annotations/CharFilterDef.java 2010-04-05 15:21:34 UTC (rev 19171) @@ -0,0 +1,53 @@ +/* $Id$ + * + * Hibernate, Relational Persistence for Idiomatic Java + * + * Copyright (c) 2009, Red Hat, Inc. and/or its affiliates or third-party contributors as + * indicated by the @author tags or express copyright attribution + * statements applied by the authors. All third-party contributions are + * distributed under license by Red Hat, Inc. + * + * This copyrighted material is made available to anyone wishing to use, modify, + * copy, or redistribute it subject to the terms and conditions of the GNU + * Lesser General Public License, as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY + * or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License + * for more details. + * + * You should have received a copy of the GNU Lesser General Public License + * along with this distribution; if not, write to: + * Free Software Foundation, Inc. + * 51 Franklin Street, Fifth Floor + * Boston, MA 02110-1301 USA + */ +package org.hibernate.search.annotations; + +import org.apache.solr.analysis.CharFilterFactory; + +import java.lang.annotation.Documented; +import java.lang.annotation.ElementType; +import java.lang.annotation.Retention; +import java.lang.annotation.RetentionPolicy; +import java.lang.annotation.Target; + +/** + * Define a <code>CharFilterFactory</code> and its parameters + * + * @author Gustavo Fernandes + */ +(a)Retention(RetentionPolicy.RUNTIME) +@Target({ ElementType.TYPE, ElementType.FIELD, ElementType.METHOD}) +@Documented +public @interface CharFilterDef { + /** + * @return the <code>TokenFilterFactory</code> class which shall be instantiated. + */ + public abstract Class<? extends CharFilterFactory> factory(); + + /** + * @return Optional parameters passed to the <code>CharFilterFactory</code>. + */ + public abstract Parameter[] params() default { }; +} Property changes on: search/trunk/hibernate-search/src/main/java/org/hibernate/search/annotations/CharFilterDef.java ___________________________________________________________________ Name: svn:keywords + Id Modified: search/trunk/hibernate-search/src/main/java/org/hibernate/search/impl/SolrAnalyzerBuilder.java =================================================================== --- search/trunk/hibernate-search/src/main/java/org/hibernate/search/impl/SolrAnalyzerBuilder.java 2010-04-05 14:08:35 UTC (rev 19170) +++ search/trunk/hibernate-search/src/main/java/org/hibernate/search/impl/SolrAnalyzerBuilder.java 2010-04-05 15:21:34 UTC (rev 19171) @@ -29,6 +29,7 @@ import java.util.Collections; import org.apache.lucene.analysis.Analyzer; +import org.apache.solr.analysis.CharFilterFactory; import org.apache.solr.analysis.TokenizerChain; import org.apache.solr.analysis.TokenFilterFactory; import org.apache.solr.analysis.TokenizerFactory; @@ -36,6 +37,7 @@ import org.apache.solr.common.ResourceLoader; import org.hibernate.search.annotations.AnalyzerDef; +import org.hibernate.search.annotations.CharFilterDef; import org.hibernate.search.annotations.TokenizerDef; import org.hibernate.search.annotations.TokenFilterDef; import org.hibernate.search.annotations.Parameter; @@ -65,7 +67,9 @@ tokenFactory.init( getMapOfParameters( token.params() ) ); final int length = analyzerDef.filters().length; + final int charLength = analyzerDef.charFilters().length; TokenFilterFactory[] filters = new TokenFilterFactory[length]; + CharFilterFactory[] charFilters = new CharFilterFactory[charLength]; ResourceLoader resourceLoader = new HibernateSearchResourceLoader(); for ( int index = 0 ; index < length ; index++ ) { TokenFilterDef filterDef = analyzerDef.filters()[index]; @@ -75,7 +79,15 @@ ((ResourceLoaderAware)filters[index]).inform( resourceLoader ); } } - return new TokenizerChain(tokenFactory, filters); + for ( int index = 0 ; index < charFilters.length ; index++ ) { + CharFilterDef charFilterDef = analyzerDef.charFilters()[index]; + charFilters[index] = (CharFilterFactory) instantiate( charFilterDef.factory() ); + charFilters[index].init( getMapOfParameters( charFilterDef.params() ) ); + if ( charFilters[index] instanceof ResourceLoaderAware ) { + ((ResourceLoaderAware)charFilters[index]).inform( resourceLoader ); + } + } + return new TokenizerChain( charFilters, tokenFactory, filters ); } private static Object instantiate(Class clazz) { Modified: search/trunk/hibernate-search/src/test/java/org/hibernate/search/test/analyzer/solr/SolrAnalyzerTest.java =================================================================== --- search/trunk/hibernate-search/src/test/java/org/hibernate/search/test/analyzer/solr/SolrAnalyzerTest.java 2010-04-05 14:08:35 UTC (rev 19170) +++ search/trunk/hibernate-search/src/test/java/org/hibernate/search/test/analyzer/solr/SolrAnalyzerTest.java 2010-04-05 15:21:34 UTC (rev 19171) @@ -174,6 +174,12 @@ tokens = AnalyzerUtils.tokensFromAnalysis( analyzer, "name", text ); AnalyzerUtils.assertTokensEqual( tokens, new String[] { "foo", "bar" } ); + // CharStreamFactories test + analyzer = fts.getSearchFactory().getAnalyzer( "mapping_char_analyzer" ); + text = "CORAÇÃO DE MELÃO"; + tokens = AnalyzerUtils.tokensFromAnalysis( analyzer, "name", text ); + AnalyzerUtils.assertTokensEqual( tokens, new String[] { "CORACAO", "DE", "MELAO" } ); + fts.close(); } Modified: search/trunk/hibernate-search/src/test/java/org/hibernate/search/test/analyzer/solr/Team.java =================================================================== --- search/trunk/hibernate-search/src/test/java/org/hibernate/search/test/analyzer/solr/Team.java 2010-04-05 14:08:35 UTC (rev 19170) +++ search/trunk/hibernate-search/src/test/java/org/hibernate/search/test/analyzer/solr/Team.java 2010-04-05 15:21:34 UTC (rev 19171) @@ -28,12 +28,12 @@ import javax.persistence.GeneratedValue; import javax.persistence.Id; -import org.apache.solr.analysis.HTMLStripStandardTokenizerFactory; -import org.apache.solr.analysis.HTMLStripWhitespaceTokenizerFactory; +import org.apache.solr.analysis.HTMLStripCharFilterFactory; import org.apache.solr.analysis.ISOLatin1AccentFilterFactory; import org.apache.solr.analysis.LengthFilterFactory; import org.apache.solr.analysis.LowerCaseFilterFactory; import org.apache.solr.analysis.LowerCaseTokenizerFactory; +import org.apache.solr.analysis.MappingCharFilterFactory; import org.apache.solr.analysis.PorterStemFilterFactory; import org.apache.solr.analysis.ShingleFilterFactory; import org.apache.solr.analysis.SnowballPorterFilterFactory; @@ -49,6 +49,7 @@ import org.hibernate.search.annotations.Analyzer; import org.hibernate.search.annotations.AnalyzerDef; import org.hibernate.search.annotations.AnalyzerDefs; +import org.hibernate.search.annotations.CharFilterDef; import org.hibernate.search.annotations.DocumentId; import org.hibernate.search.annotations.Field; import org.hibernate.search.annotations.Indexed; @@ -87,17 +88,19 @@ filters = { @TokenFilterDef(factory = StandardFilterFactory.class) }), - @AnalyzerDef(name = "html_standard_analyzer", - tokenizer = @TokenizerDef(factory = HTMLStripStandardTokenizerFactory.class), + charFilters = { + @CharFilterDef(factory = HTMLStripCharFilterFactory.class) + }, + tokenizer = @TokenizerDef(factory = StandardTokenizerFactory.class), filters = { @TokenFilterDef(factory = StandardFilterFactory.class) }), @AnalyzerDef(name = "html_whitespace_analyzer", - tokenizer = @TokenizerDef(factory = HTMLStripWhitespaceTokenizerFactory.class), - filters = { - @TokenFilterDef(factory = StandardFilterFactory.class) + tokenizer = @TokenizerDef(factory = StandardTokenizerFactory.class), + charFilters = { + @CharFilterDef(factory = HTMLStripCharFilterFactory.class) }), @AnalyzerDef(name = "trim_analyzer", @@ -123,7 +126,10 @@ }), @AnalyzerDef(name = "word_analyzer", - tokenizer = @TokenizerDef(factory = HTMLStripStandardTokenizerFactory.class), + charFilters = { + @CharFilterDef(factory = HTMLStripCharFilterFactory.class) + }, + tokenizer = @TokenizerDef(factory = StandardTokenizerFactory.class), filters = { @TokenFilterDef(factory = WordDelimiterFilterFactory.class, params = { @Parameter(name = "splitOnCaseChange", value = "1") @@ -131,7 +137,10 @@ }), @AnalyzerDef(name = "synonym_analyzer", - tokenizer = @TokenizerDef(factory = HTMLStripStandardTokenizerFactory.class), + charFilters = { + @CharFilterDef(factory = HTMLStripCharFilterFactory.class) + }, + tokenizer = @TokenizerDef(factory = StandardTokenizerFactory.class), filters = { @TokenFilterDef(factory = SynonymFilterFactory.class, params = { @Parameter(name = "synonyms", @@ -140,7 +149,10 @@ }), @AnalyzerDef(name = "shingle_analyzer", - tokenizer = @TokenizerDef(factory = HTMLStripStandardTokenizerFactory.class), + charFilters = { + @CharFilterDef(factory = HTMLStripCharFilterFactory.class) + }, + tokenizer = @TokenizerDef(factory = StandardTokenizerFactory.class), filters = { @TokenFilterDef(factory = ShingleFilterFactory.class) }), @@ -152,7 +164,23 @@ @Parameter(name = "encoder", value = "Metaphone"), @Parameter(name = "inject", value = "false") }) - }) + }), + + @AnalyzerDef(name = "html_char_analyzer", + charFilters = { + @CharFilterDef(factory = HTMLStripCharFilterFactory.class) + }, + tokenizer = @TokenizerDef(factory = StandardTokenizerFactory.class) + ), + + @AnalyzerDef(name = "mapping_char_analyzer", + charFilters = { + @CharFilterDef(factory = MappingCharFilterFactory.class, params = { + @Parameter(name = "mapping", value = "org/hibernate/search/test/analyzer/solr/mapping-chars.properties") + }) + }, + tokenizer = @TokenizerDef(factory = StandardTokenizerFactory.class) + ) }) public class Team { @Id Added: search/trunk/hibernate-search/src/test/resources/org/hibernate/search/test/analyzer/solr/mapping-chars.properties =================================================================== --- search/trunk/hibernate-search/src/test/resources/org/hibernate/search/test/analyzer/solr/mapping-chars.properties (rev 0) +++ search/trunk/hibernate-search/src/test/resources/org/hibernate/search/test/analyzer/solr/mapping-chars.properties 2010-04-05 15:21:34 UTC (rev 19171) @@ -0,0 +1,80 @@ +# À => A +"\u00C0" => "A" + +# Á => A +"\u00C1" => "A" + +#  => A +"\u00C2" => "A" + +# à => A +"\u00C3" => "A" + +# Ä => A +"\u00C4" => "A" + +# Å => A +"\u00C5" => "A" + +# Æ => AE +"\u00C6" => "AE" + +# Ç => C +"\u00C7" => "C" + +# È => E +"\u00C8" => "E" + +# É => E +"\u00C9" => "E" + +# Ê => E +"\u00CA" => "E" + +# Ë => E +"\u00CB" => "E" + +# Ì => I +"\u00CC" => "I" + +# Í => I +"\u00CD" => "I" + +# Î => I +"\u00CE" => "I" + +# Ï => I +"\u00CF" => "I" + +# IJ => IJ +"\u0132" => "IJ" + +# Ð => D +"\u00D0" => "D" + +# Ñ => N +"\u00D1" => "N" + +# Ò => O +"\u00D2" => "O" + +# Ó => O +"\u00D3" => "O" + +# Ô => O +"\u00D4" => "O" + +# Õ => O +"\u00D5" => "O" + +# Ö => O +"\u00D6" => "O" + +# Ø => O +"\u00D8" => "O" + +# Œ => OE +"\u0152" => "OE" + +# Þ +"\u00DE" => "TH" \ No newline at end of file