[hibernate-commits] Hibernate SVN: r19171 - in search/trunk/hibernate-search/src: main/java/org/hibernate/search/annotations and 3 other directories.
hibernate-commits at lists.jboss.org
hibernate-commits at lists.jboss.org
Mon Apr 5 11:21:34 EDT 2010
Author: sannegrinovero
Date: 2010-04-05 11:21:34 -0400 (Mon, 05 Apr 2010)
New Revision: 19171
Added:
search/trunk/hibernate-search/src/main/java/org/hibernate/search/annotations/CharFilterDef.java
search/trunk/hibernate-search/src/test/resources/org/hibernate/search/test/analyzer/solr/mapping-chars.properties
Modified:
search/trunk/hibernate-search/src/main/docbook/en-US/modules/mapping.xml
search/trunk/hibernate-search/src/main/java/org/hibernate/search/annotations/AnalyzerDef.java
search/trunk/hibernate-search/src/main/java/org/hibernate/search/impl/SolrAnalyzerBuilder.java
search/trunk/hibernate-search/src/test/java/org/hibernate/search/test/analyzer/solr/SolrAnalyzerTest.java
search/trunk/hibernate-search/src/test/java/org/hibernate/search/test/analyzer/solr/Team.java
Log:
HSEARCH-477 Support for the new Solr's character filters (Gustavo Fernandes)
Modified: search/trunk/hibernate-search/src/main/docbook/en-US/modules/mapping.xml
===================================================================
--- search/trunk/hibernate-search/src/main/docbook/en-US/modules/mapping.xml 2010-04-05 14:08:35 UTC (rev 19170)
+++ search/trunk/hibernate-search/src/main/docbook/en-US/modules/mapping.xml 2010-04-05 15:21:34 UTC (rev 19171)
@@ -693,6 +693,12 @@
</listitem>
<listitem>
+ <para>a list of char filters: each char filter is responsible to
+ pre-process input characters before the tokenization. Char filters can add,
+ change or remove characters; one common usage is for characters normalization</para>
+ </listitem>
+
+ <listitem>
<para>a tokenizer: responsible for tokenizing the input stream
into individual words</para>
</listitem>
@@ -704,16 +710,17 @@
</listitem>
</itemizedlist>
- <para>This separation of tasks - a tokenizer followed by a list of
+ <para>This separation of tasks - a list of char filters, and a tokenizer followed by a list of
filters - allows for easy reuse of each individual component and let
you build your customized analyzer in a very flexible way (just like
- Lego). Generally speaking the <classname>Tokenizer</classname> starts
- the analysis process by turning the character input into tokens which
+ Lego). Generally speaking the <classname>char filters</classname> do some
+ pre-processing in the character input, then the <classname>Tokenizer</classname> starts
+ the tokenizing process by turning the character input into tokens which
are then further processed by the <classname>TokenFilter</classname>s.
Hibernate Search supports this infrastructure by utilizing the Solr
analyzer framework. Make sure to add<filename> solr-core.jar and
- </filename><filename>solr-common.jar</filename> to your classpath to
- use analyzer definitions. In case you also want to utilizing a
+ </filename><filename>solr-solrj.jar</filename> to your classpath to
+ use analyzer definitions. In case you also want to use the
snowball stemmer also include the
<filename>lucene-snowball.jar.</filename> Other Solr analyzers might
depend on more libraries. For example, the
@@ -727,6 +734,11 @@
framework</title>
<programlisting>@AnalyzerDef(name="customanalyzer",
+ charFilters = {
+ @CharFilterDef(factory = MappingCharFilterFactory.class, params = {
+ @Parameter(name = "mapping", value = "org/hibernate/search/test/analyzer/solr/mapping-chars.properties")
+ })
+ },
tokenizer = @TokenizerDef(factory = StandardTokenizerFactory.class),
filters = {
@TokenFilterDef(factory = ISOLatin1AccentFilterFactory.class),
@@ -741,9 +753,12 @@
}</programlisting>
</example>
- <para>A tokenizer is defined by its factory which is responsible for
- building the tokenizer and using the optional list of parameters. This
- example use the standard tokenizer. A filter is defined by its factory
+ <para>A char filter is defined by its factory which is responsible for
+ building the char filter and using the optional list of parameters.
+ In our example, a mapping char filter is used, and will replace
+ characters in the input based on the rules specified in the mapping
+ file. A tokenizer is also defined by its factory.
+ This example use the standard tokenizer. A filter is defined by its factory
which is responsible for creating the filter instance using the
optional parameters. In our example, the StopFilter filter is built
reading the dedicated words property file and is expected to ignore
@@ -751,7 +766,7 @@
factory.</para>
<warning>
- <para>Filters are applied in the order they are defined in the
+ <para>Filters and char filters are applied in the order they are defined in the
<classname>@AnalyzerDef</classname> annotation. Make sure to think
twice about this order.</para>
</warning>
@@ -800,13 +815,53 @@
<section>
<title>Available analyzers</title>
- <para>Solr and Lucene come with a lot of useful default tokenizers and
- filters. You can find a complete list of tokenizer factories and
+ <para>Solr and Lucene come with a lot of useful default char filters, tokenizers and
+ filters. You can find a complete list of char filter factories, tokenizer factories and
filter factories at <ulink
url="http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters">http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters</ulink>.
Let check a few of them.</para>
<table>
+ <title>Some of the available char filters</title>
+ <tgroup cols="3">
+ <thead>
+ <row>
+ <entry align="center">Factory</entry>
+
+ <entry align="center">Description</entry>
+
+ <entry align="center">parameters</entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>MappingCharFilterFactory</entry>
+
+ <entry>Replaces one or more characters with one or more characters, based on mappings
+ specified in the resource file</entry>
+
+ <entry><para><literal>mapping</literal>: points to a resource file containing the mappings
+ using the format:
+ <literallayout>
+ "á" => "a"
+ "ñ" => "n"
+ "ø" => "o"
+ </literallayout>
+ </para></entry>
+ </row>
+ <row>
+ <entry>HTMLStripCharFilterFactory</entry>
+
+ <entry>Remove HTML standard tags, keeping the text</entry>
+
+ <entry>none</entry>
+ </row>
+ </tbody>
+ </tgroup>
+
+ </table>
+
+ <table>
<title>Some of the available tokenizers</title>
<tgroup cols="3">
@@ -833,7 +888,7 @@
<entry>HTMLStripStandardTokenizerFactory</entry>
<entry>Remove HTML tags, keep the text and pass it to a
- StandardTokenizer</entry>
+ StandardTokenizer. @Deprecated, use the HTMLStripCharFilterFactory instead</entry>
<entry>none</entry>
</row>
Modified: search/trunk/hibernate-search/src/main/java/org/hibernate/search/annotations/AnalyzerDef.java
===================================================================
--- search/trunk/hibernate-search/src/main/java/org/hibernate/search/annotations/AnalyzerDef.java 2010-04-05 14:08:35 UTC (rev 19170)
+++ search/trunk/hibernate-search/src/main/java/org/hibernate/search/annotations/AnalyzerDef.java 2010-04-05 15:21:34 UTC (rev 19171)
@@ -53,6 +53,11 @@
String name();
/**
+ * @return CharFilters used. The filters are applied in the defined order
+ */
+ CharFilterDef[] charFilters() default { };
+
+ /**
* @return Tokenizer used.
*/
TokenizerDef tokenizer();
Added: search/trunk/hibernate-search/src/main/java/org/hibernate/search/annotations/CharFilterDef.java
===================================================================
--- search/trunk/hibernate-search/src/main/java/org/hibernate/search/annotations/CharFilterDef.java (rev 0)
+++ search/trunk/hibernate-search/src/main/java/org/hibernate/search/annotations/CharFilterDef.java 2010-04-05 15:21:34 UTC (rev 19171)
@@ -0,0 +1,53 @@
+/* $Id$
+ *
+ * Hibernate, Relational Persistence for Idiomatic Java
+ *
+ * Copyright (c) 2009, Red Hat, Inc. and/or its affiliates or third-party contributors as
+ * indicated by the @author tags or express copyright attribution
+ * statements applied by the authors. All third-party contributions are
+ * distributed under license by Red Hat, Inc.
+ *
+ * This copyrighted material is made available to anyone wishing to use, modify,
+ * copy, or redistribute it subject to the terms and conditions of the GNU
+ * Lesser General Public License, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public License
+ * along with this distribution; if not, write to:
+ * Free Software Foundation, Inc.
+ * 51 Franklin Street, Fifth Floor
+ * Boston, MA 02110-1301 USA
+ */
+package org.hibernate.search.annotations;
+
+import org.apache.solr.analysis.CharFilterFactory;
+
+import java.lang.annotation.Documented;
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+
+/**
+ * Define a <code>CharFilterFactory</code> and its parameters
+ *
+ * @author Gustavo Fernandes
+ */
+ at Retention(RetentionPolicy.RUNTIME)
+ at Target({ ElementType.TYPE, ElementType.FIELD, ElementType.METHOD})
+ at Documented
+public @interface CharFilterDef {
+ /**
+ * @return the <code>TokenFilterFactory</code> class which shall be instantiated.
+ */
+ public abstract Class<? extends CharFilterFactory> factory();
+
+ /**
+ * @return Optional parameters passed to the <code>CharFilterFactory</code>.
+ */
+ public abstract Parameter[] params() default { };
+}
Property changes on: search/trunk/hibernate-search/src/main/java/org/hibernate/search/annotations/CharFilterDef.java
___________________________________________________________________
Name: svn:keywords
+ Id
Modified: search/trunk/hibernate-search/src/main/java/org/hibernate/search/impl/SolrAnalyzerBuilder.java
===================================================================
--- search/trunk/hibernate-search/src/main/java/org/hibernate/search/impl/SolrAnalyzerBuilder.java 2010-04-05 14:08:35 UTC (rev 19170)
+++ search/trunk/hibernate-search/src/main/java/org/hibernate/search/impl/SolrAnalyzerBuilder.java 2010-04-05 15:21:34 UTC (rev 19171)
@@ -29,6 +29,7 @@
import java.util.Collections;
import org.apache.lucene.analysis.Analyzer;
+import org.apache.solr.analysis.CharFilterFactory;
import org.apache.solr.analysis.TokenizerChain;
import org.apache.solr.analysis.TokenFilterFactory;
import org.apache.solr.analysis.TokenizerFactory;
@@ -36,6 +37,7 @@
import org.apache.solr.common.ResourceLoader;
import org.hibernate.search.annotations.AnalyzerDef;
+import org.hibernate.search.annotations.CharFilterDef;
import org.hibernate.search.annotations.TokenizerDef;
import org.hibernate.search.annotations.TokenFilterDef;
import org.hibernate.search.annotations.Parameter;
@@ -65,7 +67,9 @@
tokenFactory.init( getMapOfParameters( token.params() ) );
final int length = analyzerDef.filters().length;
+ final int charLength = analyzerDef.charFilters().length;
TokenFilterFactory[] filters = new TokenFilterFactory[length];
+ CharFilterFactory[] charFilters = new CharFilterFactory[charLength];
ResourceLoader resourceLoader = new HibernateSearchResourceLoader();
for ( int index = 0 ; index < length ; index++ ) {
TokenFilterDef filterDef = analyzerDef.filters()[index];
@@ -75,7 +79,15 @@
((ResourceLoaderAware)filters[index]).inform( resourceLoader );
}
}
- return new TokenizerChain(tokenFactory, filters);
+ for ( int index = 0 ; index < charFilters.length ; index++ ) {
+ CharFilterDef charFilterDef = analyzerDef.charFilters()[index];
+ charFilters[index] = (CharFilterFactory) instantiate( charFilterDef.factory() );
+ charFilters[index].init( getMapOfParameters( charFilterDef.params() ) );
+ if ( charFilters[index] instanceof ResourceLoaderAware ) {
+ ((ResourceLoaderAware)charFilters[index]).inform( resourceLoader );
+ }
+ }
+ return new TokenizerChain( charFilters, tokenFactory, filters );
}
private static Object instantiate(Class clazz) {
Modified: search/trunk/hibernate-search/src/test/java/org/hibernate/search/test/analyzer/solr/SolrAnalyzerTest.java
===================================================================
--- search/trunk/hibernate-search/src/test/java/org/hibernate/search/test/analyzer/solr/SolrAnalyzerTest.java 2010-04-05 14:08:35 UTC (rev 19170)
+++ search/trunk/hibernate-search/src/test/java/org/hibernate/search/test/analyzer/solr/SolrAnalyzerTest.java 2010-04-05 15:21:34 UTC (rev 19171)
@@ -174,6 +174,12 @@
tokens = AnalyzerUtils.tokensFromAnalysis( analyzer, "name", text );
AnalyzerUtils.assertTokensEqual( tokens, new String[] { "foo", "bar" } );
+ // CharStreamFactories test
+ analyzer = fts.getSearchFactory().getAnalyzer( "mapping_char_analyzer" );
+ text = "CORAÇÃO DE MELÃO";
+ tokens = AnalyzerUtils.tokensFromAnalysis( analyzer, "name", text );
+ AnalyzerUtils.assertTokensEqual( tokens, new String[] { "CORACAO", "DE", "MELAO" } );
+
fts.close();
}
Modified: search/trunk/hibernate-search/src/test/java/org/hibernate/search/test/analyzer/solr/Team.java
===================================================================
--- search/trunk/hibernate-search/src/test/java/org/hibernate/search/test/analyzer/solr/Team.java 2010-04-05 14:08:35 UTC (rev 19170)
+++ search/trunk/hibernate-search/src/test/java/org/hibernate/search/test/analyzer/solr/Team.java 2010-04-05 15:21:34 UTC (rev 19171)
@@ -28,12 +28,12 @@
import javax.persistence.GeneratedValue;
import javax.persistence.Id;
-import org.apache.solr.analysis.HTMLStripStandardTokenizerFactory;
-import org.apache.solr.analysis.HTMLStripWhitespaceTokenizerFactory;
+import org.apache.solr.analysis.HTMLStripCharFilterFactory;
import org.apache.solr.analysis.ISOLatin1AccentFilterFactory;
import org.apache.solr.analysis.LengthFilterFactory;
import org.apache.solr.analysis.LowerCaseFilterFactory;
import org.apache.solr.analysis.LowerCaseTokenizerFactory;
+import org.apache.solr.analysis.MappingCharFilterFactory;
import org.apache.solr.analysis.PorterStemFilterFactory;
import org.apache.solr.analysis.ShingleFilterFactory;
import org.apache.solr.analysis.SnowballPorterFilterFactory;
@@ -49,6 +49,7 @@
import org.hibernate.search.annotations.Analyzer;
import org.hibernate.search.annotations.AnalyzerDef;
import org.hibernate.search.annotations.AnalyzerDefs;
+import org.hibernate.search.annotations.CharFilterDef;
import org.hibernate.search.annotations.DocumentId;
import org.hibernate.search.annotations.Field;
import org.hibernate.search.annotations.Indexed;
@@ -87,17 +88,19 @@
filters = {
@TokenFilterDef(factory = StandardFilterFactory.class)
}),
-
@AnalyzerDef(name = "html_standard_analyzer",
- tokenizer = @TokenizerDef(factory = HTMLStripStandardTokenizerFactory.class),
+ charFilters = {
+ @CharFilterDef(factory = HTMLStripCharFilterFactory.class)
+ },
+ tokenizer = @TokenizerDef(factory = StandardTokenizerFactory.class),
filters = {
@TokenFilterDef(factory = StandardFilterFactory.class)
}),
@AnalyzerDef(name = "html_whitespace_analyzer",
- tokenizer = @TokenizerDef(factory = HTMLStripWhitespaceTokenizerFactory.class),
- filters = {
- @TokenFilterDef(factory = StandardFilterFactory.class)
+ tokenizer = @TokenizerDef(factory = StandardTokenizerFactory.class),
+ charFilters = {
+ @CharFilterDef(factory = HTMLStripCharFilterFactory.class)
}),
@AnalyzerDef(name = "trim_analyzer",
@@ -123,7 +126,10 @@
}),
@AnalyzerDef(name = "word_analyzer",
- tokenizer = @TokenizerDef(factory = HTMLStripStandardTokenizerFactory.class),
+ charFilters = {
+ @CharFilterDef(factory = HTMLStripCharFilterFactory.class)
+ },
+ tokenizer = @TokenizerDef(factory = StandardTokenizerFactory.class),
filters = {
@TokenFilterDef(factory = WordDelimiterFilterFactory.class, params = {
@Parameter(name = "splitOnCaseChange", value = "1")
@@ -131,7 +137,10 @@
}),
@AnalyzerDef(name = "synonym_analyzer",
- tokenizer = @TokenizerDef(factory = HTMLStripStandardTokenizerFactory.class),
+ charFilters = {
+ @CharFilterDef(factory = HTMLStripCharFilterFactory.class)
+ },
+ tokenizer = @TokenizerDef(factory = StandardTokenizerFactory.class),
filters = {
@TokenFilterDef(factory = SynonymFilterFactory.class, params = {
@Parameter(name = "synonyms",
@@ -140,7 +149,10 @@
}),
@AnalyzerDef(name = "shingle_analyzer",
- tokenizer = @TokenizerDef(factory = HTMLStripStandardTokenizerFactory.class),
+ charFilters = {
+ @CharFilterDef(factory = HTMLStripCharFilterFactory.class)
+ },
+ tokenizer = @TokenizerDef(factory = StandardTokenizerFactory.class),
filters = {
@TokenFilterDef(factory = ShingleFilterFactory.class)
}),
@@ -152,7 +164,23 @@
@Parameter(name = "encoder", value = "Metaphone"),
@Parameter(name = "inject", value = "false")
})
- })
+ }),
+
+ @AnalyzerDef(name = "html_char_analyzer",
+ charFilters = {
+ @CharFilterDef(factory = HTMLStripCharFilterFactory.class)
+ },
+ tokenizer = @TokenizerDef(factory = StandardTokenizerFactory.class)
+ ),
+
+ @AnalyzerDef(name = "mapping_char_analyzer",
+ charFilters = {
+ @CharFilterDef(factory = MappingCharFilterFactory.class, params = {
+ @Parameter(name = "mapping", value = "org/hibernate/search/test/analyzer/solr/mapping-chars.properties")
+ })
+ },
+ tokenizer = @TokenizerDef(factory = StandardTokenizerFactory.class)
+ )
})
public class Team {
@Id
Added: search/trunk/hibernate-search/src/test/resources/org/hibernate/search/test/analyzer/solr/mapping-chars.properties
===================================================================
--- search/trunk/hibernate-search/src/test/resources/org/hibernate/search/test/analyzer/solr/mapping-chars.properties (rev 0)
+++ search/trunk/hibernate-search/src/test/resources/org/hibernate/search/test/analyzer/solr/mapping-chars.properties 2010-04-05 15:21:34 UTC (rev 19171)
@@ -0,0 +1,80 @@
+# À => A
+"\u00C0" => "A"
+
+# Á => A
+"\u00C1" => "A"
+
+# Â => A
+"\u00C2" => "A"
+
+# Ã => A
+"\u00C3" => "A"
+
+# Ä => A
+"\u00C4" => "A"
+
+# Å => A
+"\u00C5" => "A"
+
+# Æ => AE
+"\u00C6" => "AE"
+
+# Ç => C
+"\u00C7" => "C"
+
+# È => E
+"\u00C8" => "E"
+
+# É => E
+"\u00C9" => "E"
+
+# Ê => E
+"\u00CA" => "E"
+
+# Ë => E
+"\u00CB" => "E"
+
+# Ì => I
+"\u00CC" => "I"
+
+# Í => I
+"\u00CD" => "I"
+
+# Î => I
+"\u00CE" => "I"
+
+# Ï => I
+"\u00CF" => "I"
+
+# IJ => IJ
+"\u0132" => "IJ"
+
+# Ð => D
+"\u00D0" => "D"
+
+# Ñ => N
+"\u00D1" => "N"
+
+# Ò => O
+"\u00D2" => "O"
+
+# Ó => O
+"\u00D3" => "O"
+
+# Ô => O
+"\u00D4" => "O"
+
+# Õ => O
+"\u00D5" => "O"
+
+# Ö => O
+"\u00D6" => "O"
+
+# Ø => O
+"\u00D8" => "O"
+
+# Π=> OE
+"\u0152" => "OE"
+
+# Þ
+"\u00DE" => "TH"
\ No newline at end of file
More information about the hibernate-commits
mailing list