Re: [hibernate-dev] providing AnalyzerUtils in Hibernate Search

2011/5/12 Emmanuel Bernard <emmanuel(a)hibernate.org&gt;: yes actually some people on the forum where asking about how to enable logging on tokenization, but currently we pass only the document and the analyzer instances to the IndexWriter, and Lucene has no logging (unless we want to use it's infostream redirecting it somehow to a proper logger). We could of course "fake it" by doing some work in the area, basically re-tokenizing the input an additional time if log.isDebugEnabled; it would be easier if we where going to use the "pre-analyze input" approach I mentioned, even tough my idea was mainly to offload work from the master and reduce I/O to the backend. Agreed, we could do better; at this point I'd only want to distribute the tool to ease debugging of @AnalyzerDef, or just helping understanding how it works (you could try figure out the output of AnalyzerDef posted below [1] - don't laugh) But there's no strong need to distribute this, as the hibernate-search-testing jar is available as well and contains it; we could just add some directions to the documentation for now and think of better logging with pre-analysis; I guess it could be a good practice to have tests in any project doing some assertions on expected analyzer output. Sanne [1] @AnalyzerDef(name = "entityAnalyser", tokenizer = @TokenizerDef(factory = StandardTokenizerFactory.class), filters = { @TokenFilterDef(factory = StandardFilterFactory.class), @TokenFilterDef(factory = LowerCaseFilterFactory.class), @TokenFilterDef(factory = StopFilterFactory.class), @TokenFilterDef(factory = SnowballPorterFilterFactory.class, params = { @Parameter(name = "language", value = "French")}), @TokenFilterDef(factory = PhoneticFilterFactory.class ,params = { @Parameter(name="encoder", value="DoubleMetaphone")}), @TokenFilterDef(factory = ASCIIFoldingFilterFactory.class), @TokenFilterDef(factory = NGramFilterFactory.class, params = { @Parameter(name = "minGramSize", value = "3"), @Parameter(name = "maxGramSize", value = "3") }) } ,charFilters = { @CharFilterDef(factory = HTMLStripCharFilterFactory.class) } )