[hibernate-dev] [SEARCH] Translating analyzer definitions from HSearch to Elasticsearch

[Sanne Grinovero]

Adding a "name" attribute to the @TokenizerDef annotation seems like a
good idea.

Make it an optional attribute of course, we can throw an exception if
it's missing and ES is being used, while maintaining compatibility
with existing apps using Lucene.
Perhaps you could be slightly forgiving in certain situations - I
guess you could use the fully qualified classname for example when
it's used only once - but your choice to see if that little benefit is
a worthy trade-off to implement.

Rather than documenting that this is useless for Lucene, we might even
take advantage of that (eventually) for some diagnostics messages /
tooling / debugging?
Not suggesting you do that now, just justifying that the "name"
attribute isn't entirely out of scope even for the Lucene embedded
case.

+1 to defer separating the filter chains into named, reusable
components: that can wait.

Thanks,
Sanne


On 13 December 2016 at 08:26, Yoann Rodiere <yoann at hibernate.org> wrote:
> Hello everyone,
>
> I'm currently working on HSEARCH-2219, "Define analyzers via the REST API",
> whose purpose is to automatically translate @AnalyzerDefs in Hibernate
> Search to settings in Elasticsearch, removing the need for users to
> configure analyzers separately in their Elasticsearch instance.
>
> The thing is, the structure of our configuration in Hibernate Search is
> different from the one in Elasticsearch. In particular, we can't name
> instances of token filters, char filters, etc, while in Elasticsearch one
> *has* to name them in order to provide parameters.
>
> See for instance:
>
> @AnalyzerDef(
>   name = "myAnalyzer",
>   tokenizer = @TokenizerDef(
>     factory = StandardTokenizerFactory.class,
>     parameters = @Parameters(@Parameter(name = "maxTokenLength", value =
> "900"))
>   )
> )
>
> compared to the Elasticsearch way:
>
> index :
>     analysis :
>         analyzer :
>             myAnalyzer :
>                 type : custom
>                 tokenizer : myTokenizer1
>         tokenizer :
>             myTokenizer1 :
>                 type : standard
>                 max_token_length : 900
>
> The analyzer name is there on both sides, @TokenizerDef.factory would give
> me the tokenizer type, and parameters are pretty obvious too. But
> "myTokenizer1", the tokenizer name, has absolutely no equivalent in
> Hibernate Search.
>
> I could try to generate names automatically, but those would need to be
> more or less stable across multiple executions in order for schema
> validation to work properly. And there's nothing we could really use as an
> identifier in our annotations, at least not reliably.
>
> To fill the gap, I'd like to add a "name" attribute to the TokenizerDef,
> CharFilterDef and TokenFilterDef annotations. This attribute would be
> optional and the documentation would mention that it's useless for embedded
> Lucene.
>
> Another solution would be to have a "magic" @Parameter, named after a
> constant (ElasticsearchParameters.TOKENIZER_NAME for instance), and detect
> that parameter automatically, but it feels wrong... mainly because
> @AnalyzerDef already has its own "name" attribute, so why wouldn't
> @TokenizerDef?
>
> And finally, we could bring our annotations closer to the Elasticsearch
> way, by providing a way to define tokenizers/char filters/token filters and
> a separate way to reference those definitions, but I don't think that's 5.6
> material, since we'd likely have to break things or lose consistency.
>
> WDYT?
>
> Yoann Rodière <yoann at hibernate.org>
> Hibernate NoORM Team
> _______________________________________________
> hibernate-dev mailing list
> hibernate-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hibernate-dev