| Note: as illustrated by
HSEARCH-2639 Open , we may want something more subtle to define "field type"; let's call it "field encoding" to avoid confusions. There are quite a few issues to address:
- The available encodings may differ from one indexing service to another. For instance the "date" encoding doesn't mean anything for Lucene, which only knows about numeric encoding or string encoding.
- Even for common encodings (integer), the details may differ from one indexing service to another. For instance we don't currently have an equivalent to the "ignore_malformed" mapping parameter in Lucene.
We could try to implement everything that is missing, but that's probably the path to failure: there's no way we can scale this method to 3, 4, 5 indexing services. We could provide only the common subset of encodings, but this approach also seems doomed: we will end up providing useless integrations with only very basic encodings (no support for date on ES... ?) I think the best approach would be to:
- Make the input type the basic characteristic of encodings: Encoding<java.util.Date> for instance. For instance default encodings would only be about their input type: LongEncoding, DateEncoding, InstantEncoding, ...
- Somehow associate one implementation per indexing service to each encoding: DateEncoding would be implemented differently on Lucene than on Elasticsearch, for instance.
- Use these encodings in our default (simple) field bridges (see ticket description)
- Allow to use these encodings in complex field bridges
- Allow to tune these encodings in complex field bridges in an indexing-service-agnostic way (minimal tuning, with dedicated enum where relevant, with validation). For instance with a builder for each encoding: DateEncoding.builder().resolution( ... ).build().
- Allow to tune these encodings in complex field bridges in an indexing-service-specific way (extensive tuning, potentially string=>string map, little to no validation). For instance with something like .withImplementation( ElasticsearchDateEncoding.builder().format( "some format description", new DateTimeFormatter() .... ).build()).
- Allow users to define their own encodings, both the "declaration" (DateEncoding) and the implementations (ElasticsearchDateEncoding).
We may think that this concept of encoding is very close to what we currently have with "simple" (non-composite) field bridges, but actually there is one key difference: we associate each field with one encoding, and this encoding will be implemented differently depending on the underlying indexing service. Whereas with field bridges, we associate each field directly with an implementation (only default field bridges can be implemented differently based on the underlying indexing service). Also (less important), with Lucene we used to add multipe fields to the document for a single value (the indexed field + the docvalues field + ...), while we obviously don't need this for Elasticsearch. Note that we don't necessarily need to do this anymore, because Lucene support custom field types, so we could, in theory, craft a specific field type for every single field in the application, and enable/disable docvalues as necessary. See org.apache.lucene.document.FieldType. |