We use simple (single-valued) doc values for location, and thus we cannot store more than one location per document per field.

It's not that bad with Lucene, because our queries would probably mix latitude and longitude from different locations anyway, but I'm pretty sure Elasticsearch would manage to handle queries on such multi-valued fields... And since we use Lucene doc values fields as a pivot format before generating the Elasticsearch document, the limitation on the Lucene implementation also affects the Elasticsearch implementation.