Adding a link to HSEARCH-2393 In Progress , where I had to fix a test (org.hibernate.search.test.query.sorting.SortOnFieldsFromCustomBridgeTest.testNumericCustomFieldLevelBridge()) and have been confronted with this issue again.

It seems that having a @NumericField annotation on the field, implementing MetadataProvidingFieldBridge in the custom field bridge, and re-defining the field type in MetadataProvidingFieldBridge.configure does the trick. One must not add a @SortableField annotation, though, or the world will fall apart with the Lucene backend (two doc values for the same field).
So to sum up, what we have to solve is the fact that adding @NumericField on the field with a custom field bridge is mandatory: it's utterly counter-intuitive, especially given that one must not add @SortableField.
One lead to achieve this: inspect the custom field bridge metadata before inspecting the standard field metadata in org.hibernate.search.query.engine.impl.AbstractHSQuery.validateCommonSortField(ExtendedSearchIntegrator, Iterable<Class<?>>, SortField). There might be other issues, though.