+1
As agreed on IRC let's add a power-user oriented extension to the FieldBridge interface which allows declaring which fields are going to be safe to sort on without us needing to run the un-inverting process.
We'll mark this new interface as both experimental and Deprecated already though, as in future versions we'll change the actual API to a better solution.