What is the section of the spec defining how exactly this shall work?

I could only find the javadoc of javax.persistence.spi.PersistenceProvider#createEntityManagerFactory, which states:

@return EntityManagerFactory for the persistence unit, or null if the provider is not the right provider.

If that's all, it doesn't seem to specify what to do in case of exceptions, e.g. illegal mappings being detected. One could just verify if "the right provider" is right by checking the provider name.