I believe the problem is in the CollectionReferenceInitializerImpl.finishUpRow(ResultSet, ResultSetProcessingContextImpl).
In the case of non primary key reference column BinderHelper creates Component as the property of the parent entity.
Then CollectionReferenceInitializerImpl uses that property but pass the primary key as its value.

I will try to send a pull request soon. Not sure yet how to correct it properly though.