I do keep that in mind. Most of those maps/sets (if not all) are used during bootstrap. We rarely (if ever) need to iterate on maps/sets created at runtime.
The only place where we instantiate LinkedHashMaps at runtime is org.hibernate.search.v6poc.entity.pojo.mapping.impl.ChangesetPojoWorkerImpl#indexedTypeDelegates, and it's already the case in its Search 5 counterpart (org.hibernate.search.engine.impl.WorkPlan#byClass).

Moreover, it is not only about my personal comfort. As user, when presented with a randomly occurring bug (like I have been in the past, before I joined this team), I wouldn't feel comfortable.

Last but not least... of course, if later performance testing reveals those maps/sets add significantly to the memory footprint of Search, we will revert these changes. But I very much doubt it will, since we are talking about a few maps/sets per indexed class, instantiated during bootstrap.