I did some profiling and the issue is not Spring. It's just the overhead of HV validating a bean which cascades to a lot of beans.
I already mitigated the issue a bit. There's not much we can do about it.