It seems to be happening because without enableDirtyTracking the dirty checking mechanism used is TypeHelper#findDirty()[1] and it calls [2] which clearly states that for a lazy property to be dirty all lazy properties must be initialized.

[1] - https://github.com/hibernate/hibernate-orm/blob/master/hibernate-core/src/main/java/org/hibernate/type/TypeHelper.java#L298
[2] - https://github.com/hibernate/hibernate-orm/blob/master/hibernate-core/src/main/java/org/hibernate/tuple/AbstractNonIdentifierAttribute.java#L89