Results with 5.1.3.Final (version used by the OP)

Time to Validate (Built-in): 4850
Time (no validation): 9
Time to Validate (Field-level): 5081
Time to Validate (Class-level): 1248

Results with 5.4.2.Final (last 5.x version)

Time to Validate (Built-in): 6819
Time (no validation): 9
Time to Validate (Field-level): 7898
Time to Validate (Class-level): 1641

Results with my current 6.0.4-SNAPSHOT

Time to Validate (Built-in): 2306
Time (no validation): 10
Time to Validate (Field-level): 2274
Time to Validate (Class-level): 861

So the situation was quite worse in 5.4 than it was in 5.1. It's now much better in 6. We still have some overhead but we also do a lot more operations when dealing with fields so it's quite normal.

We might be able to make further improvements in the future but for now I think we can mark this issue as resolved.