[hibernate-dev] Hibernate Validator 6 benchmarks results

[Guillaume Smet]

Hi,

I ran some HV 6 benchmarks results this morning, following the profiling
campaign I did last week and prior to our 6.0.0.Alpha2 release, and got
some interesting results I thought worthwhile to share.

Results are in ops/ms and are from our own JMH benchmarks. The higher, the
better.

== 5.4.1.Final

Result
"org.hibernate.validator.performance.simple.SimpleValidation.testSimpleBeanValidation":
  556.190 ±(99.9%) 3.985 ops/ms [Average]
  (min, avg, max) = (537.875, 556.190, 570.387), stdev = 8.049
  CI (99.9%): [552.205, 560.175] (assumes normal distribution)

Result
"org.hibernate.validator.performance.cascaded.CascadedValidation.testCascadedValidation":
  288.429 ±(99.9%) 3.305 ops/ms [Average]
  (min, avg, max) = (271.979, 288.429, 308.106), stdev = 6.677
  CI (99.9%): [285.124, 291.734] (assumes normal distribution)

== Current master

Result
"org.hibernate.validator.performance.simple.SimpleValidation.testSimpleBeanValidation":
  869.546 ±(99.9%) 14.734 ops/ms [Average]
  (min, avg, max) = (760.007, 869.546, 909.206), stdev = 29.763
  CI (99.9%): [854.813, 884.280] (assumes normal distribution)

Result
"org.hibernate.validator.performance.cascaded.CascadedValidation.testCascadedValidation":
  343.699 ±(99.9%) 2.077 ops/ms [Average]
  (min, avg, max) = (331.333, 343.699, 352.626), stdev = 4.196
  CI (99.9%): [341.622, 345.776] (assumes normal distribution)

== Analysis

=== SimpleValidation

So, interestingly enough, current master is 56% faster than 5.4.1.Final on
the simple validation case.

This is all due to the fact that we now cache the ValueExtractor in the
metadata whereas the value unwrapper was determined at validation time
before. All the CPU goes here:
https://github.com/hibernate/hibernate-validator/blob/5.4/engine/src/main/java/org/hibernate/validator/internal/engine/ValidationContext.java#L390
and especially on the typeResolver.resolve calls.

This is mostly a structural improvement due to the new design of
ValueExtractors (kudos Gunnar!).

=== CascadedValidation

So, last week, HV 6 was 3 times slower (yes **3 times** slower) than HV 5.4
on the CascadedValidation benchmark.

After my recent improvements, it's now 20% faster so it's quite conclusive
too. I obtained these results mostly by limiting the number of reflection
calls (again) and better caching of some metadata.

== Conclusion

So 2 conclusions mostly:
- we are on the right track
- we really need to be careful when doing reflection operations, they might
look harmless at the time but they consume a lot of cycles

Have a nice day!

-- 
Guillaume