[hibernate-dev] [HV] Hibernate Validator 6 benchmark updates

[Guillaume Smet]

Hi,

So, as we are leaning towards final, I made another round of benchmarking
on HV 6.

They are run on the same machine as the previous "Hibernate Validator 6
benchmarks results" I posted a few months ago [1] so they can be compared
to these numbers (you'll notice the 5.4.1 numbers are about the same
between the 2 emails).

Note that these benchmarks only exercises the validation engine but does
not report any violation.

I think I'll add some more benchmarks to ensure violation reporting also
moves in the right direction.

It might also be a good occasion to revisit this benchmark:
http://carinae.net/2010/06/benchmarking-hibernate-validator-and-apache-beanvalidation-the-two-jsr-303-implementations/
. With a large value of "revisit" as it's really basic but the scenarios
are interesting. Note that I'm a bit pessimistic on the parsing part: we
now do a lot more work than before to deal with container elements so the
startup cost will probably be significantly higher.

== Apache BVal 1.1.2

Result
"org.hibernate.validator.performance.simple.SimpleValidation.testSimpleBeanValidation":
  357.500 ±(99.9%) 5.327 ops/ms [Average]
  (min, avg, max) = (337.100, 357.500, 381.472), stdev = 10.760
  CI (99.9%): [352.173, 362.827] (assumes normal distribution)

Result
"org.hibernate.validator.performance.cascaded.CascadedValidation.testCascadedValidation":
  379.605 ±(99.9%) 5.817 ops/ms [Average]
  (min, avg, max) = (360.654, 379.605, 411.361), stdev = 11.750
  CI (99.9%): [373.789, 385.422] (assumes normal distribution)

(not really clear how BVal happens to be faster in the cascaded validation
case)

== HV 5.4.1.Final

Result
"org.hibernate.validator.performance.simple.SimpleValidation.testSimpleBeanValidation":
  558.196 ±(99.9%) 3.643 ops/ms [Average]
  (min, avg, max) = (542.396, 558.196, 575.360), stdev = 7.360
  CI (99.9%): [554.552, 561.839] (assumes normal distribution)

Result
"org.hibernate.validator.performance.cascaded.CascadedValidation.testCascadedValidation":
  285.788 ±(99.9%) 1.970 ops/ms [Average]
  (min, avg, max) = (278.611, 285.788, 298.530), stdev = 3.980
  CI (99.9%): [283.817, 287.758] (assumes normal distribution)

== Master from March after the first round of improvements (numbers taken
from the previous email)

Result
"org.hibernate.validator.performance.simple.SimpleValidation.testSimpleBeanValidation":
  869.546 ±(99.9%) 14.734 ops/ms [Average]
  (min, avg, max) = (760.007, 869.546, 909.206), stdev = 29.763
  CI (99.9%): [854.813, 884.280] (assumes normal distribution)

Result
"org.hibernate.validator.performance.cascaded.CascadedValidation.testCascadedValidation":
  343.699 ±(99.9%) 2.077 ops/ms [Average]
  (min, avg, max) = (331.333, 343.699, 352.626), stdev = 4.196
  CI (99.9%): [341.622, 345.776] (assumes normal distribution)

== HV 6 - Current master with
https://github.com/hibernate/hibernate-validator/pull/814 applied

Result
"org.hibernate.validator.performance.simple.SimpleValidation.testSimpleBeanValidation":
  924.121 ±(99.9%) 3.686 ops/ms [Average]
  (min, avg, max) = (905.423, 924.121, 941.295), stdev = 7.446
  CI (99.9%): [920.435, 927.807] (assumes normal distribution)

Result
"org.hibernate.validator.performance.cascaded.CascadedValidation.testCascadedValidation":
  430.092 ±(99.9%) 3.661 ops/ms [Average]
  (min, avg, max) = (416.439, 430.092, 447.607), stdev = 7.396
  CI (99.9%): [426.431, 433.754] (assumes normal distribution)

== Conclusion

The good news is that the results are even better than the ones from March,
after some further tweaking.

We are significantly faster than BVal in these scenarios and also
significantly faster than 5.4.1.Final.

Now, we need to get this PR in :).

[1] http://lists.jboss.org/pipermail/hibernate-dev/2017-March/016057.html