11:47 a.m.
Hi,
(Not a big fan of HTML emails but I gave up with formatting the table,
sorry)
So I took the time to revisit the old BVal benchmark [1] and posted an
updated version of it here: https://github.com/gsmet/
beanvalidation-benchmark.
I pushed a few more commits to optimize the parsing/metadata building phase
here: https://github.com/hibernate/hibernate-validator/pull/814.
Note that the benchmark is much more representative that our very simple
JMH tests: it generates a lot of bean definitions with constraints on
properties and getters (we were really bad on getters) that are more in
line with real life situations.
The *checkRawValidationSpeed* bench only evaluates the validation
performances while the *checkRawParsingSpeed* test only evaluates the
parsing and metadata building phase (so clearly not realistic but it's a
good way to exercise and optimize this part).
Results below.
TLDR :
- *Validation*: 6 as of my branch is ~ 2 times faster than BVal 1.1.2
and HV 5.4.1.Final
- *Parsing/Metadata building*
- 6.0.0.CR3: we had quite a regression here, it's ~ 2 times slower
than 5.4.1.Final
- as of my branch, we are still a little slower than BVal but faster
than 5.4.1.Final
*checkRawValidationSpeed* tester.repetitions=1000
in ms, lower is better
Run Apache Bval 1.1.2 HV 5.4.1.Final HV 6.0.0.CR3 HV 6.0.0-SNAPSHOT
1 24808 19756 15307 12069
2 24853 23015 17637 12031
3 24884 22610 17680 11892
4 24837 22350 17564 11821
*checkRawParsingSpeed* tester.repetitions=200
in ms, lower is better
Run Apache Bval 1.1.2 HV 5.4.1.Final HV 6.0.0.CR3 HV 6.0.0-SNAPSHOT
1 14387 18945 30046 14736
2 14409 17705 28250 14838
3 14539 17598 29500 14903
4 14279 18448 28876 15250
[1] http://carinae.net/2010/06/benchmarking-hibernate-validator-
and-apache-beanvalidation-the-two-jsr-303-implementations/
On Wed, Jul 19, 2017 at 4:44 PM, Guillaume Smet <guillaume.smet(a)gmail.com>
wrote:
Hi,
So, as we are leaning towards final, I made another round of benchmarking
on HV 6.
They are run on the same machine as the previous "Hibernate Validator 6
benchmarks results" I posted a few months ago [1] so they can be compared
to these numbers (you'll notice the 5.4.1 numbers are about the same
between the 2 emails).
Note that these benchmarks only exercises the validation engine but does
not report any violation.
I think I'll add some more benchmarks to ensure violation reporting also
moves in the right direction.
It might also be a good occasion to revisit this benchmark:
http://carinae.net/2010/06/benchmarking-hibernate-validator-and-apache-
beanvalidation-the-two-jsr-303-implementations/ . With a large value of
"revisit" as it's really basic but the scenarios are interesting. Note
that
I'm a bit pessimistic on the parsing part: we now do a lot more work than
before to deal with container elements so the startup cost will probably be
significantly higher.
== Apache BVal 1.1.2
Result "org.hibernate.validator.performance.simple.SimpleValidation.
testSimpleBeanValidation":
357.500 ±(99.9%) 5.327 ops/ms [Average]
(min, avg, max) = (337.100, 357.500, 381.472), stdev = 10.760
CI (99.9%): [352.173, 362.827] (assumes normal distribution)
Result "org.hibernate.validator.performance.cascaded.CascadedValidation.
testCascadedValidation":
379.605 ±(99.9%) 5.817 ops/ms [Average]
(min, avg, max) = (360.654, 379.605, 411.361), stdev = 11.750
CI (99.9%): [373.789, 385.422] (assumes normal distribution)
(not really clear how BVal happens to be faster in the cascaded validation
case)
== HV 5.4.1.Final
Result "org.hibernate.validator.performance.simple.SimpleValidation.
testSimpleBeanValidation":
558.196 ±(99.9%) 3.643 ops/ms [Average]
(min, avg, max) = (542.396, 558.196, 575.360), stdev = 7.360
CI (99.9%): [554.552, 561.839] (assumes normal distribution)
Result "org.hibernate.validator.performance.cascaded.CascadedValidation.
testCascadedValidation":
285.788 ±(99.9%) 1.970 ops/ms [Average]
(min, avg, max) = (278.611, 285.788, 298.530), stdev = 3.980
CI (99.9%): [283.817, 287.758] (assumes normal distribution)
== Master from March after the first round of improvements (numbers taken
from the previous email)
Result
"org.hibernate.validator.performance.simple.SimpleValidation.
testSimpleBeanValidation":
869.546 ±(99.9%) 14.734 ops/ms [Average]
(min, avg, max) = (760.007, 869.546, 909.206), stdev = 29.763
CI (99.9%): [854.813, 884.280] (assumes normal distribution)
Result
"org.hibernate.validator.performance.cascaded.CascadedValidation.
testCascadedValidation":
343.699 ±(99.9%) 2.077 ops/ms [Average]
(min, avg, max) = (331.333, 343.699, 352.626), stdev = 4.196
CI (99.9%): [341.622, 345.776] (assumes normal distribution)
== HV 6 - Current master with https://github.com/hibernate/
hibernate-validator/pull/814 applied
Result "org.hibernate.validator.performance.simple.SimpleValidation.
testSimpleBeanValidation":
924.121 ±(99.9%) 3.686 ops/ms [Average]
(min, avg, max) = (905.423, 924.121, 941.295), stdev = 7.446
CI (99.9%): [920.435, 927.807] (assumes normal distribution)
Result "org.hibernate.validator.performance.cascaded.CascadedValidation.
testCascadedValidation":
430.092 ±(99.9%) 3.661 ops/ms [Average]
(min, avg, max) = (416.439, 430.092, 447.607), stdev = 7.396
CI (99.9%): [426.431, 433.754] (assumes normal distribution)
== Conclusion
The good news is that the results are even better than the ones from
March, after some further tweaking.
We are significantly faster than BVal in these scenarios and also
significantly faster than 5.4.1.Final.
Now, we need to get this PR in :).
[1] http://lists.jboss.org/pipermail/hibernate-dev/2017-March/016057.html