[hibernate-dev] [HV] Hibernate Validator 6 benchmark updates

Mon Jul 24 08:54:49 EDT 2017

Hi Guillaume,

Awesome, thanks a lot for getting the numbers and applying the improvements.

So to be sure, at runtime (non-metadata retrieval), CR3 already is faster
than 5.4, also without your latest optimisations from the pending PR? You
mentioned this morning that having the first 6 commits from your PR would
be beneficial for the Final, but I think that's about reducing memory
usage, correct?

Thanks,

--Gunnar

2017-07-21 18:47 GMT+02:00 Guillaume Smet <guillaume.smet at gmail.com>:

> Hi,
>
> (Not a big fan of HTML emails but I gave up with formatting the table,
> sorry)
>
> So I took the time to revisit the old BVal benchmark [1] and posted an
> updated version of it here: https://github.com/gsmet/
> beanvalidation-benchmark.
>
> I pushed a few more commits to optimize the parsing/metadata building phase
> here: https://github.com/hibernate/hibernate-validator/pull/814.
>
> Note that the benchmark is much more representative that our very simple
> JMH tests: it generates a lot of bean definitions with constraints on
> properties and getters (we were really bad on getters) that are more in
> line with real life situations.
>
> The *checkRawValidationSpeed* bench only evaluates the validation
> performances while the *checkRawParsingSpeed* test only evaluates the
> parsing and metadata building phase (so clearly not realistic but it's a
> good way to exercise and optimize this part).
>
> Results below.
>
> TLDR :
>
>    - *Validation*: 6 as of my branch is ~ 2 times faster than BVal 1.1.2
>    and HV 5.4.1.Final
>    - *Parsing/Metadata building*
>       - 6.0.0.CR3: we had quite a regression here, it's ~ 2 times slower
>       than 5.4.1.Final
>       - as of my branch, we are still a little slower than BVal but faster
>       than 5.4.1.Final
>
> *checkRawValidationSpeed* tester.repetitions=1000
> in ms, lower is better
>
>
>
>
>
> Run Apache Bval 1.1.2 HV 5.4.1.Final HV 6.0.0.CR3 HV 6.0.0-SNAPSHOT
> 1 24808 19756 15307 12069
> 2 24853 23015 17637 12031
> 3 24884 22610 17680 11892
> 4 24837 22350 17564 11821
>
>
>
>
>
> *checkRawParsingSpeed* tester.repetitions=200
> in ms, lower is better
>
>
>
>
>
> Run Apache Bval 1.1.2 HV 5.4.1.Final HV 6.0.0.CR3 HV 6.0.0-SNAPSHOT
> 1 14387 18945 30046 14736
> 2 14409 17705 28250 14838
> 3 14539 17598 29500 14903
> 4 14279 18448 28876 15250
>
>
> [1] http://carinae.net/2010/06/benchmarking-hibernate-validator-
> and-apache-beanvalidation-the-two-jsr-303-implementations/
>
>
> On Wed, Jul 19, 2017 at 4:44 PM, Guillaume Smet <guillaume.smet at gmail.com>
> wrote:
>
> > Hi,
> >
> > So, as we are leaning towards final, I made another round of benchmarking
> > on HV 6.
> >
> > They are run on the same machine as the previous "Hibernate Validator 6
> > benchmarks results" I posted a few months ago [1] so they can be compared
> > to these numbers (you'll notice the 5.4.1 numbers are about the same
> > between the 2 emails).
> >
> > Note that these benchmarks only exercises the validation engine but does
> > not report any violation.
> >
> > I think I'll add some more benchmarks to ensure violation reporting also
> > moves in the right direction.
> >
> > It might also be a good occasion to revisit this benchmark:
> > http://carinae.net/2010/06/benchmarking-hibernate-validator-and-apache-
> > beanvalidation-the-two-jsr-303-implementations/ . With a large value of
> > "revisit" as it's really basic but the scenarios are interesting. Note
> that
> > I'm a bit pessimistic on the parsing part: we now do a lot more work than
> > before to deal with container elements so the startup cost will probably
> be
> > significantly higher.
> >
> > == Apache BVal 1.1.2
> >
> > Result "org.hibernate.validator.performance.simple.SimpleValidation.
> > testSimpleBeanValidation":
> >   357.500 ±(99.9%) 5.327 ops/ms [Average]
> >   (min, avg, max) = (337.100, 357.500, 381.472), stdev = 10.760
> >   CI (99.9%): [352.173, 362.827] (assumes normal distribution)
> >
> > Result "org.hibernate.validator.performance.cascaded.CascadedValidation.
> > testCascadedValidation":
> >   379.605 ±(99.9%) 5.817 ops/ms [Average]
> >   (min, avg, max) = (360.654, 379.605, 411.361), stdev = 11.750
> >   CI (99.9%): [373.789, 385.422] (assumes normal distribution)
> >
> > (not really clear how BVal happens to be faster in the cascaded
> validation
> > case)
> >
> > == HV 5.4.1.Final
> >
> > Result "org.hibernate.validator.performance.simple.SimpleValidation.
> > testSimpleBeanValidation":
> >   558.196 ±(99.9%) 3.643 ops/ms [Average]
> >   (min, avg, max) = (542.396, 558.196, 575.360), stdev = 7.360
> >   CI (99.9%): [554.552, 561.839] (assumes normal distribution)
> >
> > Result "org.hibernate.validator.performance.cascaded.CascadedValidation.
> > testCascadedValidation":
> >   285.788 ±(99.9%) 1.970 ops/ms [Average]
> >   (min, avg, max) = (278.611, 285.788, 298.530), stdev = 3.980
> >   CI (99.9%): [283.817, 287.758] (assumes normal distribution)
> >
> > == Master from March after the first round of improvements (numbers taken
> > from the previous email)
> >
> > Result
> > "org.hibernate.validator.performance.simple.SimpleValidation.
> > testSimpleBeanValidation":
> >   869.546 ±(99.9%) 14.734 ops/ms [Average]
> >   (min, avg, max) = (760.007, 869.546, 909.206), stdev = 29.763
> >   CI (99.9%): [854.813, 884.280] (assumes normal distribution)
> >
> > Result
> > "org.hibernate.validator.performance.cascaded.CascadedValidation.
> > testCascadedValidation":
> >   343.699 ±(99.9%) 2.077 ops/ms [Average]
> >   (min, avg, max) = (331.333, 343.699, 352.626), stdev = 4.196
> >   CI (99.9%): [341.622, 345.776] (assumes normal distribution)
> >
> > == HV 6 - Current master with https://github.com/hibernate/
> > hibernate-validator/pull/814 applied
> >
> > Result "org.hibernate.validator.performance.simple.SimpleValidation.
> > testSimpleBeanValidation":
> >   924.121 ±(99.9%) 3.686 ops/ms [Average]
> >   (min, avg, max) = (905.423, 924.121, 941.295), stdev = 7.446
> >   CI (99.9%): [920.435, 927.807] (assumes normal distribution)
> >
> > Result "org.hibernate.validator.performance.cascaded.CascadedValidation.
> > testCascadedValidation":
> >   430.092 ±(99.9%) 3.661 ops/ms [Average]
> >   (min, avg, max) = (416.439, 430.092, 447.607), stdev = 7.396
> >   CI (99.9%): [426.431, 433.754] (assumes normal distribution)
> >
> > == Conclusion
> >
> > The good news is that the results are even better than the ones from
> > March, after some further tweaking.
> >
> > We are significantly faster than BVal in these scenarios and also
> > significantly faster than 5.4.1.Final.
> >
> > Now, we need to get this PR in :).
> >
> > [1] http://lists.jboss.org/pipermail/hibernate-dev/2017-
> March/016057.html
> >
> _______________________________________________
> hibernate-dev mailing list
> hibernate-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hibernate-dev
>