[hibernate-dev] [HV] Hibernate Validator 6 benchmark updates

[Yoann Rodiere]

Kudos to you! I'm slowly beginning to understand what a pain such work can
be... Nice to have a reminder that this work can ultimately be rewarded ;)

Yoann Rodière
Hibernate NoORM Team
yoann at hibernate.org

On 19 July 2017 at 16:44, Guillaume Smet <guillaume.smet at gmail.com> wrote:

> Hi,
>
> So, as we are leaning towards final, I made another round of benchmarking
> on HV 6.
>
> They are run on the same machine as the previous "Hibernate Validator 6
> benchmarks results" I posted a few months ago [1] so they can be compared
> to these numbers (you'll notice the 5.4.1 numbers are about the same
> between the 2 emails).
>
> Note that these benchmarks only exercises the validation engine but does
> not report any violation.
>
> I think I'll add some more benchmarks to ensure violation reporting also
> moves in the right direction.
>
> It might also be a good occasion to revisit this benchmark:
> http://carinae.net/2010/06/benchmarking-hibernate-validator-and-apache-
> beanvalidation-the-two-jsr-303-implementations/
> . With a large value of "revisit" as it's really basic but the scenarios
> are interesting. Note that I'm a bit pessimistic on the parsing part: we
> now do a lot more work than before to deal with container elements so the
> startup cost will probably be significantly higher.
>
> == Apache BVal 1.1.2
>
> Result
> "org.hibernate.validator.performance.simple.SimpleValidation.
> testSimpleBeanValidation":
>   357.500 ±(99.9%) 5.327 ops/ms [Average]
>   (min, avg, max) = (337.100, 357.500, 381.472), stdev = 10.760
>   CI (99.9%): [352.173, 362.827] (assumes normal distribution)
>
> Result
> "org.hibernate.validator.performance.cascaded.CascadedValidation.
> testCascadedValidation":
>   379.605 ±(99.9%) 5.817 ops/ms [Average]
>   (min, avg, max) = (360.654, 379.605, 411.361), stdev = 11.750
>   CI (99.9%): [373.789, 385.422] (assumes normal distribution)
>
> (not really clear how BVal happens to be faster in the cascaded validation
> case)
>
> == HV 5.4.1.Final
>
> Result
> "org.hibernate.validator.performance.simple.SimpleValidation.
> testSimpleBeanValidation":
>   558.196 ±(99.9%) 3.643 ops/ms [Average]
>   (min, avg, max) = (542.396, 558.196, 575.360), stdev = 7.360
>   CI (99.9%): [554.552, 561.839] (assumes normal distribution)
>
> Result
> "org.hibernate.validator.performance.cascaded.CascadedValidation.
> testCascadedValidation":
>   285.788 ±(99.9%) 1.970 ops/ms [Average]
>   (min, avg, max) = (278.611, 285.788, 298.530), stdev = 3.980
>   CI (99.9%): [283.817, 287.758] (assumes normal distribution)
>
> == Master from March after the first round of improvements (numbers taken
> from the previous email)
>
> Result
> "org.hibernate.validator.performance.simple.SimpleValidation.
> testSimpleBeanValidation":
>   869.546 ±(99.9%) 14.734 ops/ms [Average]
>   (min, avg, max) = (760.007, 869.546, 909.206), stdev = 29.763
>   CI (99.9%): [854.813, 884.280] (assumes normal distribution)
>
> Result
> "org.hibernate.validator.performance.cascaded.CascadedValidation.
> testCascadedValidation":
>   343.699 ±(99.9%) 2.077 ops/ms [Average]
>   (min, avg, max) = (331.333, 343.699, 352.626), stdev = 4.196
>   CI (99.9%): [341.622, 345.776] (assumes normal distribution)
>
> == HV 6 - Current master with
> https://github.com/hibernate/hibernate-validator/pull/814 applied
>
> Result
> "org.hibernate.validator.performance.simple.SimpleValidation.
> testSimpleBeanValidation":
>   924.121 ±(99.9%) 3.686 ops/ms [Average]
>   (min, avg, max) = (905.423, 924.121, 941.295), stdev = 7.446
>   CI (99.9%): [920.435, 927.807] (assumes normal distribution)
>
> Result
> "org.hibernate.validator.performance.cascaded.CascadedValidation.
> testCascadedValidation":
>   430.092 ±(99.9%) 3.661 ops/ms [Average]
>   (min, avg, max) = (416.439, 430.092, 447.607), stdev = 7.396
>   CI (99.9%): [426.431, 433.754] (assumes normal distribution)
>
> == Conclusion
>
> The good news is that the results are even better than the ones from March,
> after some further tweaking.
>
> We are significantly faster than BVal in these scenarios and also
> significantly faster than 5.4.1.Final.
>
> Now, we need to get this PR in :).
>
> [1] http://lists.jboss.org/pipermail/hibernate-dev/2017-March/016057.html
> _______________________________________________
> hibernate-dev mailing list
> hibernate-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hibernate-dev