[hibernate-dev] Analyzing the ORM testsuite

Fri Apr 20 08:29:56 EDT 2018

The postgres build is fine again, thanks for all the help!

We re-enabled the CI job.

On Fri, 20 Apr 2018, 01:14 Steve Ebersole, <steve at hibernate.org> wrote:

>
> I checked - `build` does in fact pull in a bunch of tasks that you do not
> care about (checkstyle, findbugs, etc).
>
> The build I just ran did take longer than I have been seeing - it took 16
> minutes, but that's not the magnitude you see.
>
> I mentioned performing test caused javadoc to be generated.  I found the
> culprit for that - hibernate-orm-modules.  Any time tests
> use hibernate-orm-modules, hibernate-orm-modules triggers a full build and
> publish of the project's artifacts (jar, source jar, javadoc jar).  This
> accounts for at least some of the additional time.  I think using build
> rather than test probably accounts for the rest
>
>
> On Thu, Apr 19, 2018 at 5:35 PM Sanne Grinovero <sanne at hibernate.org>
> wrote:
>
>> On 19 April 2018 at 22:50, Chris Cranford <chris at hibernate.org> wrote:
>> > The entire build process under OpenJDK 8 here:
>> >
>> > BUILD SUCCESSFUL in 47m 26s
>> > 251 actionable tasks: 233 executed, 18 up-to-date
>> >
>> > The only thing I noticed was that while executing the hibernate-core
>> tests,
>> > the memory consumption seemed quite high toward the end of the test
>> suite,
>> > particularly ~3.5GB memory consumed just to run the hibernate-core tests
>> > alone and that FindBugs complained about some magic numbers on some zip
>> > files while inspecting hibernate-osgi.
>> >
>> > Any idea if any substantial amount of that memory consumption related
>> to the
>> > test framework tracking the output/errors of those executed tests?
>>
>> I guess that confirms there's something specific to my environment.
>> That's a great hint to have, thanks!
>>
>> Regarding the memory, yes I'm seeing the same. >90% of that is
>> Mockito, I'm exploring some ideas to reduce its memory usage; I have
>> some working experiments but I'm not happy with them: they make
>> maintenance of tests error prone so tomorrow I'll try some
>> alternatives.
>>
>>
>> >
>> > On 04/19/2018 05:20 PM, Sanne Grinovero wrote:
>> >
>> > On 19 April 2018 at 21:47, Chris Cranford <chris at hibernate.org> wrote:
>> >
>> > Sanne -
>> >
>> > Are you running the build task or are you executing other more specific
>> > tasks and noticing the slowness?
>> >
>> > Just running "gradle clean build" from the root. I had the same
>> > results with OpenJDK 8 and 9, twice. Getting exactly the same failures
>> > as before.
>> >
>> > Thanks!
>> >
>> > On 04/19/2018 04:25 PM, Steve Ebersole wrote:
>> >
>> > Nothing off the top of my head.  The build time is in normal range on my
>> > machine as of this morning (8 - 10 minutes).
>> >
>> > You mentioned a specific commit.  Is that just when you noticed a
>> change,
>> > or do you suspect something in that commit?
>> >
>> > I'm not at computer right now, but when I get back I will check out that
>> > commit and see if anything is strange locally.
>> >
>> >
>> >
>> > On Thu, Apr 19, 2018, 3:11 PM Sanne Grinovero <sanne at hibernate.org>
>> wrote:
>> >
>> > Hi all,
>> >
>> > I'm trying to analyze the memory usage of current master of Hibernate
>> > ORM; I started looking because of OOM errors on the PostgreSQL
>> > testsuite (as mentioned in another thread).
>> >
>> > So far I've seen that Mockito is allocating a "lot of stuff"; I might
>> > be able to improve some things in that area but it's a distraction
>> > from my goal as clearly we use Mockito even when not running the
>> > PostgreSQL backed tests and that seems to run fine. But it's polluting
>> > the data reports so making it hard (and slow!) to verify if there is a
>> > real issue.
>> >
>> > I could use some help to verify some tangential issues though; when
>> > running the testsuite locally it's taking much longer than what I've
>> > been used to see for the same testsuite some weeks ago, and I have
>> > some failures - regularly seeing the same failures.
>> >
>> > These are triggered by running a default build, on H2:
>> >
>> > Task :hibernate-core:test
>> >
>> > org.hibernate.test.criteria.CriteriaLockingTest >
>> >
>> >
>> testSetLockModeDifferentFromNONELogAWarnMessageWhenTheDialectUseFollowOnLockingIsTrue
>> > FAILED
>> >     org.junit.runners.model.TestTimedOutException
>> >
>> > org.hibernate.test.criteria.CriteriaLockingTest >
>> >
>> >
>> testSetLockModeNONEDoNotLogAWarnMessageWhenTheDialectUseFollowOnLockingIsTrue
>> > FAILED
>> >     java.lang.Exception
>> >
>> > org.hibernate.test.criteria.CriteriaLockingTest >
>> > org.hibernate.test.criteria.CriteriaLockingTest FAILED
>> >     java.lang.Exception
>> >
>> > org.hibernate.test.locking.warning.LockNoneWarmingTest >
>> > org.hibernate.test.locking.warning.LockNoneWarmingTest FAILED
>> >     java.lang.Exception
>> >
>> > 6307 tests completed, 4 failed, 327 skipped
>> >
>> > Yet the puzzling point: on ci.hibernate.org I don't see these failures
>> > happening, nor the testsuite time seems to have increased of any
>> > significant amount (it regularly completes in somewhere between 15 and
>> > 20 minutes). It takes me more than an hour to get this result on a
>> > machine which would normally complete the testsuite in 8 minutes, and
>> > I got the same 4 failures when re-running it a second time.
>> >
>> > Could other developers also please checkout master - specifically at
>> > commit 291d4a3eeaa2ade32a42cfbcad5868b2114c34fe - and let me know if
>> > it works allright for you and if the build time is consistent with the
>> > usual times for your machine? I need to understand what might be
>> > different between my workstation and the CI server.
>> >
>> > I might even have some great improvements ready to reduce the
>> > testsuite time, but I'm not comfortable in committing them in this
>> > context.
>> >
>> > Thanks,
>> > Sanne
>> > _______________________________________________
>> > hibernate-dev mailing list
>> > hibernate-dev at lists.jboss.org
>> > https://lists.jboss.org/mailman/listinfo/hibernate-dev
>> >
>> > _______________________________________________
>> > hibernate-dev mailing list
>> > hibernate-dev at lists.jboss.org
>> > https://lists.jboss.org/mailman/listinfo/hibernate-dev
>> >
>> >
>> > _______________________________________________
>> > hibernate-dev mailing list
>> > hibernate-dev at lists.jboss.org
>> > https://lists.jboss.org/mailman/listinfo/hibernate-dev
>> >
>> >
>> _______________________________________________
>> hibernate-dev mailing list
>> hibernate-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/hibernate-dev
>>
>