[hibernate-dev] ORM Jenkins Builds

Sanne Grinovero sanne at hibernate.org
Sat Mar 28 12:26:42 EDT 2015

On 27 March 2015 at 22:25, Steve Ebersole <steve at hibernate.org> wrote:
> We should remove that -XX:MaxPermSize=512m setting as we discussed before.

I've changed this to:
-server -Xmx2G -Xms800M  -Djava.net.preferIPv4Stack=true
-Djgroups.bind_addr= -XX:+PrintTenuringDistribution
-XX:+PrintGCDetails -Xloggc:/home/jenkins/gradle-gcdetails.log
-XX:+PrintFlagsFinal -XX:NewRatio=1 -XX:+UseG1GC

The logging option should not slow down significantly, but let us know
if there ever are issues with heap memory.

> I understand all of that.  I still think its probably worthwhile to look at
> minimizing the amount of time and resources.  For a couple of reasons.
> First I am doing a lot of dev right now.  In the 40-50 minutes it takes to
> run the CI job after a push I often have other pushes queued up.  Then when
> that first jobs fails, I need to go back and shift gears back into that
> first issue again.  And spending roughly 75-80% of that time in a task that
> we don't even fail the build for seems excessive.
> My preliminary thinking here is to define a more than one job.  Setting
> aside the idea of db-jobs for the moment...
> 1) Per-push job - triggered by push
> a) runs tests
> b) simplified checks (findbugs, checkstyle) that we actually fail the build
> for
> c) publish
> 2) Nightly build - triggered nightly (only with change in source?)
> a) full checks (both failures and warnings)
> b) aggregated javadocs
> * I'd actually like the nightly build to trigger once a day when a push
> actually occured.  Not sure that is possible.

+1, feel free to create as many jobs as needed. There is a "clone job"
option which I've found handy.

> The "per-push" job really is a "main" one meant to run against H2.  We still
> need to set up the db specific jobs:

Right, I even suspect that using the locally installed MySQL or
PostgreSQL instances could result in faster builds than with H2

> 3) DB job - triggered on successful execution of (1)?
> a) run tests
> And ultimately I'd love to setup some jobs for JDK testing that pull the
> artifacts built by (1) and tests them in a different JDK.  Currently we need
> the Java 8 JDK to do a full build of ORM (JSR 310 support).  For the modules
> other than hibernate-java8 we set source and target compatibility to Java 6.
> We could simply set up some jobs that use Java 6 JDK for the build, but
> that's not quite the same as pulling the artifacts built using Java 8 and
> targetting Java 6 and trying to run those in Java 6 or 7 environment.  I
> don't know if Gradle has any existing support for this or whether this is
> another plugin I will have to develop to make that happen.

Consider also that since we now have N slaves, there is no guarantee
that a second job will run on the same build slave.
So if the JDK8 build produces artifacts but only deployes them in the
local repository, the second node might not find the same as it has an
independent repository cache.
One way would be to actually upload the snapshots to the jboss Nexus,
but it could be easier / more efficient to have a single job which
does the compilation & packaging only with JDK8 but then runs the
tests with JDK6 ?
The JDK paths are well defined, so that should be scriptable with a
couple of sequential invocations to Gradle?


> On Fri, Mar 27, 2015 at 4:38 PM, Sanne Grinovero <sanne at hibernate.org>
> wrote:
>> Hi Steve,
>> don't worry for the resources they consume. Each build job which we
>> run is fully isolated and running on independent hardware, so it's not
>> going to slow down any other build.
>> Since we now have a set of small servers, one each dedicated for a job
>> at a time, I've opted for low spec machines as an experiment. This
>> performance is however disappointing, we can try different types of
>> CPU/Memory/Disk categories.
>> The current specs of each build node are:
>>  2 CPUs
>>  3 GB of RAM
>>  18GB of root disk (of which 9GB available for our builds)
>>  50GB available to run various docker images and temporary databases
>> (allocated per job need)
>> They run Fedora 21, Cloud Edition.
>> As you can see, resources on a single build node are very limited -
>> especially space and memory.
>> But since there is no swap space configured (!) I expect that in case
>> the memory is not enough, it will crash (and therefore let us know)
>> rather than slow down.
>> It's possible though that since the ORM builds allocate 2GB of heap,
>> we're preventing the operating system from doing proper filesystem
>> caching.
>> currently we have:
>> GRADLE_OPTS="-Xmx2G -Xms2G -XX:MaxPermSize=512m"
>> I'm going to try to lower that, bear with me if there are failures. If
>> that doesn't work we can definitely replace these with larger
>> instances, it should take us just some minutes as it's all automated:
>> the only reason for me to be conservative is to make fair use of these
>> (free) resource, and also I'd like us to actually test for not needing
>> much ram - we can get there in smaller steps as needed of course.
>> Sanne
>> On 27 March 2015 at 21:01, Steve Ebersole <steve at hibernate.org> wrote:
>> > I was curious why it took so long to run the master ORM jobs on the CI
>> > machine compared to running the job locally.  Locally I run `clean test`
>> > at
>> > the root prject quite often and it takes roughly 9-10 minutes.  The
>> > master
>> > CI jobs generally take 45-50 minutes to complete.
>> >
>> > So I enabled "Gradle build profiling" in our job.  The results were
>> > surprising in terms of ratios.
>> >
>> > I figured findBugs, checkStyle etc probably added significant times to
>> > the
>> > build.  But I was shocked how much it added.
>> >
>> > BTW, you can view these profile reports in
>> > {root}/build/reports/profile...
>> >
>> > So hibernate-core, overall took 17m22.19s to run for one job.  Of that,
>> > 12.5 was findBugs!  checkStyle as actually "reasonable" at just under
>> > 30s.
>> > The ratios were similar across all modules.
>> >
>> > The aggregatedJavadoc task took a shade over 2m.
>> >
>> > Considering that these jobs are run on ever check-in (and eventually it
>> > would be great to auto-run them against PRs too), plus the fact that we
>> > aren't even failing the build for the majority of findBug/checkStyle
>> > hits I
>> > think we should define these jobs a little differently.  Its not just
>> > the
>> > time it takes.  Yes we all hate to wait.  But it's also the CI resources
>> > taken up.
>> >
>> > I'm going to put some thought into this after the 5.0 Beta release, but
>> > I
>> > wanted to get some thoughts and feedback in the meantime.  Things to
>> > consider.
>> > _______________________________________________
>> > hibernate-dev mailing list
>> > hibernate-dev at lists.jboss.org
>> > https://lists.jboss.org/mailman/listinfo/hibernate-dev

More information about the hibernate-dev mailing list