[hibernate-dev] ORM Jenkins Builds

[Steve Ebersole]

We should remove that -XX:MaxPermSize=512m setting as we discussed before.

I understand all of that.  I still think its probably worthwhile to look at
minimizing the amount of time and resources.  For a couple of reasons.
First I am doing a lot of dev right now.  In the 40-50 minutes it takes to
run the CI job after a push I often have other pushes queued up.  Then when
that first jobs fails, I need to go back and shift gears back into that
first issue again.  And spending roughly 75-80% of that time in a task that
we don't even fail the build for seems excessive.

My preliminary thinking here is to define a more than one job.  Setting
aside the idea of db-jobs for the moment...

1) Per-push job - triggered by push
a) runs tests
b) simplified checks (findbugs, checkstyle) that we actually fail the build
for
c) publish

2) Nightly build - triggered nightly (only with change in source?)
a) full checks (both failures and warnings)
b) aggregated javadocs

* I'd actually like the nightly build to trigger once a day when a push
actually occured.  Not sure that is possible.

The "per-push" job really is a "main" one meant to run against H2.  We
still need to set up the db specific jobs:

3) DB job - triggered on successful execution of (1)?
a) run tests

And ultimately I'd love to setup some jobs for JDK testing that pull the
artifacts built by (1) and tests them in a different JDK.  Currently we
need the Java 8 JDK to do a full build of ORM (JSR 310 support).  For the
modules other than hibernate-java8 we set source and target compatibility
to Java 6.  We could simply set up some jobs that use Java 6 JDK for the
build, but that's not quite the same as pulling the artifacts built using
Java 8 and targetting Java 6 and trying to run those in Java 6 or 7
environment.  I don't know if Gradle has any existing support for this or
whether this is another plugin I will have to develop to make that happen.


On Fri, Mar 27, 2015 at 4:38 PM, Sanne Grinovero <sanne at hibernate.org>
wrote:

> Hi Steve,
> don't worry for the resources they consume. Each build job which we
> run is fully isolated and running on independent hardware, so it's not
> going to slow down any other build.
>
> Since we now have a set of small servers, one each dedicated for a job
> at a time, I've opted for low spec machines as an experiment. This
> performance is however disappointing, we can try different types of
> CPU/Memory/Disk categories.
>
> The current specs of each build node are:
>  2 CPUs
>  3 GB of RAM
>  18GB of root disk (of which 9GB available for our builds)
>  50GB available to run various docker images and temporary databases
> (allocated per job need)
>
> They run Fedora 21, Cloud Edition.
>
> As you can see, resources on a single build node are very limited -
> especially space and memory.
> But since there is no swap space configured (!) I expect that in case
> the memory is not enough, it will crash (and therefore let us know)
> rather than slow down.
> It's possible though that since the ORM builds allocate 2GB of heap,
> we're preventing the operating system from doing proper filesystem
> caching.
>
> currently we have:
> GRADLE_OPTS="-Xmx2G -Xms2G -XX:MaxPermSize=512m"
>
> I'm going to try to lower that, bear with me if there are failures. If
> that doesn't work we can definitely replace these with larger
> instances, it should take us just some minutes as it's all automated:
> the only reason for me to be conservative is to make fair use of these
> (free) resource, and also I'd like us to actually test for not needing
> much ram - we can get there in smaller steps as needed of course.
>
> Sanne
>
> On 27 March 2015 at 21:01, Steve Ebersole <steve at hibernate.org> wrote:
> > I was curious why it took so long to run the master ORM jobs on the CI
> > machine compared to running the job locally.  Locally I run `clean test`
> at
> > the root prject quite often and it takes roughly 9-10 minutes.  The
> master
> > CI jobs generally take 45-50 minutes to complete.
> >
> > So I enabled "Gradle build profiling" in our job.  The results were
> > surprising in terms of ratios.
> >
> > I figured findBugs, checkStyle etc probably added significant times to
> the
> > build.  But I was shocked how much it added.
> >
> > BTW, you can view these profile reports in
> {root}/build/reports/profile...
> >
> > So hibernate-core, overall took 17m22.19s to run for one job.  Of that,
> > 12.5 was findBugs!  checkStyle as actually "reasonable" at just under
> 30s.
> > The ratios were similar across all modules.
> >
> > The aggregatedJavadoc task took a shade over 2m.
> >
> > Considering that these jobs are run on ever check-in (and eventually it
> > would be great to auto-run them against PRs too), plus the fact that we
> > aren't even failing the build for the majority of findBug/checkStyle
> hits I
> > think we should define these jobs a little differently.  Its not just the
> > time it takes.  Yes we all hate to wait.  But it's also the CI resources
> > taken up.
> >
> > I'm going to put some thought into this after the 5.0 Beta release, but I
> > wanted to get some thoughts and feedback in the meantime.  Things to
> > consider.
> > _______________________________________________
> > hibernate-dev mailing list
> > hibernate-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/hibernate-dev
>