[hibernate-dev] Jenkins job priorities

[Steve Ebersole]

And in advance I say I would not be cool with you killing my jobs for your
job to run

On Wed, Jan 10, 2018 at 9:52 AM Sanne Grinovero <sanne at hibernate.org> wrote:

> On 10 January 2018 at 11:33, Guillaume Smet <guillaume.smet at gmail.com>
> wrote:
> > On Wed, Jan 10, 2018 at 12:08 PM, Sanne Grinovero <sanne at hibernate.org>
> > wrote:
> >>
> >> Some of our test suites used to take 2 hours to run (even 5 days some
> >> years ago); now you say waiting 20 minutes is not good enough? You'll
> >> have to optimise our code better :P
> >
> >
> > What I'm saying is that in the current setup, I don't wait at all when I
> > have something to release.
> >
> > All is passed in parallel to the currently running jobs.
> >
> > And it works well.
>
> I'm confused now. AFAIK this has never been the case? I understand
> that the release process itself runs without running the tests, but
> I'd still run the tests by triggering a full build before.
> You made the example of the TCK and various tests; to run them you'd
> not be allowed to run them in parallel with other builds, so you
> wanted to release and the jobs happened to be building ORM and all its
> RDBMS, you'd have had to wait for a couple hours.
>
> >
> >>
> >> It's very easy to spin up extra nodes; my recommendation is that when
> >> you know you're about to release [for example approximately one hour
> >> in advance while you might be double-checking JIRA state and such
> >> things] hit that manual scale-up button and have CI "warmed up" with
> >> one or two extra nodes.
> >>
> >> By the time you need to trigger the release job you'll have the build
> >> queue flushed, the priority plugin helping you out, and still
> >> additional extra slaves running to run it all in parallel.
> >>
> >> And of course for many releases we don't care for an extra 30 minutes
> >> so you're free to skip this all if it's not important; incidentally
> >> for "work in progress" milestones like the module packs which we
> >> recently re-released several times while polishing up the PR I've been
> >> releasing from my local machine; it's good to have CI automate things
> >> but I don't think we should get in a position to require 100%
> >> availability from CI: practice releases locally sometimes.
> >
> >
> > Well, the ultimate goal of releasing on CI is to have consistent releases
> > and an automated process.
> >
> > I really don't want to build a release locally and be at risk of doing
> > something wrong.
> >
> > That's the main purpose of an automated process and having a stable
> machine
> > doing it.
> >
> >>
> >> Let's not forget that many Apache projects take a week or two to
> >> perform a release, we all know of other projects needing months, so by
> >> the law of diminishing returns I don't think we should invest much
> >> more of out time to shave on the 10 minutes.. just spin up some extra
> >> nodes :)
> >
> >
> > What I'm saying is that the current setup is working very well for
> releases
> > and the proposed setup won't work as well.
> >
> > You can find all sorts of workarounds but it won't work as well and be as
> > practical as it used to be. Yeah, you can think of starting another node
> 1
> > hour before doing your release and hope it will still be there and you
> won't
> > have another project's commit triggering 4 jobs just before you start.
> Sure.
> > But I'm pretty sure it's going to be a pain.
>
> Still I don't really understand if you're having a better idea. In a
> nutshell these jobs need resources, if they are busy you either add
> more resources, or change priorities, or you wait. That's the three
> aspects you can play with "safely".
>
> Then there's the option of playing with parallelism, but it's really
> dangerous: it risks failing both your release and causing failures in
> the other tests which are hard to expliain, cause confusion among us
> all, and ultimately lead to have to repeat all involved jobs so
> consuming unnecessarily more resources and time.
> In many cases parallelism isn't even an option, for examplethe ORM
> builds consume most system memory so you just can't run additional
> JVMs to run the TCK or similar jobs; if it was safe, I would be using
> smaller machines.
>
> > I'm probably the one doing releases the most frequently with HV, that's
> why
> > I am vocal about it.
> >
> > And maybe I'm the only one but, when I'm working on a release, I don't
> like
> > to do stuff in parallel because I don't want to forget something or make
> a
> > mistake. So I'm fully focused on it. Waiting 20 minutes before having my
> job
> > running will be a complete waste of time. And if it has to happen more
> than
> > one time on a given release time, I can predict I will get grumpy :).
> >
> > That being said, if you have nothing against me cancelling the running
> jobs
> > because they are in the way, we can do that. But I'm not sure people will
> > like it very much.
>
> Just make sure you ask for permissions, but yea we've done that
> previously, hopefully won't be needed often, but it's always an
> option.
>
> >
> > --
> > Guillaume
> >
> _______________________________________________
> hibernate-dev mailing list
> hibernate-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hibernate-dev
>