[hibernate-dev] Jenkins job priorities

[Sanne Grinovero]

On 10 January 2018 at 11:33, Guillaume Smet <guillaume.smet at gmail.com> wrote:
> On Wed, Jan 10, 2018 at 12:08 PM, Sanne Grinovero <sanne at hibernate.org>
> wrote:
>>
>> Some of our test suites used to take 2 hours to run (even 5 days some
>> years ago); now you say waiting 20 minutes is not good enough? You'll
>> have to optimise our code better :P
>
>
> What I'm saying is that in the current setup, I don't wait at all when I
> have something to release.
>
> All is passed in parallel to the currently running jobs.
>
> And it works well.

I'm confused now. AFAIK this has never been the case? I understand
that the release process itself runs without running the tests, but
I'd still run the tests by triggering a full build before.
You made the example of the TCK and various tests; to run them you'd
not be allowed to run them in parallel with other builds, so you
wanted to release and the jobs happened to be building ORM and all its
RDBMS, you'd have had to wait for a couple hours.

>
>>
>> It's very easy to spin up extra nodes; my recommendation is that when
>> you know you're about to release [for example approximately one hour
>> in advance while you might be double-checking JIRA state and such
>> things] hit that manual scale-up button and have CI "warmed up" with
>> one or two extra nodes.
>>
>> By the time you need to trigger the release job you'll have the build
>> queue flushed, the priority plugin helping you out, and still
>> additional extra slaves running to run it all in parallel.
>>
>> And of course for many releases we don't care for an extra 30 minutes
>> so you're free to skip this all if it's not important; incidentally
>> for "work in progress" milestones like the module packs which we
>> recently re-released several times while polishing up the PR I've been
>> releasing from my local machine; it's good to have CI automate things
>> but I don't think we should get in a position to require 100%
>> availability from CI: practice releases locally sometimes.
>
>
> Well, the ultimate goal of releasing on CI is to have consistent releases
> and an automated process.
>
> I really don't want to build a release locally and be at risk of doing
> something wrong.
>
> That's the main purpose of an automated process and having a stable machine
> doing it.
>
>>
>> Let's not forget that many Apache projects take a week or two to
>> perform a release, we all know of other projects needing months, so by
>> the law of diminishing returns I don't think we should invest much
>> more of out time to shave on the 10 minutes.. just spin up some extra
>> nodes :)
>
>
> What I'm saying is that the current setup is working very well for releases
> and the proposed setup won't work as well.
>
> You can find all sorts of workarounds but it won't work as well and be as
> practical as it used to be. Yeah, you can think of starting another node 1
> hour before doing your release and hope it will still be there and you won't
> have another project's commit triggering 4 jobs just before you start. Sure.
> But I'm pretty sure it's going to be a pain.

Still I don't really understand if you're having a better idea. In a
nutshell these jobs need resources, if they are busy you either add
more resources, or change priorities, or you wait. That's the three
aspects you can play with "safely".

Then there's the option of playing with parallelism, but it's really
dangerous: it risks failing both your release and causing failures in
the other tests which are hard to expliain, cause confusion among us
all, and ultimately lead to have to repeat all involved jobs so
consuming unnecessarily more resources and time.
In many cases parallelism isn't even an option, for examplethe ORM
builds consume most system memory so you just can't run additional
JVMs to run the TCK or similar jobs; if it was safe, I would be using
smaller machines.

> I'm probably the one doing releases the most frequently with HV, that's why
> I am vocal about it.
>
> And maybe I'm the only one but, when I'm working on a release, I don't like
> to do stuff in parallel because I don't want to forget something or make a
> mistake. So I'm fully focused on it. Waiting 20 minutes before having my job
> running will be a complete waste of time. And if it has to happen more than
> one time on a given release time, I can predict I will get grumpy :).
>
> That being said, if you have nothing against me cancelling the running jobs
> because they are in the way, we can do that. But I'm not sure people will
> like it very much.

Just make sure you ask for permissions, but yea we've done that
previously, hopefully won't be needed often, but it's always an
option.

>
> --
> Guillaume
>