[wildfly-dev] Add pull request CI coverage of Galleon layers, drop Windows JDK 8

[Brian Stansberry]

On Thu, Oct 31, 2019 at 7:10 PM James Perkins <jperkins at redhat.com> wrote:

>
>
> On Mon, Oct 28, 2019 at 6:47 PM Brian Stansberry <
> brian.stansberry at redhat.com> wrote:
>
>> I'd like to add jobs to the automatic testing of PRs to run the testsuite
>> using slimmed installations provisioned by Galleon. But, we already run a
>> lot of jobs for each PR, enough so that our CI can overburdened during busy
>> times around deadlines. So I don't want to just add jobs. Instead I also
>> propose to drop the Windows + JDK 8 jobs.
>>
>> Galleon Testing
>>
>> During our work on WildFly 18 I added the ability to run portions of the
>> WildFly and WildFly Core testsuites with the tests executing against
>> slimmed server installations provisioned by Galleon instead of against the
>> complete installations normally used. The point of that was to get test
>> coverage of those slimmed installations.
>>
>> Currently we have nightly jobs that run the testsuite this way.[1] As we
>> continue to evolve our set of Galleon layers, e.g. adding layers for
>> MicroProfile specs, I want to be sure we catch problems before PRs get
>> merged.
>>
>> To run tests locally this way you pass -Dts.layers as an arg to maven.
>>
>> Dropping Windows JDK 8 Jobs
>>
>> If we'd drop something in order to free up resources for these Galleon
>> jobs, the Windows JDK 8 ones seem a good choice. We'd still run PRs against
>> Windows JDK 11, and we'd still run PRs against Linux JDK 8. I can't recall
>> any situation where CI found a regression that was specific to the
>> Windows + JDK 8 combination.
>>
>> When CI gets overloaded during rush times, it's the Windows jobs that are
>> most problematic. The Windows jobs take longer because the storage drives
>> they use are slower. Plus we have fewer Windows agents. The effect is
>> during a rush, overall CI for PRs ends up taking hours longer while we wait
>> for Windows agents to come free and then run the job.
>>
>> We'd still run nightly jobs with Windows + JDK 8 so in the off chance
>> there's a problem it would get noticed that way.
>>
>
> This seems reasonable to me. One more option to add would be that before
> bulk PR's are merged maybe we run Windows JDK8 jobs against those. Though
> those jobs are slow so...
>
> I'm not enthused about more steps in the merge process. :) Of course it's
fine if people want to do it and it's good to do when dealing with a PR
that has the look of possibly being different, e.g. windows script stuff.


> One other option too is maybe Windows JDK8 jobs just don't run the full
> test suite. The main place I could possibly see issues would be if scripts
> change since there are some decisions made based on the JVM. Though kicking
> off a manual job there isn't a huge deal as it's likely not common.
>

If you have some ideas on how the ts + ci jobs could be configured to get
that result, that would be good.

What I did with ts.layers is the root pom has a profile that turns off the
default surefire execution. And then in parts of the ts that I wanted to
run, there's a profile that turns things on. You have to deal with a few
maven modules that have more than the default surefire execution though.


> Somewhat related if we do remove Windows JDK8 jobs I think we should just
> remove them from the aggreator job, but keep them under the Pull Request.
> The reason for this is if we do want to run a custom one we could use the
> PR number from "Changes" tab when running a custom job and it would report
> back to the PR.
>

+1; I'll do it that way.

>
>
>>
>> Specifics
>>
>> For PRs against wildfly/wildfly I'd add a job equivalent to
>> https://ci.wildfly.org/viewType.html?buildTypeId=WF_GalleonLinuxJdk11
>> and then drop
>> https://ci.wildfly.org/viewType.html?buildTypeId=WildFlyCore_PullRequestWindows
>>
>> For PRs against wildfly/wildfly-core I'd add jobs equivalent to
>> https://ci.wildfly.org/viewType.html?buildTypeId=WildFlyCore_GalleonLinuxJdk8
>> and https://ci.wildfly.org/viewType.html?buildTypeId=WF_GalleonLinuxJdk8
>> and then drop
>> https://ci.wildfly.org/viewType.html?buildTypeId=WildFlyCore_PullRequestWindows
>>
>> Any thoughts on this?
>>
>
> No objection from me.
>
>
>>
>>
>> Best regards,
>> Brian
>>
>> [1] See
>> https://ci.wildfly.org/viewType.html?buildTypeId=WF_GalleonLinuxJdk11
>> https://ci.wildfly.org/viewType.html?buildTypeId=WF_GalleonLinuxJdk8
>> https://ci.wildfly.org/viewType.html?buildTypeId=WF_GalleonWindowsJdk11
>>
>> https://ci.wildfly.org/viewType.html?buildTypeId=WildFlyCore_GalleonLinuxJdk8
>>
>> https://ci.wildfly.org/viewType.html?buildTypeId=WildFlyCore_GalleonLinuxJdk11
>> _______________________________________________
>> wildfly-dev mailing list
>> wildfly-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
>
>
> --
> James R. Perkins
> JBoss by Red Hat
>


-- 
Brian Stansberry
Manager, Senior Principal Software Engineer
Red Hat
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/wildfly-dev/attachments/20191101/b4ff3093/attachment.html