[wildfly-dev] Add pull request CI coverage of Galleon layers, drop Windows JDK 8

[James Perkins]

On Mon, Oct 28, 2019 at 6:47 PM Brian Stansberry <
brian.stansberry at redhat.com> wrote:

> I'd like to add jobs to the automatic testing of PRs to run the testsuite
> using slimmed installations provisioned by Galleon. But, we already run a
> lot of jobs for each PR, enough so that our CI can overburdened during busy
> times around deadlines. So I don't want to just add jobs. Instead I also
> propose to drop the Windows + JDK 8 jobs.
>
> Galleon Testing
>
> During our work on WildFly 18 I added the ability to run portions of the
> WildFly and WildFly Core testsuites with the tests executing against
> slimmed server installations provisioned by Galleon instead of against the
> complete installations normally used. The point of that was to get test
> coverage of those slimmed installations.
>
> Currently we have nightly jobs that run the testsuite this way.[1] As we
> continue to evolve our set of Galleon layers, e.g. adding layers for
> MicroProfile specs, I want to be sure we catch problems before PRs get
> merged.
>
> To run tests locally this way you pass -Dts.layers as an arg to maven.
>
> Dropping Windows JDK 8 Jobs
>
> If we'd drop something in order to free up resources for these Galleon
> jobs, the Windows JDK 8 ones seem a good choice. We'd still run PRs against
> Windows JDK 11, and we'd still run PRs against Linux JDK 8. I can't recall
> any situation where CI found a regression that was specific to the
> Windows + JDK 8 combination.
>
> When CI gets overloaded during rush times, it's the Windows jobs that are
> most problematic. The Windows jobs take longer because the storage drives
> they use are slower. Plus we have fewer Windows agents. The effect is
> during a rush, overall CI for PRs ends up taking hours longer while we wait
> for Windows agents to come free and then run the job.
>
> We'd still run nightly jobs with Windows + JDK 8 so in the off chance
> there's a problem it would get noticed that way.
>

This seems reasonable to me. One more option to add would be that before
bulk PR's are merged maybe we run Windows JDK8 jobs against those. Though
those jobs are slow so...

One other option too is maybe Windows JDK8 jobs just don't run the full
test suite. The main place I could possibly see issues would be if scripts
change since there are some decisions made based on the JVM. Though kicking
off a manual job there isn't a huge deal as it's likely not common.

Somewhat related if we do remove Windows JDK8 jobs I think we should just
remove them from the aggreator job, but keep them under the Pull Request.
The reason for this is if we do want to run a custom one we could use the
PR number from "Changes" tab when running a custom job and it would report
back to the PR.


>
> Specifics
>
> For PRs against wildfly/wildfly I'd add a job equivalent to
> https://ci.wildfly.org/viewType.html?buildTypeId=WF_GalleonLinuxJdk11 and
> then drop
> https://ci.wildfly.org/viewType.html?buildTypeId=WildFlyCore_PullRequestWindows
>
> For PRs against wildfly/wildfly-core I'd add jobs equivalent to
> https://ci.wildfly.org/viewType.html?buildTypeId=WildFlyCore_GalleonLinuxJdk8
> and https://ci.wildfly.org/viewType.html?buildTypeId=WF_GalleonLinuxJdk8
> and then drop
> https://ci.wildfly.org/viewType.html?buildTypeId=WildFlyCore_PullRequestWindows
>
> Any thoughts on this?
>

No objection from me.


>
>
> Best regards,
> Brian
>
> [1] See
> https://ci.wildfly.org/viewType.html?buildTypeId=WF_GalleonLinuxJdk11
> https://ci.wildfly.org/viewType.html?buildTypeId=WF_GalleonLinuxJdk8
> https://ci.wildfly.org/viewType.html?buildTypeId=WF_GalleonWindowsJdk11
>
> https://ci.wildfly.org/viewType.html?buildTypeId=WildFlyCore_GalleonLinuxJdk8
>
> https://ci.wildfly.org/viewType.html?buildTypeId=WildFlyCore_GalleonLinuxJdk11
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev



-- 
James R. Perkins
JBoss by Red Hat
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/wildfly-dev/attachments/20191031/6556face/attachment.html