Re: [wildfly-dev] Add pull request CI coverage of Galleon layers, drop Windows JDK 8

On Thu, Oct 31, 2019 at 7:10 PM James Perkins <jperkins(a)redhat.com&gt; wrote: > > > On Mon, Oct 28, 2019 at 6:47 PM Brian Stansberry < > brian.stansberry(a)redhat.com&gt; wrote: > >> I'd like to add jobs to the automatic testing of PRs to run the >> testsuite using slimmed installations provisioned by Galleon. But, we >> already run a lot of jobs for each PR, enough so that our CI can >> overburdened during busy times around deadlines. So I don't want to just >> add jobs. Instead I also propose to drop the Windows + JDK 8 jobs. >> >> Galleon Testing >> >> During our work on WildFly 18 I added the ability to run portions of the >> WildFly and WildFly Core testsuites with the tests executing against >> slimmed server installations provisioned by Galleon instead of against the >> complete installations normally used. The point of that was to get test >> coverage of those slimmed installations. >> >> Currently we have nightly jobs that run the testsuite this way.[1] As we >> continue to evolve our set of Galleon layers, e.g. adding layers for >> MicroProfile specs, I want to be sure we catch problems before PRs get >> merged. >> >> To run tests locally this way you pass -Dts.layers as an arg to maven. >> >> Dropping Windows JDK 8 Jobs >> >> If we'd drop something in order to free up resources for these Galleon >> jobs, the Windows JDK 8 ones seem a good choice. We'd still run PRs against >> Windows JDK 11, and we'd still run PRs against Linux JDK 8. I can't recall >> any situation where CI found a regression that was specific to the >> Windows + JDK 8 combination. >> >> When CI gets overloaded during rush times, it's the Windows jobs that >> are most problematic. The Windows jobs take longer because the storage >> drives they use are slower. Plus we have fewer Windows agents. The effect >> is during a rush, overall CI for PRs ends up taking hours longer while we >> wait for Windows agents to come free and then run the job. >> >> We'd still run nightly jobs with Windows + JDK 8 so in the off chance >> there's a problem it would get noticed that way. >> > > This seems reasonable to me. One more option to add would be that before > bulk PR's are merged maybe we run Windows JDK8 jobs against those. Though > those jobs are slow so... > > I'm not enthused about more steps in the merge process. :) Of course it's fine if people want to do it and it's good to do when dealing with a PR that has the look of possibly being different, e.g. windows script stuff.

> One other option too is maybe Windows JDK8 jobs just don't run the full > test suite. The main place I could possibly see issues would be if scripts > change since there are some decisions made based on the JVM. Though kicking > off a manual job there isn't a huge deal as it's likely not common. > If you have some ideas on how the ts + ci jobs could be configured to get that result, that would be good. What I did with ts.layers is the root pom has a profile that turns off the default surefire execution. And then in parts of the ts that I wanted to run, there's a profile that turns things on. You have to deal with a few maven modules that have more than the default surefire execution though.

> Somewhat related if we do remove Windows JDK8 jobs I think we should just > remove them from the aggreator job, but keep them under the Pull Request. > The reason for this is if we do want to run a custom one we could use the > PR number from "Changes" tab when running a custom job and it would report > back to the PR. > +1; I'll do it that way. > > >> >> Specifics >> >> For PRs against wildfly/wildfly I'd add a job equivalent to >> https://ci.wildfly.org/viewType.html?buildTypeId=WF_GalleonLinuxJdk11 >> and then drop >> https://ci.wildfly.org/viewType.html?buildTypeId=WildFlyCore_PullRequestW... >> >> For PRs against wildfly/wildfly-core I'd add jobs equivalent to >> https://ci.wildfly.org/viewType.html?buildTypeId=WildFlyCore_GalleonLinux... >> and https://ci.wildfly.org/viewType.html?buildTypeId=WF_GalleonLinuxJdk8 >> and then drop >> https://ci.wildfly.org/viewType.html?buildTypeId=WildFlyCore_PullRequestW... >> >> Any thoughts on this? >> > > No objection from me. > > >> >> >> Best regards, >> Brian >> >> [1] See >> https://ci.wildfly.org/viewType.html?buildTypeId=WF_GalleonLinuxJdk11 >> https://ci.wildfly.org/viewType.html?buildTypeId=WF_GalleonLinuxJdk8 >> https://ci.wildfly.org/viewType.html?buildTypeId=WF_GalleonWindowsJdk11 >> >> https://ci.wildfly.org/viewType.html?buildTypeId=WildFlyCore_GalleonLinux... >> >> https://ci.wildfly.org/viewType.html?buildTypeId=WildFlyCore_GalleonLinux... >> _______________________________________________ >> wildfly-dev mailing list >> wildfly-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/wildfly-dev > > > > -- > James R. Perkins > JBoss by Red Hat > -- Brian Stansberry Manager, Senior Principal Software Engineer Red Hat