[wildfly-dev] Add pull request CI coverage of Galleon layers, drop Windows JDK 8

Fri Nov 1 16:31:48 EDT 2019

On Fri, Nov 1, 2019 at 12:08 PM Brian Stansberry <
brian.stansberry at redhat.com> wrote:

>
>
> On Thu, Oct 31, 2019 at 7:10 PM James Perkins <jperkins at redhat.com> wrote:
>
>>
>>
>> On Mon, Oct 28, 2019 at 6:47 PM Brian Stansberry <
>> brian.stansberry at redhat.com> wrote:
>>
>>> I'd like to add jobs to the automatic testing of PRs to run the
>>> testsuite using slimmed installations provisioned by Galleon. But, we
>>> already run a lot of jobs for each PR, enough so that our CI can
>>> overburdened during busy times around deadlines. So I don't want to just
>>> add jobs. Instead I also propose to drop the Windows + JDK 8 jobs.
>>>
>>> Galleon Testing
>>>
>>> During our work on WildFly 18 I added the ability to run portions of the
>>> WildFly and WildFly Core testsuites with the tests executing against
>>> slimmed server installations provisioned by Galleon instead of against the
>>> complete installations normally used. The point of that was to get test
>>> coverage of those slimmed installations.
>>>
>>> Currently we have nightly jobs that run the testsuite this way.[1] As we
>>> continue to evolve our set of Galleon layers, e.g. adding layers for
>>> MicroProfile specs, I want to be sure we catch problems before PRs get
>>> merged.
>>>
>>> To run tests locally this way you pass -Dts.layers as an arg to maven.
>>>
>>> Dropping Windows JDK 8 Jobs
>>>
>>> If we'd drop something in order to free up resources for these Galleon
>>> jobs, the Windows JDK 8 ones seem a good choice. We'd still run PRs against
>>> Windows JDK 11, and we'd still run PRs against Linux JDK 8. I can't recall
>>> any situation where CI found a regression that was specific to the
>>> Windows + JDK 8 combination.
>>>
>>> When CI gets overloaded during rush times, it's the Windows jobs that
>>> are most problematic. The Windows jobs take longer because the storage
>>> drives they use are slower. Plus we have fewer Windows agents. The effect
>>> is during a rush, overall CI for PRs ends up taking hours longer while we
>>> wait for Windows agents to come free and then run the job.
>>>
>>> We'd still run nightly jobs with Windows + JDK 8 so in the off chance
>>> there's a problem it would get noticed that way.
>>>
>>
>> This seems reasonable to me. One more option to add would be that before
>> bulk PR's are merged maybe we run Windows JDK8 jobs against those. Though
>> those jobs are slow so...
>>
>> I'm not enthused about more steps in the merge process. :) Of course it's
> fine if people want to do it and it's good to do when dealing with a PR
> that has the look of possibly being different, e.g. windows script stuff.
>

Yeah I almost removed this line before sending, but figured it didn't hurt
to keep it open for discussion at least.

>
>
>> One other option too is maybe Windows JDK8 jobs just don't run the full
>> test suite. The main place I could possibly see issues would be if scripts
>> change since there are some decisions made based on the JVM. Though kicking
>> off a manual job there isn't a huge deal as it's likely not common.
>>
>
> If you have some ideas on how the ts + ci jobs could be configured to get
> that result, that would be good.
>
> What I did with ts.layers is the root pom has a profile that turns off the
> default surefire execution. And then in parts of the ts that I wanted to
> run, there's a profile that turns things on. You have to deal with a few
> maven modules that have more than the default surefire execution though.
>

My initial thought was just to run without -DallTests, but I suppose using
a more fine grained approach with profiles could work as well.

>
>
>> Somewhat related if we do remove Windows JDK8 jobs I think we should just
>> remove them from the aggreator job, but keep them under the Pull Request.
>> The reason for this is if we do want to run a custom one we could use the
>> PR number from "Changes" tab when running a custom job and it would report
>> back to the PR.
>>
>
> +1; I'll do it that way.
>
>>
>>
>>>
>>> Specifics
>>>
>>> For PRs against wildfly/wildfly I'd add a job equivalent to
>>> https://ci.wildfly.org/viewType.html?buildTypeId=WF_GalleonLinuxJdk11
>>> and then drop
>>> https://ci.wildfly.org/viewType.html?buildTypeId=WildFlyCore_PullRequestWindows
>>>
>>> For PRs against wildfly/wildfly-core I'd add jobs equivalent to
>>> https://ci.wildfly.org/viewType.html?buildTypeId=WildFlyCore_GalleonLinuxJdk8
>>> and https://ci.wildfly.org/viewType.html?buildTypeId=WF_GalleonLinuxJdk8
>>> and then drop
>>> https://ci.wildfly.org/viewType.html?buildTypeId=WildFlyCore_PullRequestWindows
>>>
>>> Any thoughts on this?
>>>
>>
>> No objection from me.
>>
>>
>>>
>>>
>>> Best regards,
>>> Brian
>>>
>>> [1] See
>>> https://ci.wildfly.org/viewType.html?buildTypeId=WF_GalleonLinuxJdk11
>>> https://ci.wildfly.org/viewType.html?buildTypeId=WF_GalleonLinuxJdk8
>>> https://ci.wildfly.org/viewType.html?buildTypeId=WF_GalleonWindowsJdk11
>>>
>>> https://ci.wildfly.org/viewType.html?buildTypeId=WildFlyCore_GalleonLinuxJdk8
>>>
>>> https://ci.wildfly.org/viewType.html?buildTypeId=WildFlyCore_GalleonLinuxJdk11
>>> _______________________________________________
>>> wildfly-dev mailing list
>>> wildfly-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>
>>
>>
>> --
>> James R. Perkins
>> JBoss by Red Hat
>>
>
>
> --
> Brian Stansberry
> Manager, Senior Principal Software Engineer
> Red Hat
>

-- 
James R. Perkins
JBoss by Red Hat
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/wildfly-dev/attachments/20191101/75233f0f/attachment.html 