[wildfly-dev] Add pull request CI coverage of Galleon layers, drop Windows JDK 8

Fri Nov 1 18:03:56 EDT 2019

On Fri, Nov 1, 2019 at 3:32 PM James Perkins <jperkins at redhat.com> wrote:

>
>
> On Fri, Nov 1, 2019 at 12:08 PM Brian Stansberry <
> brian.stansberry at redhat.com> wrote:
>
>>
>>
>> On Thu, Oct 31, 2019 at 7:10 PM James Perkins <jperkins at redhat.com>
>> wrote:
>>
>>>
>>>
>>> On Mon, Oct 28, 2019 at 6:47 PM Brian Stansberry <
>>> brian.stansberry at redhat.com> wrote:
>>>
>>>> I'd like to add jobs to the automatic testing of PRs to run the
>>>> testsuite using slimmed installations provisioned by Galleon. But, we
>>>> already run a lot of jobs for each PR, enough so that our CI can
>>>> overburdened during busy times around deadlines. So I don't want to just
>>>> add jobs. Instead I also propose to drop the Windows + JDK 8 jobs.
>>>>
>>>> Galleon Testing
>>>>
>>>> During our work on WildFly 18 I added the ability to run portions of
>>>> the WildFly and WildFly Core testsuites with the tests executing against
>>>> slimmed server installations provisioned by Galleon instead of against the
>>>> complete installations normally used. The point of that was to get test
>>>> coverage of those slimmed installations.
>>>>
>>>> Currently we have nightly jobs that run the testsuite this way.[1] As
>>>> we continue to evolve our set of Galleon layers, e.g. adding layers for
>>>> MicroProfile specs, I want to be sure we catch problems before PRs get
>>>> merged.
>>>>
>>>> To run tests locally this way you pass -Dts.layers as an arg to maven.
>>>>
>>>> Dropping Windows JDK 8 Jobs
>>>>
>>>> If we'd drop something in order to free up resources for these Galleon
>>>> jobs, the Windows JDK 8 ones seem a good choice. We'd still run PRs against
>>>> Windows JDK 11, and we'd still run PRs against Linux JDK 8. I can't recall
>>>> any situation where CI found a regression that was specific to the
>>>> Windows + JDK 8 combination.
>>>>
>>>> When CI gets overloaded during rush times, it's the Windows jobs that
>>>> are most problematic. The Windows jobs take longer because the storage
>>>> drives they use are slower. Plus we have fewer Windows agents. The effect
>>>> is during a rush, overall CI for PRs ends up taking hours longer while we
>>>> wait for Windows agents to come free and then run the job.
>>>>
>>>> We'd still run nightly jobs with Windows + JDK 8 so in the off chance
>>>> there's a problem it would get noticed that way.
>>>>
>>>
>>> This seems reasonable to me. One more option to add would be that before
>>> bulk PR's are merged maybe we run Windows JDK8 jobs against those. Though
>>> those jobs are slow so...
>>>
>>> I'm not enthused about more steps in the merge process. :) Of course
>> it's fine if people want to do it and it's good to do when dealing with a
>> PR that has the look of possibly being different, e.g. windows script stuff.
>>
>
> Yeah I almost removed this line before sending, but figured it didn't hurt
> to keep it open for discussion at least.
>
>
>>
>>
>>> One other option too is maybe Windows JDK8 jobs just don't run the full
>>> test suite. The main place I could possibly see issues would be if scripts
>>> change since there are some decisions made based on the JVM. Though kicking
>>> off a manual job there isn't a huge deal as it's likely not common.
>>>
>>
>> If you have some ideas on how the ts + ci jobs could be configured to get
>> that result, that would be good.
>>
>> What I did with ts.layers is the root pom has a profile that turns off
>> the default surefire execution. And then in parts of the ts that I wanted
>> to run, there's a profile that turns things on. You have to deal with a few
>> maven modules that have more than the default surefire execution though.
>>
>
> My initial thought was just to run without -DallTests, but I suppose using
> a more fine grained approach with profiles could work as well.
>

Or a mix. It's true that without -DallTests the bulk of the ts doesn't run,
which probably eliminates the maven modules I mentioned that have more than
the default surefire execution.

The -Dts.layers profile doesn't just turn off the testsuite/* stuff. It
turns off surefire execution for all the other maven modules.

Is it really just testsuite/scripts in WildFly Core that tests scripts?

>
>>
>>
>>> Somewhat related if we do remove Windows JDK8 jobs I think we should
>>> just remove them from the aggreator job, but keep them under the Pull
>>> Request. The reason for this is if we do want to run a custom one we could
>>> use the PR number from "Changes" tab when running a custom job and it would
>>> report back to the PR.
>>>
>>
>> +1; I'll do it that way.
>>
>>>
>>>
>>>>
>>>> Specifics
>>>>
>>>> For PRs against wildfly/wildfly I'd add a job equivalent to
>>>> https://ci.wildfly.org/viewType.html?buildTypeId=WF_GalleonLinuxJdk11
>>>> and then drop
>>>> https://ci.wildfly.org/viewType.html?buildTypeId=WildFlyCore_PullRequestWindows
>>>>
>>>> For PRs against wildfly/wildfly-core I'd add jobs equivalent to
>>>> https://ci.wildfly.org/viewType.html?buildTypeId=WildFlyCore_GalleonLinuxJdk8
>>>> and
>>>> https://ci.wildfly.org/viewType.html?buildTypeId=WF_GalleonLinuxJdk8
>>>> and then drop
>>>> https://ci.wildfly.org/viewType.html?buildTypeId=WildFlyCore_PullRequestWindows
>>>>
>>>> Any thoughts on this?
>>>>
>>>
>>> No objection from me.
>>>
>>>
>>>>
>>>>
>>>> Best regards,
>>>> Brian
>>>>
>>>> [1] See
>>>> https://ci.wildfly.org/viewType.html?buildTypeId=WF_GalleonLinuxJdk11
>>>> https://ci.wildfly.org/viewType.html?buildTypeId=WF_GalleonLinuxJdk8
>>>> https://ci.wildfly.org/viewType.html?buildTypeId=WF_GalleonWindowsJdk11
>>>>
>>>> https://ci.wildfly.org/viewType.html?buildTypeId=WildFlyCore_GalleonLinuxJdk8
>>>>
>>>> https://ci.wildfly.org/viewType.html?buildTypeId=WildFlyCore_GalleonLinuxJdk11
>>>> _______________________________________________
>>>> wildfly-dev mailing list
>>>> wildfly-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>>
>>>
>>>
>>> --
>>> James R. Perkins
>>> JBoss by Red Hat
>>>
>>
>>
>> --
>> Brian Stansberry
>> Manager, Senior Principal Software Engineer
>> Red Hat
>>
>
>
> --
> James R. Perkins
> JBoss by Red Hat
>

-- 
Brian Stansberry
Manager, Senior Principal Software Engineer
Red Hat
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/wildfly-dev/attachments/20191101/17468968/attachment-0001.html 