[wildfly-dev] Add pull request CI coverage of Galleon layers, drop Windows JDK 8

Fri Nov 1 18:57:52 EDT 2019

On Fri, Nov 1, 2019 at 3:04 PM Brian Stansberry <brian.stansberry at redhat.com>
wrote:

>
>
> On Fri, Nov 1, 2019 at 3:32 PM James Perkins <jperkins at redhat.com> wrote:
>
>>
>>
>> On Fri, Nov 1, 2019 at 12:08 PM Brian Stansberry <
>> brian.stansberry at redhat.com> wrote:
>>
>>>
>>>
>>> On Thu, Oct 31, 2019 at 7:10 PM James Perkins <jperkins at redhat.com>
>>> wrote:
>>>
>>>>
>>>>
>>>> On Mon, Oct 28, 2019 at 6:47 PM Brian Stansberry <
>>>> brian.stansberry at redhat.com> wrote:
>>>>
>>>>> I'd like to add jobs to the automatic testing of PRs to run the
>>>>> testsuite using slimmed installations provisioned by Galleon. But, we
>>>>> already run a lot of jobs for each PR, enough so that our CI can
>>>>> overburdened during busy times around deadlines. So I don't want to just
>>>>> add jobs. Instead I also propose to drop the Windows + JDK 8 jobs.
>>>>>
>>>>> Galleon Testing
>>>>>
>>>>> During our work on WildFly 18 I added the ability to run portions of
>>>>> the WildFly and WildFly Core testsuites with the tests executing against
>>>>> slimmed server installations provisioned by Galleon instead of against the
>>>>> complete installations normally used. The point of that was to get test
>>>>> coverage of those slimmed installations.
>>>>>
>>>>> Currently we have nightly jobs that run the testsuite this way.[1] As
>>>>> we continue to evolve our set of Galleon layers, e.g. adding layers for
>>>>> MicroProfile specs, I want to be sure we catch problems before PRs get
>>>>> merged.
>>>>>
>>>>> To run tests locally this way you pass -Dts.layers as an arg to maven.
>>>>>
>>>>> Dropping Windows JDK 8 Jobs
>>>>>
>>>>> If we'd drop something in order to free up resources for these Galleon
>>>>> jobs, the Windows JDK 8 ones seem a good choice. We'd still run PRs against
>>>>> Windows JDK 11, and we'd still run PRs against Linux JDK 8. I can't recall
>>>>> any situation where CI found a regression that was specific to the
>>>>> Windows + JDK 8 combination.
>>>>>
>>>>> When CI gets overloaded during rush times, it's the Windows jobs that
>>>>> are most problematic. The Windows jobs take longer because the storage
>>>>> drives they use are slower. Plus we have fewer Windows agents. The effect
>>>>> is during a rush, overall CI for PRs ends up taking hours longer while we
>>>>> wait for Windows agents to come free and then run the job.
>>>>>
>>>>> We'd still run nightly jobs with Windows + JDK 8 so in the off chance
>>>>> there's a problem it would get noticed that way.
>>>>>
>>>>
>>>> This seems reasonable to me. One more option to add would be that
>>>> before bulk PR's are merged maybe we run Windows JDK8 jobs against those.
>>>> Though those jobs are slow so...
>>>>
>>>> I'm not enthused about more steps in the merge process. :) Of course
>>> it's fine if people want to do it and it's good to do when dealing with a
>>> PR that has the look of possibly being different, e.g. windows script stuff.
>>>
>>
>> Yeah I almost removed this line before sending, but figured it didn't
>> hurt to keep it open for discussion at least.
>>
>>
>>>
>>>
>>>> One other option too is maybe Windows JDK8 jobs just don't run the full
>>>> test suite. The main place I could possibly see issues would be if scripts
>>>> change since there are some decisions made based on the JVM. Though kicking
>>>> off a manual job there isn't a huge deal as it's likely not common.
>>>>
>>>
>>> If you have some ideas on how the ts + ci jobs could be configured to
>>> get that result, that would be good.
>>>
>>> What I did with ts.layers is the root pom has a profile that turns off
>>> the default surefire execution. And then in parts of the ts that I wanted
>>> to run, there's a profile that turns things on. You have to deal with a few
>>> maven modules that have more than the default surefire execution though.
>>>
>>
>> My initial thought was just to run without -DallTests, but I suppose
>> using a more fine grained approach with profiles could work as well.
>>
>
> Or a mix. It's true that without -DallTests the bulk of the ts doesn't
> run, which probably eliminates the maven modules I mentioned that have more
> than the default surefire execution.
>
> The -Dts.layers profile doesn't just turn off the testsuite/* stuff. It
> turns off surefire execution for all the other maven modules.
>
> Is it really just testsuite/scripts in WildFly Core that tests scripts?
>

Yeah that's all it is. It honestly might not be a big deal as the scripts
don't change that often. It was just the one place I had of concern.

>
>
>>
>>>
>>>
>>>> Somewhat related if we do remove Windows JDK8 jobs I think we should
>>>> just remove them from the aggreator job, but keep them under the Pull
>>>> Request. The reason for this is if we do want to run a custom one we could
>>>> use the PR number from "Changes" tab when running a custom job and it would
>>>> report back to the PR.
>>>>
>>>
>>> +1; I'll do it that way.
>>>
>>>>
>>>>
>>>>>
>>>>> Specifics
>>>>>
>>>>> For PRs against wildfly/wildfly I'd add a job equivalent to
>>>>> https://ci.wildfly.org/viewType.html?buildTypeId=WF_GalleonLinuxJdk11
>>>>> and then drop
>>>>> https://ci.wildfly.org/viewType.html?buildTypeId=WildFlyCore_PullRequestWindows
>>>>>
>>>>> For PRs against wildfly/wildfly-core I'd add jobs equivalent to
>>>>> https://ci.wildfly.org/viewType.html?buildTypeId=WildFlyCore_GalleonLinuxJdk8
>>>>> and
>>>>> https://ci.wildfly.org/viewType.html?buildTypeId=WF_GalleonLinuxJdk8
>>>>> and then drop
>>>>> https://ci.wildfly.org/viewType.html?buildTypeId=WildFlyCore_PullRequestWindows
>>>>>
>>>>> Any thoughts on this?
>>>>>
>>>>
>>>> No objection from me.
>>>>
>>>>
>>>>>
>>>>>
>>>>> Best regards,
>>>>> Brian
>>>>>
>>>>> [1] See
>>>>> https://ci.wildfly.org/viewType.html?buildTypeId=WF_GalleonLinuxJdk11
>>>>> https://ci.wildfly.org/viewType.html?buildTypeId=WF_GalleonLinuxJdk8
>>>>> https://ci.wildfly.org/viewType.html?buildTypeId=WF_GalleonWindowsJdk11
>>>>>
>>>>> https://ci.wildfly.org/viewType.html?buildTypeId=WildFlyCore_GalleonLinuxJdk8
>>>>>
>>>>> https://ci.wildfly.org/viewType.html?buildTypeId=WildFlyCore_GalleonLinuxJdk11
>>>>> _______________________________________________
>>>>> wildfly-dev mailing list
>>>>> wildfly-dev at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>>>
>>>>
>>>>
>>>> --
>>>> James R. Perkins
>>>> JBoss by Red Hat
>>>>
>>>
>>>
>>> --
>>> Brian Stansberry
>>> Manager, Senior Principal Software Engineer
>>> Red Hat
>>>
>>
>>
>> --
>> James R. Perkins
>> JBoss by Red Hat
>>
>
>
> --
> Brian Stansberry
> Manager, Senior Principal Software Engineer
> Red Hat
>

-- 
James R. Perkins
JBoss by Red Hat
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/wildfly-dev/attachments/20191101/2c42dd1c/attachment.html 