On Fri, Nov 1, 2019 at 3:32 PM James Perkins <jperkins(a)redhat.com> wrote:
On Fri, Nov 1, 2019 at 12:08 PM Brian Stansberry <
brian.stansberry(a)redhat.com> wrote:
>
>
> On Thu, Oct 31, 2019 at 7:10 PM James Perkins <jperkins(a)redhat.com>
> wrote:
>
>>
>>
>> On Mon, Oct 28, 2019 at 6:47 PM Brian Stansberry <
>> brian.stansberry(a)redhat.com> wrote:
>>
>>> I'd like to add jobs to the automatic testing of PRs to run the
>>> testsuite using slimmed installations provisioned by Galleon. But, we
>>> already run a lot of jobs for each PR, enough so that our CI can
>>> overburdened during busy times around deadlines. So I don't want to just
>>> add jobs. Instead I also propose to drop the Windows + JDK 8 jobs.
>>>
>>> Galleon Testing
>>>
>>> During our work on WildFly 18 I added the ability to run portions of
>>> the WildFly and WildFly Core testsuites with the tests executing against
>>> slimmed server installations provisioned by Galleon instead of against the
>>> complete installations normally used. The point of that was to get test
>>> coverage of those slimmed installations.
>>>
>>> Currently we have nightly jobs that run the testsuite this way.[1] As
>>> we continue to evolve our set of Galleon layers, e.g. adding layers for
>>> MicroProfile specs, I want to be sure we catch problems before PRs get
>>> merged.
>>>
>>> To run tests locally this way you pass -Dts.layers as an arg to maven.
>>>
>>> Dropping Windows JDK 8 Jobs
>>>
>>> If we'd drop something in order to free up resources for these Galleon
>>> jobs, the Windows JDK 8 ones seem a good choice. We'd still run PRs
against
>>> Windows JDK 11, and we'd still run PRs against Linux JDK 8. I can't
recall
>>> any situation where CI found a regression that was specific to the
>>> Windows + JDK 8 combination.
>>>
>>> When CI gets overloaded during rush times, it's the Windows jobs that
>>> are most problematic. The Windows jobs take longer because the storage
>>> drives they use are slower. Plus we have fewer Windows agents. The effect
>>> is during a rush, overall CI for PRs ends up taking hours longer while we
>>> wait for Windows agents to come free and then run the job.
>>>
>>> We'd still run nightly jobs with Windows + JDK 8 so in the off chance
>>> there's a problem it would get noticed that way.
>>>
>>
>> This seems reasonable to me. One more option to add would be that before
>> bulk PR's are merged maybe we run Windows JDK8 jobs against those. Though
>> those jobs are slow so...
>>
>> I'm not enthused about more steps in the merge process. :) Of course
> it's fine if people want to do it and it's good to do when dealing with a
> PR that has the look of possibly being different, e.g. windows script stuff.
>
Yeah I almost removed this line before sending, but figured it didn't hurt
to keep it open for discussion at least.
>
>
>> One other option too is maybe Windows JDK8 jobs just don't run the full
>> test suite. The main place I could possibly see issues would be if scripts
>> change since there are some decisions made based on the JVM. Though kicking
>> off a manual job there isn't a huge deal as it's likely not common.
>>
>
> If you have some ideas on how the ts + ci jobs could be configured to get
> that result, that would be good.
>
> What I did with ts.layers is the root pom has a profile that turns off
> the default surefire execution. And then in parts of the ts that I wanted
> to run, there's a profile that turns things on. You have to deal with a few
> maven modules that have more than the default surefire execution though.
>
My initial thought was just to run without -DallTests, but I suppose using
a more fine grained approach with profiles could work as well.
Or a mix. It's true that without -DallTests the bulk of the ts doesn't run,
which probably eliminates the maven modules I mentioned that have more than
the default surefire execution.
The -Dts.layers profile doesn't just turn off the testsuite/* stuff. It
turns off surefire execution for all the other maven modules.
Is it really just testsuite/scripts in WildFly Core that tests scripts?
>
>
>> Somewhat related if we do remove Windows JDK8 jobs I think we should
>> just remove them from the aggreator job, but keep them under the Pull
>> Request. The reason for this is if we do want to run a custom one we could
>> use the PR number from "Changes" tab when running a custom job and it
would
>> report back to the PR.
>>
>
> +1; I'll do it that way.
>
>>
>>
>>>
>>> Specifics
>>>
>>> For PRs against wildfly/wildfly I'd add a job equivalent to
>>>
https://ci.wildfly.org/viewType.html?buildTypeId=WF_GalleonLinuxJdk11
>>> and then drop
>>>
https://ci.wildfly.org/viewType.html?buildTypeId=WildFlyCore_PullRequestW...
>>>
>>> For PRs against wildfly/wildfly-core I'd add jobs equivalent to
>>>
https://ci.wildfly.org/viewType.html?buildTypeId=WildFlyCore_GalleonLinux...
>>> and
>>>
https://ci.wildfly.org/viewType.html?buildTypeId=WF_GalleonLinuxJdk8
>>> and then drop
>>>
https://ci.wildfly.org/viewType.html?buildTypeId=WildFlyCore_PullRequestW...
>>>
>>> Any thoughts on this?
>>>
>>
>> No objection from me.
>>
>>
>>>
>>>
>>> Best regards,
>>> Brian
>>>
>>> [1] See
>>>
https://ci.wildfly.org/viewType.html?buildTypeId=WF_GalleonLinuxJdk11
>>>
https://ci.wildfly.org/viewType.html?buildTypeId=WF_GalleonLinuxJdk8
>>>
https://ci.wildfly.org/viewType.html?buildTypeId=WF_GalleonWindowsJdk11
>>>
>>>
https://ci.wildfly.org/viewType.html?buildTypeId=WildFlyCore_GalleonLinux...
>>>
>>>
https://ci.wildfly.org/viewType.html?buildTypeId=WildFlyCore_GalleonLinux...
>>> _______________________________________________
>>> wildfly-dev mailing list
>>> wildfly-dev(a)lists.jboss.org
>>>
https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>
>>
>>
>> --
>> James R. Perkins
>> JBoss by Red Hat
>>
>
>
> --
> Brian Stansberry
> Manager, Senior Principal Software Engineer
> Red Hat
>
--
James R. Perkins
JBoss by Red Hat
--
Brian Stansberry
Manager, Senior Principal Software Engineer
Red Hat