[wildfly-dev] Policies for merging PRs on master

Rostislav Svoboda rsvoboda at redhat.com
Thu Dec 7 03:15:38 EST 2017 

> Excellent suggestions Rostislav!  There are some trade offs due to the
> amount of time needed for each test run.  Whatever the implementation, it

If there is prepared script or docker image TCK execution could be offloaded from your TCK Jenkins to components (component Central CI instance or their bare-metal machine/notebook)

> would be good to have a prioritized way of running the tests that
> accomplish a-c.

On script / job / docker level:
 - instead of list of modules to be executed you would need two lists - list of prioritized modules and list regular modules + add delay before triggering regular modules
 - add weigh to the "list"

Jenkins itself can prioritize some jobs, we used it to prioritize acceptance testing jobs in MW Jenkins to have early results.
This way you could have priority on regular WFLY master / master-ignore jobs (or the other way) in your TCK Jenkins.

Rostislav

> Scott
> 
> On Wed, Dec 6, 2017 at 7:31 AM, Rostislav Svoboda <rsvoboda at redhat.com>
> wrote:
> 
> > I see 3 main use-cases for TCK runs outside WFLY master
> >
> >  a) pre-checks before final merging to WFLY master - master-ignore way
> > discussed here
> >  b) checks on big feature(s) development branches
> >       something like ladybird or RFE development across multiple components
> >       selected TCK modules can be executed, depends on scope of changes
> >  c) regular component checks on component master
> >       early / regression checks on component level that they do not regress
> >       TCK modules related to the component and layered components should
> > be executed
> >       I know only few components which run related TCK modules with their
> > master
> >
> > With proposed changes for quick WF delivery I believe use-cases b) and c)
> > will need to be addressed.
> >
> > Use-cases b) and c) could be done on component level or via central
> > pipeline.
> >   component level - prepare automated way to run TCK with custom build -
> > e.g. bash script, docker image
> >   central pipeline - set of jobs can be prepared and linked via Jenkins
> > pipeline so the end user (or curl request) provides just URL of custom WFLY
> > build zip + list of modules to be executed
> >
> > Rostislav
> >
> > ----- Original Message -----
> > > On Tue, Dec 5, 2017 at 4:37 AM, Scott Marlow < smarlow at redhat.com >
> > wrote:
> > >
> > >
> > > It would be great if we could have a branch that includes all of the
> > commits
> > > that we are considering to merge at a particular time of day, such that
> > we
> > > would run the TCK against that branch, only once a day.
> > >
> > > Can this be done that often? I had in my mind that if we did one of
> > these it
> > > would amount to stealing one of the regular runs, but perhaps that's not
> > the
> > > case.
> > >
> > > Now, I don't think we'd want to do these anywhere near that often, but
> > it's
> > > good to know what the limits would be. For example, I could imagine
> > doing 10
> > > of these over the course of a WF release, but by luck or whatever 3 of
> > them
> > > come in the same week.
> > >
> > > +1 to using a branch. We have a branch like that, master-ignore, that we
> > use
> > > for batching up PRs to test as a group before merging. I wouldn't want to
> > > use master-ignore for this, but a differently named branch run the same
> > way
> > > sounds good.
> > >
> > >
> > >
> > > If one of the changes cause a TCK failure, none of them get merged
> > > (investigation follows that to determine which change caused the
> > > failure(s)), if the test succeeds, we can then merge that batch of
> > changes
> > > into WildFly master.
> > >
> > > We likely would want to avoid running the testing, on days when we
> > haven't
> > > merged any changes to the WF testing branching.
> > >
> > >
> > > Can the TCK be set up to run based on a check for a change in the sha of
> > the
> > > head of a branch? So every day at a fixed time it checks the branch, and
> > > does nothing if there is no change. If we want a run, we force push the
> > > branch before that time. We have CI jobs that check master-ignore that
> > way,
> > > except they poll regularly, not just once a day. That works for those as
> > > they aren't so resource intensive that we worry a lot about limiting how
> > > often they run.
> > >
> > >
> > > Would that approach help how we merge PRs on master?
> > >
> > >
> > > I think it could be helpful earlier in the release cycle before merging
> > big
> > > changes, and then perhaps late in the release cycle if we're worried
> > about
> > > possible regressions.
> > >
> > >
> > >
> > > Scott
> > >
> > > On 12/04/2017 09:33 PM, Stuart Douglas wrote:
> > >
> > >
> > >
> > >
> > > On Tue, Dec 5, 2017 at 3:40 AM, Brian Stansberry <
> > > brian.stansberry at redhat.com <mailto: brian.stansberry at redhat.com >>
> > wrote:
> > >
> > > Great. :)
> > >
> > > One thing I think we need to do is figure out how to get custom TCK
> > > runs for PR branches. The TCK is a big part of our test coverage,
> > > and one way to not "use master as a test bed" is to get a check of a
> > > branch on the TCK before we merge it.
> > >
> > > I know we've gotten TCK runs of ad-hoc branches before, so by
> > > "figure out" I mean work out how to make that not overly painful,
> > > come to some sort of consensus on when it's worthwhile, etc.
> > >
> > >
> > > I think if we were going to do this it should probably be something
> > reviewers
> > > can ask for on specific PR. The TCK uses a *lot* more resources than a
> > > standard CI run, so we need to make sure we limit it to cases where it is
> > > required.
> > >
> > > Stuart
> > >
> > >
> > > On Fri, Dec 1, 2017 at 10:04 AM, Alessio Soldano
> > > < asoldano at redhat.com <mailto: asoldano at redhat.com >> wrote:
> > >
> > > There you go... PR updated to consume the same api jar now
> > > released as final.
> > >
> > > Cheers
> > > Alessio
> > >
> > > On Fri, Dec 1, 2017 at 3:30 PM, David Lloyd
> > > < david.lloyd at redhat.com <mailto: david.lloyd at redhat.com >> wrote:
> > >
> > > On Thu, Nov 30, 2017 at 5:50 PM, Alessio Soldano
> > > < asoldano at redhat.com <mailto: asoldano at redhat.com >> wrote:
> > > > As suggested by Brian, I'd like to draw attention to the discussion on
> > > > https://github.com/wildfly/wildfly/pull/10604
> > > < https://github.com/wildfly/wildfly/pull/10604 > .
> > > > The PR is an upgrade of the webservices stack, including JBossWS,
> > Apache
> > > > CXF, JAXB-RI and JAXB API. In particular, the JAXB upgrade is for EE8
> > and
> > > > better JDK 9 compatibility.
> > > > Now, due to the upgrade of the JAXB API spec jar, the PR is essentially
> > > > stalled since 20 days; the new spec is released as an alpha (as it's
> > been
> > > > tested within JBossWS only) and that does not satisfy a rule that
> > requires
> > > > any artifact being pulled to be Final.
> > > > We're talking about a spec jar, we could simply re-tag that as Final,
> > > > chances are we won't need changes any time soon there anyway, but as
> > Tomaz
> > > > pointed out, in principle that would be dishonest.
> > >
> > > My opinion is that you should go ahead and make a .Final
> > > tag. In the
> > > (unlikely?) event that the spec has to be modified for some
> > > reason, I
> > > think you could make a 1.0.1.Final tag and call it a "bug fix".
> > >
> > > The alternative is to simply wait. I don't think there is
> > > any middle position.
> > >
> > > > While I see the point in requiring that only sufficiently stable
> > upgrades
> > > > are applied to the codebase, I'm wondering whether, maybe, we're going
> > a
> > > > bit
> > > > too far with the rules. Brian wrote on this topic: "how to determine
> > that
> > > > something is good enough to go in without using master as a test bed" ?
> > >
> > > I don't think we are; I agree with the policy as it stands. If you
> > > look at it in terms of being able to release at any time,
> > > then it
> > > follows that everything _must_ be stable.
> > >
> > > --
> > > - DML
> > >
> > >
> > >
> > > _______________________________________________
> > > wildfly-dev mailing list
> > > wildfly-dev at lists.jboss.org <mailto: wildfly-dev at lists.jboss.org >
> > > https://lists.jboss.org/mailman/listinfo/wildfly-dev
> > > < https://lists.jboss.org/mailman/listinfo/wildfly-dev >
> > >
> > >
> > >
> > >
> > > -- Brian Stansberry
> > > Manager, Senior Principal Software Engineer
> > > Red Hat
> > >
> > > _______________________________________________
> > > wildfly-dev mailing list
> > > wildfly-dev at lists.jboss.org <mailto: wildfly-dev at lists.jboss.org >
> > > https://lists.jboss.org/mailman/listinfo/wildfly-dev
> > > < https://lists.jboss.org/mailman/listinfo/wildfly-dev >
> > >
> > >
> > >
> > >
> > > _______________________________________________
> > > wildfly-dev mailing list
> > > wildfly-dev at lists.jboss.org
> > > https://lists.jboss.org/mailman/listinfo/wildfly-dev
> > >
> > >
> > >
> > >
> > > --
> > > Brian Stansberry
> > > Manager, Senior Principal Software Engineer
> > > Red Hat
> > >
> > > _______________________________________________
> > > wildfly-dev mailing list
> > > wildfly-dev at lists.jboss.org
> > > https://lists.jboss.org/mailman/listinfo/wildfly-dev
> >
>

More information about the wildfly-dev mailing list