I see 3 main use-cases for TCK runs outside WFLY master

 a) pre-checks before final merging to WFLY master - master-ignore way discussed here
 b) checks on big feature(s) development branches
      something like ladybird or RFE development across multiple components
      selected TCK modules can be executed, depends on scope of changes
 c) regular component checks on component master
      early / regression checks on component level that they do not regress
      TCK modules related to the component and layered components should be executed
      I know only few components which run related TCK modules with their master

With proposed changes for quick WF delivery I believe use-cases b) and c) will need to be addressed.

Use-cases b) and c) could be done on component level or via central pipeline.
  component level - prepare automated way to run TCK with custom build - e.g. bash script, docker image
  central pipeline - set of jobs can be prepared and linked via Jenkins pipeline so the end user (or curl request) provides just URL of custom WFLY build zip + list of modules to be executed

Rostislav

----- Original Message -----
> On Tue, Dec 5, 2017 at 4:37 AM, Scott Marlow < smarlow@redhat.com > wrote:
>
>
> It would be great if we could have a branch that includes all of the commits
> that we are considering to merge at a particular time of day, such that we
> would run the TCK against that branch, only once a day.
>
> Can this be done that often? I had in my mind that if we did one of these it
> would amount to stealing one of the regular runs, but perhaps that's not the
> case.
>
> Now, I don't think we'd want to do these anywhere near that often, but it's
> good to know what the limits would be. For example, I could imagine doing 10
> of these over the course of a WF release, but by luck or whatever 3 of them
> come in the same week.
>
> +1 to using a branch. We have a branch like that, master-ignore, that we use
> for batching up PRs to test as a group before merging. I wouldn't want to
> use master-ignore for this, but a differently named branch run the same way
> sounds good.
>
>
>
> If one of the changes cause a TCK failure, none of them get merged
> (investigation follows that to determine which change caused the
> failure(s)), if the test succeeds, we can then merge that batch of changes
> into WildFly master.
>
> We likely would want to avoid running the testing, on days when we haven't
> merged any changes to the WF testing branching.
>
>
> Can the TCK be set up to run based on a check for a change in the sha of the
> head of a branch? So every day at a fixed time it checks the branch, and
> does nothing if there is no change. If we want a run, we force push the
> branch before that time. We have CI jobs that check master-ignore that way,
> except they poll regularly, not just once a day. That works for those as
> they aren't so resource intensive that we worry a lot about limiting how
> often they run.
>
>
> Would that approach help how we merge PRs on master?
>
>
> I think it could be helpful earlier in the release cycle before merging big
> changes, and then perhaps late in the release cycle if we're worried about
> possible regressions.
>
>
>
> Scott
>
> On 12/04/2017 09:33 PM, Stuart Douglas wrote:
>
>
>
>
> On Tue, Dec 5, 2017 at 3:40 AM, Brian Stansberry <
> brian.stansberry@redhat.com <mailto: brian.stansberry@redhat.com >> wrote:
>
> Great. :)
>
> One thing I think we need to do is figure out how to get custom TCK
> runs for PR branches. The TCK is a big part of our test coverage,
> and one way to not "use master as a test bed" is to get a check of a
> branch on the TCK before we merge it.
>
> I know we've gotten TCK runs of ad-hoc branches before, so by
> "figure out" I mean work out how to make that not overly painful,
> come to some sort of consensus on when it's worthwhile, etc.
>
>
> I think if we were going to do this it should probably be something reviewers
> can ask for on specific PR. The TCK uses a *lot* more resources than a
> standard CI run, so we need to make sure we limit it to cases where it is
> required.
>
> Stuart
>
>
> On Fri, Dec 1, 2017 at 10:04 AM, Alessio Soldano
> < asoldano@redhat.com <mailto: asoldano@redhat.com >> wrote:
>
> There you go... PR updated to consume the same api jar now
> released as final.
>
> Cheers
> Alessio
>
> On Fri, Dec 1, 2017 at 3:30 PM, David Lloyd
> < david.lloyd@redhat.com <mailto: david.lloyd@redhat.com >> wrote:
>
> On Thu, Nov 30, 2017 at 5:50 PM, Alessio Soldano
> < asoldano@redhat.com <mailto: asoldano@redhat.com >> wrote:
> > As suggested by Brian, I'd like to draw attention to the discussion on
> > https://github.com/wildfly/wildfly/pull/10604
> < https://github.com/wildfly/wildfly/pull/10604 > .
> > The PR is an upgrade of the webservices stack, including JBossWS, Apache
> > CXF, JAXB-RI and JAXB API. In particular, the JAXB upgrade is for EE8 and
> > better JDK 9 compatibility.
> > Now, due to the upgrade of the JAXB API spec jar, the PR is essentially
> > stalled since 20 days; the new spec is released as an alpha (as it's been
> > tested within JBossWS only) and that does not satisfy a rule that requires
> > any artifact being pulled to be Final.
> > We're talking about a spec jar, we could simply re-tag that as Final,
> > chances are we won't need changes any time soon there anyway, but as Tomaz
> > pointed out, in principle that would be dishonest.
>
> My opinion is that you should go ahead and make a .Final
> tag. In the
> (unlikely?) event that the spec has to be modified for some
> reason, I
> think you could make a 1.0.1.Final tag and call it a "bug fix".
>
> The alternative is to simply wait. I don't think there is
> any middle position.
>
> > While I see the point in requiring that only sufficiently stable upgrades
> > are applied to the codebase, I'm wondering whether, maybe, we're going a
> > bit
> > too far with the rules. Brian wrote on this topic: "how to determine that
> > something is good enough to go in without using master as a test bed" ?
>
> I don't think we are; I agree with the policy as it stands. If you
> look at it in terms of being able to release at any time,
> then it
> follows that everything _must_ be stable.
>
> --
> - DML
>
>
>
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev@lists.jboss.org <mailto: wildfly-dev@lists.jboss.org >
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
> < https://lists.jboss.org/mailman/listinfo/wildfly-dev >
>
>
>
>
> -- Brian Stansberry
> Manager, Senior Principal Software Engineer
> Red Hat
>
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev@lists.jboss.org <mailto: wildfly-dev@lists.jboss.org >
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
> < https://lists.jboss.org/mailman/listinfo/wildfly-dev >
>
>
>
>
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
>
>
>
> --
> Brian Stansberry
> Manager, Senior Principal Software Engineer
> Red Hat
>
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev