<div dir="ltr">Excellent suggestions Rostislav! There are some trade offs due to the amount of time needed for each test run. Whatever the implementation, it would be good to have a prioritized way of running the tests that accomplish a-c.<div><br></div><div>Scott <div><br></div><div><div><div class="gmail_extra"><div class="gmail_quote">On Wed, Dec 6, 2017 at 7:31 AM, Rostislav Svoboda <span dir="ltr"><<a href="mailto:rsvoboda@redhat.com" target="_blank">rsvoboda@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I see 3 main use-cases for TCK runs outside WFLY master<br>
<br>
a) pre-checks before final merging to WFLY master - master-ignore way discussed here<br>
b) checks on big feature(s) development branches<br>
something like ladybird or RFE development across multiple components<br>
selected TCK modules can be executed, depends on scope of changes<br>
c) regular component checks on component master<br>
early / regression checks on component level that they do not regress<br>
TCK modules related to the component and layered components should be executed<br>
I know only few components which run related TCK modules with their master<br>
<br>
With proposed changes for quick WF delivery I believe use-cases b) and c) will need to be addressed.<br>
<br>
Use-cases b) and c) could be done on component level or via central pipeline.<br>
component level - prepare automated way to run TCK with custom build - e.g. bash script, docker image<br>
central pipeline - set of jobs can be prepared and linked via Jenkins pipeline so the end user (or curl request) provides just URL of custom WFLY build zip + list of modules to be executed<br>
<span class="HOEnZb"><font color="#888888"><br>
Rostislav<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
----- Original Message -----<br>
> On Tue, Dec 5, 2017 at 4:37 AM, Scott Marlow < <a href="mailto:smarlow@redhat.com">smarlow@redhat.com</a> > wrote:<br>
><br>
><br>
> It would be great if we could have a branch that includes all of the commits<br>
> that we are considering to merge at a particular time of day, such that we<br>
> would run the TCK against that branch, only once a day.<br>
><br>
> Can this be done that often? I had in my mind that if we did one of these it<br>
> would amount to stealing one of the regular runs, but perhaps that's not the<br>
> case.<br>
><br>
> Now, I don't think we'd want to do these anywhere near that often, but it's<br>
> good to know what the limits would be. For example, I could imagine doing 10<br>
> of these over the course of a WF release, but by luck or whatever 3 of them<br>
> come in the same week.<br>
><br>
> +1 to using a branch. We have a branch like that, master-ignore, that we use<br>
> for batching up PRs to test as a group before merging. I wouldn't want to<br>
> use master-ignore for this, but a differently named branch run the same way<br>
> sounds good.<br>
><br>
><br>
><br>
> If one of the changes cause a TCK failure, none of them get merged<br>
> (investigation follows that to determine which change caused the<br>
> failure(s)), if the test succeeds, we can then merge that batch of changes<br>
> into WildFly master.<br>
><br>
> We likely would want to avoid running the testing, on days when we haven't<br>
> merged any changes to the WF testing branching.<br>
><br>
><br>
> Can the TCK be set up to run based on a check for a change in the sha of the<br>
> head of a branch? So every day at a fixed time it checks the branch, and<br>
> does nothing if there is no change. If we want a run, we force push the<br>
> branch before that time. We have CI jobs that check master-ignore that way,<br>
> except they poll regularly, not just once a day. That works for those as<br>
> they aren't so resource intensive that we worry a lot about limiting how<br>
> often they run.<br>
><br>
><br>
> Would that approach help how we merge PRs on master?<br>
><br>
><br>
> I think it could be helpful earlier in the release cycle before merging big<br>
> changes, and then perhaps late in the release cycle if we're worried about<br>
> possible regressions.<br>
><br>
><br>
><br>
> Scott<br>
><br>
> On 12/04/2017 09:33 PM, Stuart Douglas wrote:<br>
><br>
><br>
><br>
><br>
> On Tue, Dec 5, 2017 at 3:40 AM, Brian Stansberry <<br>
> <a href="mailto:brian.stansberry@redhat.com">brian.stansberry@redhat.com</a> <mailto: <a href="mailto:brian.stansberry@redhat.com">brian.stansberry@redhat.com</a> >> wrote:<br>
><br>
> Great. :)<br>
><br>
> One thing I think we need to do is figure out how to get custom TCK<br>
> runs for PR branches. The TCK is a big part of our test coverage,<br>
> and one way to not "use master as a test bed" is to get a check of a<br>
> branch on the TCK before we merge it.<br>
><br>
> I know we've gotten TCK runs of ad-hoc branches before, so by<br>
> "figure out" I mean work out how to make that not overly painful,<br>
> come to some sort of consensus on when it's worthwhile, etc.<br>
><br>
><br>
> I think if we were going to do this it should probably be something reviewers<br>
> can ask for on specific PR. The TCK uses a *lot* more resources than a<br>
> standard CI run, so we need to make sure we limit it to cases where it is<br>
> required.<br>
><br>
> Stuart<br>
><br>
><br>
> On Fri, Dec 1, 2017 at 10:04 AM, Alessio Soldano<br>
> < <a href="mailto:asoldano@redhat.com">asoldano@redhat.com</a> <mailto: <a href="mailto:asoldano@redhat.com">asoldano@redhat.com</a> >> wrote:<br>
><br>
> There you go... PR updated to consume the same api jar now<br>
> released as final.<br>
><br>
> Cheers<br>
> Alessio<br>
><br>
> On Fri, Dec 1, 2017 at 3:30 PM, David Lloyd<br>
> < <a href="mailto:david.lloyd@redhat.com">david.lloyd@redhat.com</a> <mailto: <a href="mailto:david.lloyd@redhat.com">david.lloyd@redhat.com</a> >> wrote:<br>
><br>
> On Thu, Nov 30, 2017 at 5:50 PM, Alessio Soldano<br>
> < <a href="mailto:asoldano@redhat.com">asoldano@redhat.com</a> <mailto: <a href="mailto:asoldano@redhat.com">asoldano@redhat.com</a> >> wrote:<br>
> > As suggested by Brian, I'd like to draw attention to the discussion on<br>
> > <a href="https://github.com/wildfly/wildfly/pull/10604" rel="noreferrer" target="_blank">https://github.com/wildfly/<wbr>wildfly/pull/10604</a><br>
> < <a href="https://github.com/wildfly/wildfly/pull/10604" rel="noreferrer" target="_blank">https://github.com/wildfly/<wbr>wildfly/pull/10604</a> > .<br>
> > The PR is an upgrade of the webservices stack, including JBossWS, Apache<br>
> > CXF, JAXB-RI and JAXB API. In particular, the JAXB upgrade is for EE8 and<br>
> > better JDK 9 compatibility.<br>
> > Now, due to the upgrade of the JAXB API spec jar, the PR is essentially<br>
> > stalled since 20 days; the new spec is released as an alpha (as it's been<br>
> > tested within JBossWS only) and that does not satisfy a rule that requires<br>
> > any artifact being pulled to be Final.<br>
> > We're talking about a spec jar, we could simply re-tag that as Final,<br>
> > chances are we won't need changes any time soon there anyway, but as Tomaz<br>
> > pointed out, in principle that would be dishonest.<br>
><br>
> My opinion is that you should go ahead and make a .Final<br>
> tag. In the<br>
> (unlikely?) event that the spec has to be modified for some<br>
> reason, I<br>
> think you could make a 1.0.1.Final tag and call it a "bug fix".<br>
><br>
> The alternative is to simply wait. I don't think there is<br>
> any middle position.<br>
><br>
> > While I see the point in requiring that only sufficiently stable upgrades<br>
> > are applied to the codebase, I'm wondering whether, maybe, we're going a<br>
> > bit<br>
> > too far with the rules. Brian wrote on this topic: "how to determine that<br>
> > something is good enough to go in without using master as a test bed" ?<br>
><br>
> I don't think we are; I agree with the policy as it stands. If you<br>
> look at it in terms of being able to release at any time,<br>
> then it<br>
> follows that everything _must_ be stable.<br>
><br>
> --<br>
> - DML<br>
><br>
><br>
><br>
> ______________________________<wbr>_________________<br>
> wildfly-dev mailing list<br>
> <a href="mailto:wildfly-dev@lists.jboss.org">wildfly-dev@lists.jboss.org</a> <mailto: <a href="mailto:wildfly-dev@lists.jboss.org">wildfly-dev@lists.jboss.org</a> ><br>
> <a href="https://lists.jboss.org/mailman/listinfo/wildfly-dev" rel="noreferrer" target="_blank">https://lists.jboss.org/<wbr>mailman/listinfo/wildfly-dev</a><br>
> < <a href="https://lists.jboss.org/mailman/listinfo/wildfly-dev" rel="noreferrer" target="_blank">https://lists.jboss.org/<wbr>mailman/listinfo/wildfly-dev</a> ><br>
><br>
><br>
><br>
><br>
> -- Brian Stansberry<br>
> Manager, Senior Principal Software Engineer<br>
> Red Hat<br>
><br>
> ______________________________<wbr>_________________<br>
> wildfly-dev mailing list<br>
> <a href="mailto:wildfly-dev@lists.jboss.org">wildfly-dev@lists.jboss.org</a> <mailto: <a href="mailto:wildfly-dev@lists.jboss.org">wildfly-dev@lists.jboss.org</a> ><br>
> <a href="https://lists.jboss.org/mailman/listinfo/wildfly-dev" rel="noreferrer" target="_blank">https://lists.jboss.org/<wbr>mailman/listinfo/wildfly-dev</a><br>
> < <a href="https://lists.jboss.org/mailman/listinfo/wildfly-dev" rel="noreferrer" target="_blank">https://lists.jboss.org/<wbr>mailman/listinfo/wildfly-dev</a> ><br>
><br>
><br>
><br>
><br>
> ______________________________<wbr>_________________<br>
> wildfly-dev mailing list<br>
> <a href="mailto:wildfly-dev@lists.jboss.org">wildfly-dev@lists.jboss.org</a><br>
> <a href="https://lists.jboss.org/mailman/listinfo/wildfly-dev" rel="noreferrer" target="_blank">https://lists.jboss.org/<wbr>mailman/listinfo/wildfly-dev</a><br>
><br>
><br>
><br>
><br>
> --<br>
> Brian Stansberry<br>
> Manager, Senior Principal Software Engineer<br>
> Red Hat<br>
><br>
> ______________________________<wbr>_________________<br>
> wildfly-dev mailing list<br>
> <a href="mailto:wildfly-dev@lists.jboss.org">wildfly-dev@lists.jboss.org</a><br>
> <a href="https://lists.jboss.org/mailman/listinfo/wildfly-dev" rel="noreferrer" target="_blank">https://lists.jboss.org/<wbr>mailman/listinfo/wildfly-dev</a><br>
</div></div></blockquote></div><br></div></div></div></div></div>