Re: [wildfly-dev] Srcdeps in WildFly and WildFly Core

On 26/01/17 12:42, Peter Palaga wrote: > Thanks for the comments, Darran, more inline... > > On 2017-01-26 12:40, Darran Lofthouse wrote: >> On 26/01/17 01:02, Brian Stansberry wrote: >>> My only concerns with this would relate to comitting this kind of src >>> dependency to the poms in the main branches in the widlfly/wildfly >>> and wildfly/wildfly-core repos. We’ve managed to survive up to now >>> with little or no need for that kind of thing, so until we get used >>> to using this in other ways IMHO we should follow the KISS principle >>> and forbid that. >> >> I like the idea of this being available for teams working on a topic >> branch but also think we should avoid this kind of dependency in master >> - engineers are continually running builds so I don't think we should >> add additional source checkouts and builds. > > So you also vote for having the support for source dependencies when the > e.g. the wildfly-core CI builds a PR that integrates a change in Elytron > or when a reviewer does the same, but such PRs should wait for a proper > release of the component, right? I think by the time we get to the PR stage engineers should have had an opportunity to discuss the proposed solution sufficiently that we shouldn't need a srcdep and should have been able to tag Elytron.

>> I know Elytron has caused huge amounts of instability but generally I >> think we want these branches in a state where we could tag if we >> needed to. > > "these branches" - you mean topic branches in a component (such as > Elytron) or in a consuming project (such as WF Core) or both? I think WildFly Core / master and WildFly / master are generally not supposed to be as unstable as we have made them recently. >> When these downloaded builds are executed are the test cases within also >> executed? > > The builds of source dependencies run with -DskipTests by default so > that they finish faster. An explicit per git repository setting in > srcdeps.yaml is needed if somebody wants a dependency build to run > without -DskipTests. > >>> A trick is avoiding doing that by mistake; i.e. a PR is sent up with >>> a SRC dependency to get CI or review and accidentally gets merged. >>> But I suppose that’s not the end of the world, so long as the release >>> process will eventually detect it and fail. >> >> Merging topic branches these temporary SHAs would be in the history but >> maybe the PR CI runs could verify they are not present in the final >> merge. >> >>> Can making srcdeps fail (or just disabling it) be turned on via a >>> maven profile? With that we could set up such a profile and turn it >>> on in CI jobs that are testing branches where it’s forbidden (e.g. >>> the nightly builds of master.) >>> >>> Oh, one other concern — how robust is this in the face of poor >>> maintenance? I see a lot of boilerplate in that .mvn/srcdeps.yaml. If >>> that gets out of date or something is the only effect that using a >>> src dependency for the affected item doesn't work? >> >> I think that file would be better almost empty > > The size of the srcdeps.yaml file depends on how many dependencies the > given project has and which of those are selected for the srcdeps > support. In the PR https://git.io/vMjUC , I have included only the > projects owned by us, excluding non-Maven and non-git projects. I am > open to discussion whether we should include or exclude more projects. > >> but not hidden. > > The .mvn directory was introduced by Maven 3.3.1. Maven reads > extensions.xml file from there and I found it rather natural to have > srcdeps.yaml file there too. What that means is that a file that can directly influence the outcome of a local build is hidden from the engineer. There are already quite a few things in out builds where when they go wrong we have to spend time tracking them down, having a hidden configuration may not help this.

>> Thinking about the topic branch development, within the Elytron team we >> are using some SNAPSHOTs from the official repos and at times we have >> forked projects into our incubator and developed against SNAPSHOTs from >> those > > Allowing SNAPSHOTs on any remote Maven repository leads to loosing > reproducibility of the builds of the dependent project. Srcdeps offers > much more reproducibility and portalbility over environments in such > situations. Yes I am saying in the development of the Elytron topic branches this could help a lot - but the assumptions as to which git repos we are using would be wrong. >> as as you select the SHA for the build you would also need to >> select which repository you really want. > > Yes, the mapping which dependencies are built from which git > repositories is defined in srcdeps.yaml. I see no problem in having one > git URL in one branch and having different git URL in another branch. Which I think as this is a file engineers could be editing to tweak their builds means it should not be hidden. > Thanks, > > Peter > >>>> On Jan 25, 2017, at 3:45 PM, Peter Palaga <ppalaga(a)redhat.com&gt; wrote: >>>> >>>> Hi *, >>>> >>>> this is not new to those of you who attended my talk on the F2F >>>> 2016 in >>>> Brno. Let me explain the idea here again for all others who did not >>>> have >>>> a chance to be there. >>>> >>>> Srcdeps [1] is a tool to build Maven dependencies from their sources. >>>> With srcdeps, wildfly-core can depend on a specific commit of, e.g., >>>> undertow: >>>> >>>> <version.io.undertow>1.4.8.Final-SRC-revision-aabbccd</version.io.undertow> >>>> >>>> >>>> >>>> where aabbccd is the git commit id to build when any undertow artifact >>>> is requested during the build of wildfly-core. >>>> >>>> [1] describes in detail, how it works. >>>> >>>> The main advantage of srcdeps is that changes in components can be >>>> integrated and tested in wildfly-core immediately after they are >>>> committed to a public component branch. There is no need to wait for >>>> the >>>> component release. >>>> >>>> Here in the WildFly family of projects, it is often the case that >>>> something needs to be fixed in a component, but the verification >>>> (using >>>> bug reproducer, or integration test) is possible only at the level of >>>> wildfly or wildfly-core. Engineers typically work with snapshots >>>> locally, but when their changes need to get shared (CI, reviews) in a >>>> reproducible manner, snapshots cannot be used anymore. >>>> In such situations a source dependency come in handy: it is very >>>> easy to >>>> share and it is as reproducible as a Maven build from a specific >>>> commit >>>> can be. All CIs and reviewers can work with it, because all source >>>> dependency compilation is done under the hood by Maven. >>>> >>>> Developers working on changes that span over multiple interdependent >>>> git >>>> repos can thus get feedback (i-tests, reviews) quickly without waiting >>>> for releases of components. >>>> >>>> Srcdeps emerged in the Hawkular family of projects to solve exactly >>>> this >>>> kind of situation and is in use there since around October 2015. >>>> >>>> When I said there is no need to wait for releases of components, I did >>>> not mean that we can get rid of component releases altogether. >>>> Clearly, >>>> we cannot, because i.a. for any tooling uninformed about how srcdeps >>>> work, those source dependencies would simply be non-resolvable from >>>> public Maven repositories. So, before releasing the dependent >>>> component >>>> (such as wildfly-core) all its dependencies need to be released. To >>>> enforce this, srcdeps is by default configured to make the release >>>> fail, >>>> as long as there are source dependencies. >>>> >>>> I have sent a PR introducing srcdeps to wildfly-core: >>>> https://github.com/wildfly/wildfly-core/pull/2122 >>>> To get a feeling how it works, checkout the branch, switch to e.g. >>>> <version.io.undertow>1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd</version.io.undertow> >>>> >>>> >>>> (that happens to be the commit id of the 1.4.8.Final tag) >>>> and build wildfly-core as usual with "mvn clean install". You'll >>>> see in >>>> the build log that undertow is being cloned to >>>> ~/.m2/srcdeps/io/undertow >>>> and that it is built there. After the build, check that the >>>> 1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd >>>> version of Undertow got installed to your local Maven repo (usually >>>> ~/m2/repository/io/undertow/undertow-core ) >>>> >>>> Are there any questions or comments? >>>> >>>> [1] https://github.com/srcdeps/srcdeps-maven#srcdeps-maven >>>> >>>> Thanks, >>>> >>>> Peter >>>> >>>> P.S.: I will be talking about srcdeps on Saturday 2017-01-28 at >>>> 14:30 at >>>> DevConf Brno. >>>> _______________________________________________ >>>> wildfly-dev mailing list >>>> wildfly-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >>> >> _______________________________________________ >> wildfly-dev mailing list >> wildfly-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/wildfly-dev >> >