[wildfly-dev] Srcdeps in WildFly and WildFly Core

Kabir Khan kabir.khan at jboss.com
Tue Jan 31 06:03:50 EST 2017 

> On 30 Jan 2017, at 15:36, Peter Palaga <ppalaga at redhat.com> wrote:
> 
> Hi Darran, inline...
> 
> On 2017-01-26 13:56, Darran Lofthouse wrote:
>> 
>> 
>> On 26/01/17 12:42, Peter Palaga wrote:
>>> Thanks for the comments, Darran, more inline...
>>> 
>>> On 2017-01-26 12:40, Darran Lofthouse wrote:
>>>> On 26/01/17 01:02, Brian Stansberry wrote:
>>>>> My only concerns with this would relate to comitting this kind of src
>>>>> dependency to the poms in the main branches in the widlfly/wildfly
>>>>> and wildfly/wildfly-core repos. We’ve managed to survive up to now
>>>>> with little or no need for that kind of thing, so until we get used
>>>>> to using this in other ways IMHO we should follow the KISS principle
>>>>> and forbid that.
>>>> 
>>>> I like the idea of this being available for teams working on a topic
>>>> branch but also think we should avoid this kind of dependency in master
>>>> - engineers are continually running builds so I don't think we should
>>>> add additional source checkouts and builds.
>>> 
>>> So you also vote for having the support for source dependencies when the
>>> e.g. the wildfly-core CI builds a PR that integrates a change in Elytron
>>> or when a reviewer does the same, but such PRs should wait for a proper
>>> release of the component, right?
>> 
>> I think by the time we get to the PR stage engineers should have had an
>> opportunity to discuss the proposed solution sufficiently that we
>> shouldn't need a srcdep and should have been able to tag Elytron.
> 
> OK, thanks for explaining!
> 
>>>> I know Elytron has caused huge amounts of instability but generally I
>>>> think we want these branches in a state where we could tag if we
>>>> needed to.
>>> 
>>> "these branches" - you mean topic branches in a component (such as
>>> Elytron) or in a consuming project (such as WF Core) or both?
>> 
>> I think WildFly Core / master and WildFly / master are generally not
>> supposed to be as unstable as we have made them recently.
>> 
>>>> When these downloaded builds are executed are the test cases within also
>>>> executed?
>>> 
>>> The builds of source dependencies run with -DskipTests by default so
>>> that they finish faster. An explicit per git repository setting in
>>> srcdeps.yaml is needed if somebody wants a dependency build to run
>>> without -DskipTests.
>>> 
>>>>> A trick is avoiding doing that by mistake; i.e. a PR is sent up with
>>>>> a SRC dependency to get CI or review and accidentally gets merged.
>>>>> But I suppose that’s not the end of the world, so long as the release
>>>>> process will eventually detect it and fail.
>>>> 
>>>> Merging topic branches these temporary SHAs would be in the history but
>>>> maybe the PR CI runs could verify they are not present in the final
>>>> merge.
>>>> 
>>>>> Can making srcdeps fail (or just disabling it) be turned on via a
>>>>> maven profile? With that we could set up such a profile and turn it
>>>>> on in CI jobs that are testing branches where it’s forbidden (e.g.
>>>>> the nightly builds of master.)
>>>>> 
>>>>> Oh, one other concern — how robust is this in the face of poor
>>>>> maintenance? I see a lot of boilerplate in that .mvn/srcdeps.yaml. If
>>>>> that gets out of date or something is the only effect that using a
>>>>> src dependency for the affected item doesn't work?
>>>> 
>>>> I think that file would be better almost empty
>>> 
>>> The size of the srcdeps.yaml file depends on how many dependencies the
>>> given project has and which of those are selected for the srcdeps
>>> support. In the PR https://git.io/vMjUC , I have included only the
>>> projects owned by us, excluding non-Maven and non-git projects. I am
>>> open to discussion whether we should include or exclude more projects.
>>> 
>>>> but not hidden.
>>> 
>>> The .mvn directory was introduced by Maven 3.3.1. Maven reads
>>> extensions.xml file from there and I found it rather natural to have
>>> srcdeps.yaml file there too.
>> 
>> What that means is that a file that can directly influence the outcome
>> of a local build is hidden from the engineer.  There are already quite a
>> few things in out builds where when they go wrong we have to spend time
>> tracking them down, having a hidden configuration may not help this.
> 
> If I heart this earlier, I'd most probably choose a non-hidden location. 
> I perhaps do not see this as a problem, because I configure each of my 
> Eclipse workspaces to show all .* resources because .mvn is not the only 
> important file that is hidden: there is also .gitignore, .gitattributes, 
> .travis.yml, .appveyor.yml and even .git and others that one wants to 
> open from time to time.
> But now that srcdeps has some users, sorry, I do not think I find this 
> important enough to break backwards compatibility.
You could perhaps provide both? It will first look in the hidden location, and next in the visible location (wherever that may be). If both are used, give an error.
> 
> -- P
> 
>>>> Thinking about the topic branch development, within the Elytron team we
>>>> are using some SNAPSHOTs from the official repos and at times we have
>>>> forked projects into our incubator and developed against SNAPSHOTs from
>>>> those
>>> 
>>> Allowing SNAPSHOTs on any remote Maven repository leads to loosing
>>> reproducibility of the builds of the dependent project. Srcdeps offers
>>> much more reproducibility and portalbility over environments in such
>>> situations.
>> 
>> Yes I am saying in the development of the Elytron topic branches this
>> could help a lot - but the assumptions as to which git repos we are
>> using would be wrong.
>> 
>>>> as as you select the SHA for the build you would also need to
>>>> select which repository you really want.
>>> 
>>> Yes, the mapping which dependencies are built from which git
>>> repositories is defined in srcdeps.yaml. I see no problem in having one
>>> git URL in one branch and having different git URL in another branch.
>> 
>> Which I think as this is a file engineers could be editing to tweak
>> their builds means it should not be hidden.
>> 
>>> Thanks,
>>> 
>>> Peter
>>> 
>>>>>> On Jan 25, 2017, at 3:45 PM, Peter Palaga <ppalaga at redhat.com> wrote:
>>>>>> 
>>>>>> Hi *,
>>>>>> 
>>>>>> this is not new to those of you who attended my talk on the F2F
>>>>>> 2016 in
>>>>>> Brno. Let me explain the idea here again for all others who did not
>>>>>> have
>>>>>> a chance to be there.
>>>>>> 
>>>>>> Srcdeps [1] is a tool to build Maven dependencies from their sources.
>>>>>> With srcdeps, wildfly-core can depend on a specific commit of, e.g.,
>>>>>> undertow:
>>>>>> 
>>>>>> <version.io.undertow>1.4.8.Final-SRC-revision-aabbccd</version.io.undertow>
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> where aabbccd is the git commit id to build when any undertow artifact
>>>>>> is requested during the build of wildfly-core.
>>>>>> 
>>>>>> [1] describes in detail, how it works.
>>>>>> 
>>>>>> The main advantage of srcdeps is that changes in components can be
>>>>>> integrated and tested in wildfly-core immediately after they are
>>>>>> committed to a public component branch. There is no need to wait for
>>>>>> the
>>>>>> component release.
>>>>>> 
>>>>>> Here in the WildFly family of projects, it is often the case that
>>>>>> something needs to be fixed in a component, but the verification
>>>>>> (using
>>>>>> bug reproducer, or integration test) is possible only at the level of
>>>>>> wildfly or wildfly-core. Engineers typically work with snapshots
>>>>>> locally, but when their changes need to get shared (CI, reviews) in a
>>>>>> reproducible manner, snapshots cannot be used anymore.
>>>>>> In such situations a source dependency come in handy: it is very
>>>>>> easy to
>>>>>> share and it is as reproducible as a Maven build from a specific
>>>>>> commit
>>>>>> can be. All CIs and reviewers can work with it, because all source
>>>>>> dependency compilation is done under the hood by Maven.
>>>>>> 
>>>>>> Developers working on changes that span over multiple interdependent
>>>>>> git
>>>>>> repos can thus get feedback (i-tests, reviews) quickly without waiting
>>>>>> for releases of components.
>>>>>> 
>>>>>> Srcdeps emerged in the Hawkular family of projects to solve exactly
>>>>>> this
>>>>>> kind of situation and is in use there since around October 2015.
>>>>>> 
>>>>>> When I said there is no need to wait for releases of components, I did
>>>>>> not mean that we can get rid of component releases altogether.
>>>>>> Clearly,
>>>>>> we cannot, because i.a. for any tooling uninformed about how srcdeps
>>>>>> work, those source dependencies would simply be non-resolvable from
>>>>>> public Maven repositories. So, before releasing the dependent
>>>>>> component
>>>>>> (such as wildfly-core) all its dependencies need to be released. To
>>>>>> enforce this, srcdeps is by default configured to make the release
>>>>>> fail,
>>>>>> as long as there are source dependencies.
>>>>>> 
>>>>>> I have sent a PR introducing srcdeps to wildfly-core:
>>>>>> https://github.com/wildfly/wildfly-core/pull/2122
>>>>>> To get a feeling how it works, checkout the branch, switch to e.g.
>>>>>> <version.io.undertow>1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd</version.io.undertow>
>>>>>> 
>>>>>> 
>>>>>> (that happens to be the commit id of the 1.4.8.Final tag)
>>>>>> and build wildfly-core as usual with "mvn clean install". You'll
>>>>>> see in
>>>>>> the build log that undertow is being cloned to
>>>>>> ~/.m2/srcdeps/io/undertow
>>>>>> and that it is built there. After the build, check that the
>>>>>> 1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd
>>>>>> version of Undertow got installed to your local Maven repo (usually
>>>>>> ~/m2/repository/io/undertow/undertow-core )
>>>>>> 
>>>>>> Are there any questions or comments?
>>>>>> 
>>>>>> [1] https://github.com/srcdeps/srcdeps-maven#srcdeps-maven
>>>>>> 
>>>>>> Thanks,
>>>>>> 
>>>>>> Peter
>>>>>> 
>>>>>> P.S.: I will be talking about srcdeps on Saturday 2017-01-28 at
>>>>>> 14:30 at
>>>>>> DevConf Brno.
>>>>>> _______________________________________________
>>>>>> wildfly-dev mailing list
>>>>>> wildfly-dev at lists.jboss.org
>>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>>>> 
>>>> _______________________________________________
>>>> wildfly-dev mailing list
>>>> wildfly-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>>> 
>>> 
> 
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev

More information about the wildfly-dev mailing list