[wildfly-dev] Srcdeps in WildFly and WildFly Core

Thu Jan 26 07:56:39 EST 2017

On 26/01/17 12:42, Peter Palaga wrote:
> Thanks for the comments, Darran, more inline...
>
> On 2017-01-26 12:40, Darran Lofthouse wrote:
>> On 26/01/17 01:02, Brian Stansberry wrote:
>>> My only concerns with this would relate to comitting this kind of src
>>> dependency to the poms in the main branches in the widlfly/wildfly
>>> and wildfly/wildfly-core repos. We’ve managed to survive up to now
>>> with little or no need for that kind of thing, so until we get used
>>> to using this in other ways IMHO we should follow the KISS principle
>>> and forbid that.
>>
>> I like the idea of this being available for teams working on a topic
>> branch but also think we should avoid this kind of dependency in master
>> - engineers are continually running builds so I don't think we should
>> add additional source checkouts and builds.
>
> So you also vote for having the support for source dependencies when the
> e.g. the wildfly-core CI builds a PR that integrates a change in Elytron
> or when a reviewer does the same, but such PRs should wait for a proper
> release of the component, right?

I think by the time we get to the PR stage engineers should have had an 
opportunity to discuss the proposed solution sufficiently that we 
shouldn't need a srcdep and should have been able to tag Elytron.

>> I know Elytron has caused huge amounts of instability but generally I
>> think we want these branches in a state where we could tag if we
>> needed to.
>
> "these branches" - you mean topic branches in a component (such as
> Elytron) or in a consuming project (such as WF Core) or both?

I think WildFly Core / master and WildFly / master are generally not 
supposed to be as unstable as we have made them recently.

>> When these downloaded builds are executed are the test cases within also
>> executed?
>
> The builds of source dependencies run with -DskipTests by default so
> that they finish faster. An explicit per git repository setting in
> srcdeps.yaml is needed if somebody wants a dependency build to run
> without -DskipTests.
>
>>> A trick is avoiding doing that by mistake; i.e. a PR is sent up with
>>> a SRC dependency to get CI or review and accidentally gets merged.
>>> But I suppose that’s not the end of the world, so long as the release
>>> process will eventually detect it and fail.
>>
>> Merging topic branches these temporary SHAs would be in the history but
>> maybe the PR CI runs could verify they are not present in the final
>> merge.
>>
>>> Can making srcdeps fail (or just disabling it) be turned on via a
>>> maven profile? With that we could set up such a profile and turn it
>>> on in CI jobs that are testing branches where it’s forbidden (e.g.
>>> the nightly builds of master.)
>>>
>>> Oh, one other concern — how robust is this in the face of poor
>>> maintenance? I see a lot of boilerplate in that .mvn/srcdeps.yaml. If
>>> that gets out of date or something is the only effect that using a
>>> src dependency for the affected item doesn't work?
>>
>> I think that file would be better almost empty
>
> The size of the srcdeps.yaml file depends on how many dependencies the
> given project has and which of those are selected for the srcdeps
> support. In the PR https://git.io/vMjUC , I have included only the
> projects owned by us, excluding non-Maven and non-git projects. I am
> open to discussion whether we should include or exclude more projects.
>
>> but not hidden.
>
> The .mvn directory was introduced by Maven 3.3.1. Maven reads
> extensions.xml file from there and I found it rather natural to have
> srcdeps.yaml file there too.

What that means is that a file that can directly influence the outcome 
of a local build is hidden from the engineer.  There are already quite a 
few things in out builds where when they go wrong we have to spend time 
tracking them down, having a hidden configuration may not help this.

>> Thinking about the topic branch development, within the Elytron team we
>> are using some SNAPSHOTs from the official repos and at times we have
>> forked projects into our incubator and developed against SNAPSHOTs from
>> those
>
> Allowing SNAPSHOTs on any remote Maven repository leads to loosing
> reproducibility of the builds of the dependent project. Srcdeps offers
> much more reproducibility and portalbility over environments in such
> situations.

Yes I am saying in the development of the Elytron topic branches this 
could help a lot - but the assumptions as to which git repos we are 
using would be wrong.

>> as as you select the SHA for the build you would also need to
>> select which repository you really want.
>
> Yes, the mapping which dependencies are built from which git
> repositories is defined in srcdeps.yaml. I see no problem in having one
> git URL in one branch and having different git URL in another branch.

Which I think as this is a file engineers could be editing to tweak 
their builds means it should not be hidden.

> Thanks,
>
> Peter
>
>>>> On Jan 25, 2017, at 3:45 PM, Peter Palaga <ppalaga at redhat.com> wrote:
>>>>
>>>> Hi *,
>>>>
>>>> this is not new to those of you who attended my talk on the F2F 2016 in
>>>> Brno. Let me explain the idea here again for all others who did not
>>>> have
>>>> a chance to be there.
>>>>
>>>> Srcdeps [1] is a tool to build Maven dependencies from their sources.
>>>> With srcdeps, wildfly-core can depend on a specific commit of, e.g.,
>>>> undertow:
>>>>
>>>> <version.io.undertow>1.4.8.Final-SRC-revision-aabbccd</version.io.undertow>
>>>>
>>>>
>>>> where aabbccd is the git commit id to build when any undertow artifact
>>>> is requested during the build of wildfly-core.
>>>>
>>>> [1] describes in detail, how it works.
>>>>
>>>> The main advantage of srcdeps is that changes in components can be
>>>> integrated and tested in wildfly-core immediately after they are
>>>> committed to a public component branch. There is no need to wait for
>>>> the
>>>> component release.
>>>>
>>>> Here in the WildFly family of projects, it is often the case that
>>>> something needs to be fixed in a component, but the verification (using
>>>> bug reproducer, or integration test) is possible only at the level of
>>>> wildfly or wildfly-core. Engineers typically work with snapshots
>>>> locally, but when their changes need to get shared (CI, reviews) in a
>>>> reproducible manner, snapshots cannot be used anymore.
>>>> In such situations a source dependency come in handy: it is very
>>>> easy to
>>>> share and it is as reproducible as a Maven build from a specific commit
>>>> can be. All CIs and reviewers can work with it, because all source
>>>> dependency compilation is done under the hood by Maven.
>>>>
>>>> Developers working on changes that span over multiple interdependent
>>>> git
>>>> repos can thus get feedback (i-tests, reviews) quickly without waiting
>>>> for releases of components.
>>>>
>>>> Srcdeps emerged in the Hawkular family of projects to solve exactly
>>>> this
>>>> kind of situation and is in use there since around October 2015.
>>>>
>>>> When I said there is no need to wait for releases of components, I did
>>>> not mean that we can get rid of component releases altogether. Clearly,
>>>> we cannot, because i.a. for any tooling uninformed about how srcdeps
>>>> work, those source dependencies would simply be non-resolvable from
>>>> public Maven repositories. So, before releasing the dependent component
>>>> (such as wildfly-core) all its dependencies need to be released. To
>>>> enforce this, srcdeps is by default configured to make the release
>>>> fail,
>>>> as long as there are source dependencies.
>>>>
>>>> I have sent a PR introducing srcdeps to wildfly-core:
>>>> https://github.com/wildfly/wildfly-core/pull/2122
>>>> To get a feeling how it works, checkout the branch, switch to e.g.
>>>> <version.io.undertow>1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd</version.io.undertow>
>>>>
>>>> (that happens to be the commit id of the 1.4.8.Final tag)
>>>> and build wildfly-core as usual with "mvn clean install". You'll see in
>>>> the build log that undertow is being cloned to
>>>> ~/.m2/srcdeps/io/undertow
>>>> and that it is built there. After the build, check that the
>>>> 1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd
>>>> version of Undertow got installed to your local Maven repo (usually
>>>> ~/m2/repository/io/undertow/undertow-core )
>>>>
>>>> Are there any questions or comments?
>>>>
>>>> [1] https://github.com/srcdeps/srcdeps-maven#srcdeps-maven
>>>>
>>>> Thanks,
>>>>
>>>> Peter
>>>>
>>>> P.S.: I will be talking about srcdeps on Saturday 2017-01-28 at
>>>> 14:30 at
>>>> DevConf Brno.
>>>> _______________________________________________
>>>> wildfly-dev mailing list
>>>> wildfly-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>>
>> _______________________________________________
>> wildfly-dev mailing list
>> wildfly-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>
>