[wildfly-dev] Srcdeps in WildFly and WildFly Core

Thu Jan 26 07:42:20 EST 2017

Thanks for the comments, Darran, more inline...

On 2017-01-26 12:40, Darran Lofthouse wrote:
> On 26/01/17 01:02, Brian Stansberry wrote:
>> My only concerns with this would relate to comitting this kind of src dependency to the poms in the main branches in the widlfly/wildfly and wildfly/wildfly-core repos. We’ve managed to survive up to now with little or no need for that kind of thing, so until we get used to using this in other ways IMHO we should follow the KISS principle and forbid that.
>
> I like the idea of this being available for teams working on a topic
> branch but also think we should avoid this kind of dependency in master
> - engineers are continually running builds so I don't think we should
> add additional source checkouts and builds.

So you also vote for having the support for source dependencies when the 
e.g. the wildfly-core CI builds a PR that integrates a change in Elytron 
or when a reviewer does the same, but such PRs should wait for a proper 
release of the component, right?

> I know Elytron has caused huge amounts of instability but generally I
> think we want these branches in a state where we could tag if we needed to.

"these branches" - you mean topic branches in a component (such as 
Elytron) or in a consuming project (such as WF Core) or both?

> When these downloaded builds are executed are the test cases within also
> executed?

The builds of source dependencies run with -DskipTests by default so 
that they finish faster. An explicit per git repository setting in 
srcdeps.yaml is needed if somebody wants a dependency build to run 
without -DskipTests.

>> A trick is avoiding doing that by mistake; i.e. a PR is sent up with a SRC dependency to get CI or review and accidentally gets merged. But I suppose that’s not the end of the world, so long as the release process will eventually detect it and fail.
>
> Merging topic branches these temporary SHAs would be in the history but
> maybe the PR CI runs could verify they are not present in the final merge.
>
>> Can making srcdeps fail (or just disabling it) be turned on via a maven profile? With that we could set up such a profile and turn it on in CI jobs that are testing branches where it’s forbidden (e.g. the nightly builds of master.)
>>
>> Oh, one other concern — how robust is this in the face of poor maintenance? I see a lot of boilerplate in that .mvn/srcdeps.yaml. If that gets out of date or something is the only effect that using a src dependency for the affected item doesn't work?
>
> I think that file would be better almost empty

The size of the srcdeps.yaml file depends on how many dependencies the 
given project has and which of those are selected for the srcdeps 
support. In the PR https://git.io/vMjUC , I have included only the 
projects owned by us, excluding non-Maven and non-git projects. I am 
open to discussion whether we should include or exclude more projects.

> but not hidden.

The .mvn directory was introduced by Maven 3.3.1. Maven reads 
extensions.xml file from there and I found it rather natural to have 
srcdeps.yaml file there too.

> Thinking about the topic branch development, within the Elytron team we
> are using some SNAPSHOTs from the official repos and at times we have
> forked projects into our incubator and developed against SNAPSHOTs from
> those

Allowing SNAPSHOTs on any remote Maven repository leads to loosing 
reproducibility of the builds of the dependent project. Srcdeps offers 
much more reproducibility and portalbility over environments in such 
situations.

> as as you select the SHA for the build you would also need to
> select which repository you really want.

Yes, the mapping which dependencies are built from which git 
repositories is defined in srcdeps.yaml. I see no problem in having one 
git URL in one branch and having different git URL in another branch.

Thanks,

Peter

>>> On Jan 25, 2017, at 3:45 PM, Peter Palaga <ppalaga at redhat.com> wrote:
>>>
>>> Hi *,
>>>
>>> this is not new to those of you who attended my talk on the F2F 2016 in
>>> Brno. Let me explain the idea here again for all others who did not have
>>> a chance to be there.
>>>
>>> Srcdeps [1] is a tool to build Maven dependencies from their sources.
>>> With srcdeps, wildfly-core can depend on a specific commit of, e.g.,
>>> undertow:
>>>
>>> <version.io.undertow>1.4.8.Final-SRC-revision-aabbccd</version.io.undertow>
>>>
>>> where aabbccd is the git commit id to build when any undertow artifact
>>> is requested during the build of wildfly-core.
>>>
>>> [1] describes in detail, how it works.
>>>
>>> The main advantage of srcdeps is that changes in components can be
>>> integrated and tested in wildfly-core immediately after they are
>>> committed to a public component branch. There is no need to wait for the
>>> component release.
>>>
>>> Here in the WildFly family of projects, it is often the case that
>>> something needs to be fixed in a component, but the verification (using
>>> bug reproducer, or integration test) is possible only at the level of
>>> wildfly or wildfly-core. Engineers typically work with snapshots
>>> locally, but when their changes need to get shared (CI, reviews) in a
>>> reproducible manner, snapshots cannot be used anymore.
>>> In such situations a source dependency come in handy: it is very easy to
>>> share and it is as reproducible as a Maven build from a specific commit
>>> can be. All CIs and reviewers can work with it, because all source
>>> dependency compilation is done under the hood by Maven.
>>>
>>> Developers working on changes that span over multiple interdependent git
>>> repos can thus get feedback (i-tests, reviews) quickly without waiting
>>> for releases of components.
>>>
>>> Srcdeps emerged in the Hawkular family of projects to solve exactly this
>>> kind of situation and is in use there since around October 2015.
>>>
>>> When I said there is no need to wait for releases of components, I did
>>> not mean that we can get rid of component releases altogether. Clearly,
>>> we cannot, because i.a. for any tooling uninformed about how srcdeps
>>> work, those source dependencies would simply be non-resolvable from
>>> public Maven repositories. So, before releasing the dependent component
>>> (such as wildfly-core) all its dependencies need to be released. To
>>> enforce this, srcdeps is by default configured to make the release fail,
>>> as long as there are source dependencies.
>>>
>>> I have sent a PR introducing srcdeps to wildfly-core:
>>> https://github.com/wildfly/wildfly-core/pull/2122
>>> To get a feeling how it works, checkout the branch, switch to e.g.
>>> <version.io.undertow>1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd</version.io.undertow>
>>> (that happens to be the commit id of the 1.4.8.Final tag)
>>> and build wildfly-core as usual with "mvn clean install". You'll see in
>>> the build log that undertow is being cloned to ~/.m2/srcdeps/io/undertow
>>> and that it is built there. After the build, check that the
>>> 1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd
>>> version of Undertow got installed to your local Maven repo (usually
>>> ~/m2/repository/io/undertow/undertow-core )
>>>
>>> Are there any questions or comments?
>>>
>>> [1] https://github.com/srcdeps/srcdeps-maven#srcdeps-maven
>>>
>>> Thanks,
>>>
>>> Peter
>>>
>>> P.S.: I will be talking about srcdeps on Saturday 2017-01-28 at 14:30 at
>>> DevConf Brno.
>>> _______________________________________________
>>> wildfly-dev mailing list
>>> wildfly-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>