[wildfly-dev] Srcdeps in WildFly and WildFly Core

Thu Jan 26 03:54:45 EST 2017

Hi Brian, thanks for your comments, more inline...

On 2017-01-26 02:02, Brian Stansberry wrote:
> My only concerns with this would relate to comitting this kind of src
> dependency to the poms in the main branches in the widlfly/wildfly
> and wildfly/wildfly-core repos. We’ve managed to survive up to now
> with little or no need for that kind of thing, so until we get used
> to using this in other ways IMHO we should follow the KISS principle
> and forbid that.

Maybe I overestimate the amount of changes that span over multiple git 
repos. Maybe you in the Core team do not do this often. But for us in 
the Sustaining Engineering Team, this is quite a typical situation. A 
substantial part of the reports from customers come with a description 
how to reproduce on the whole server, but they need to be fixed in a 
component. Having srcdeps would make the CP process simpler and faster, 
allowing us to uncover the conflicts and regressions earlier.

> A trick is avoiding doing that by mistake; i.e. a PR is sent up with
> a SRC dependency to get CI or review and accidentally gets merged.

Oh, I am just realizing I have not said anything about merging. I 
actually do want to propose that commits with source dependencies get 
merged to e.g. wildfly-core master as early as possible. Those are the 
key points of Continuous Integration: get feedback quickly, and merge as 
soon as possible. This is exactly what Hawkular is doing since more than 
a year.

> But I suppose that’s not the end of the world, so long as the release
> process will eventually detect it and fail.

Yes, source dependencies on a stable branch do not harm. They just need 
to be avoided in releases (for which srcdeps offers technical means).

> Can making srcdeps fail (or just disabling it) be turned on via a
> maven profile? With that we could set up such a profile and turn it
> on in CI jobs that are testing branches where it’s forbidden (e.g.
> the nightly builds of master.)

Yes, the feature is called "failWith profiles" and can be configured in 
.mvn/srcdeps.yaml, like here in this srcdeps quickstart: 
https://github.com/srcdeps/srcdeps-maven/blob/master/srcdeps-maven-quickstarts/srcdeps-mvn-git-profile-quickstart/.mvn/srcdeps.yaml#L33 

There is also "failWith properties" and "failWith goals". It is 
documented here: 
https://github.com/srcdeps/srcdeps-core/blob/master/doc/srcdeps.yaml#L130
By default there is failWith: {goals: release:prepare, release:perform}. 
Projects that do not use the release plugin can set e.g. failWith: 
{goals: deploy:deploy} or whatever else distinguishes their releases.

> Oh, one other concern — how robust is this in the face of poor
> maintenance? I see a lot of boilerplate in that .mvn/srcdeps.yaml.

Which parts are boilerpate?

> If
> that gets out of date or something is the only effect that using a
> src dependency for the affected item doesn't work?

Yes, I think so. As long as the .mvn/srcdeps.yaml file is syntactically 
correct, any misconfiguration there should not have any other effect 
than eventually breaking an embedded build.

Generally, the things configured in .mvn/srcdeps.yaml tend to be quite 
stable - it is basically just mapping from GAVs to their respective git 
URLs. Git URLs do not change often. It is true that dependency artifacts 
come and go, but as long as their groupIds are selected reasonably (one 
groupId occurs in not more than one git repo) the mapping itself can be 
quite stable over time too.

Thanks,

Peter

>
>> On Jan 25, 2017, at 3:45 PM, Peter Palaga <ppalaga at redhat.com>
>> wrote:
>>
>> Hi *,
>>
>> this is not new to those of you who attended my talk on the F2F
>> 2016 in Brno. Let me explain the idea here again for all others who
>> did not have a chance to be there.
>>
>> Srcdeps [1] is a tool to build Maven dependencies from their
>> sources. With srcdeps, wildfly-core can depend on a specific commit
>> of, e.g., undertow:
>>
>> <version.io.undertow>1.4.8.Final-SRC-revision-aabbccd</version.io.undertow>
>>
>>
>>
where aabbccd is the git commit id to build when any undertow artifact
>> is requested during the build of wildfly-core.
>>
>> [1] describes in detail, how it works.
>>
>> The main advantage of srcdeps is that changes in components can be
>>  integrated and tested in wildfly-core immediately after they are
>> committed to a public component branch. There is no need to wait
>> for the component release.
>>
>> Here in the WildFly family of projects, it is often the case that
>> something needs to be fixed in a component, but the verification
>> (using bug reproducer, or integration test) is possible only at the
>> level of wildfly or wildfly-core. Engineers typically work with
>> snapshots locally, but when their changes need to get shared (CI,
>> reviews) in a reproducible manner, snapshots cannot be used
>> anymore. In such situations a source dependency come in handy: it
>> is very easy to share and it is as reproducible as a Maven build
>> from a specific commit can be. All CIs and reviewers can work with
>> it, because all source dependency compilation is done under the
>> hood by Maven.
>>
>> Developers working on changes that span over multiple
>> interdependent git repos can thus get feedback (i-tests, reviews)
>> quickly without waiting for releases of components.
>>
>> Srcdeps emerged in the Hawkular family of projects to solve exactly
>> this kind of situation and is in use there since around October
>> 2015.
>>
>> When I said there is no need to wait for releases of components, I
>> did not mean that we can get rid of component releases altogether.
>> Clearly, we cannot, because i.a. for any tooling uninformed about
>> how srcdeps work, those source dependencies would simply be
>> non-resolvable from public Maven repositories. So, before releasing
>> the dependent component (such as wildfly-core) all its dependencies
>> need to be released. To enforce this, srcdeps is by default
>> configured to make the release fail, as long as there are source
>> dependencies.
>>
>> I have sent a PR introducing srcdeps to wildfly-core:
>> https://github.com/wildfly/wildfly-core/pull/2122 To get a feeling
>> how it works, checkout the branch, switch to e.g.
>> <version.io.undertow>1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd</version.io.undertow>
>>  (that happens to be the commit id of the 1.4.8.Final tag) and
>> build wildfly-core as usual with "mvn clean install". You'll see in
>>  the build log that undertow is being cloned to
>> ~/.m2/srcdeps/io/undertow and that it is built there. After the
>> build, check that the
>> 1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd
>> version of Undertow got installed to your local Maven repo (usually
>>  ~/m2/repository/io/undertow/undertow-core )
>>
>> Are there any questions or comments?
>>
>> [1] https://github.com/srcdeps/srcdeps-maven#srcdeps-maven
>>
>> Thanks,
>>
>> Peter
>>
>> P.S.: I will be talking about srcdeps on Saturday 2017-01-28 at
>> 14:30 at DevConf Brno.
>> _______________________________________________ wildfly-dev mailing
>> list wildfly-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>