[wildfly-dev] Srcdeps in WildFly and WildFly Core

Thu Jan 26 04:56:56 EST 2017

Ahoj Petře, which particular problems would be caused by a SRC 
dependency merged to a master branch? -- P

On 2017-01-26 10:52, Petr Sakar wrote:
>
>
> On 01/26/2017 09:54 AM, Peter Palaga wrote:
>> Hi Brian, thanks for your comments, more inline...
>>
>> On 2017-01-26 02:02, Brian Stansberry wrote:
>>> My only concerns with this would relate to comitting this kind of src
>>> dependency to the poms in the main branches in the widlfly/wildfly
>>> and wildfly/wildfly-core repos. We’ve managed to survive up to now
>>> with little or no need for that kind of thing, so until we get used
>>> to using this in other ways IMHO we should follow the KISS principle
>>> and forbid that.
>> Maybe I overestimate the amount of changes that span over multiple git
>> repos. Maybe you in the Core team do not do this often. But for us in
>> the Sustaining Engineering Team, this is quite a typical situation. A
>> substantial part of the reports from customers come with a description
>> how to reproduce on the whole server, but they need to be fixed in a
>> component. Having srcdeps would make the CP process simpler and faster,
>> allowing us to uncover the conflicts and regressions earlier.
>>
>>> A trick is avoiding doing that by mistake; i.e. a PR is sent up with
>>> a SRC dependency to get CI or review and accidentally gets merged.
>> Oh, I am just realizing I have not said anything about merging. I
>> actually do want to propose that commits with source dependencies get
>> merged to e.g. wildfly-core master as early as possible. Those are the
>> key points of Continuous Integration: get feedback quickly, and merge as
>> soon as possible. This is exactly what Hawkular is doing since more than
>> a year.
>
> I agree with Brian. PR with SRC dependency *should not* be merged to master branch, that would create problems down the road.
> You can use other branch as base for such merges and related CI
> Petr
>
>>> But I suppose that’s not the end of the world, so long as the release
>>> process will eventually detect it and fail.
>> Yes, source dependencies on a stable branch do not harm. They just need
>> to be avoided in releases (for which srcdeps offers technical means).
>>
>>> Can making srcdeps fail (or just disabling it) be turned on via a
>>> maven profile? With that we could set up such a profile and turn it
>>> on in CI jobs that are testing branches where it’s forbidden (e.g.
>>> the nightly builds of master.)
>> Yes, the feature is called "failWith profiles" and can be configured in
>> .mvn/srcdeps.yaml, like here in this srcdeps quickstart:
>> https://github.com/srcdeps/srcdeps-maven/blob/master/srcdeps-maven-quickstarts/srcdeps-mvn-git-profile-quickstart/.mvn/srcdeps.yaml#L33
>>
>> There is also "failWith properties" and "failWith goals". It is
>> documented here:
>> https://github.com/srcdeps/srcdeps-core/blob/master/doc/srcdeps.yaml#L130
>> By default there is failWith: {goals: release:prepare, release:perform}.
>> Projects that do not use the release plugin can set e.g. failWith:
>> {goals: deploy:deploy} or whatever else distinguishes their releases.
>>
>>> Oh, one other concern — how robust is this in the face of poor
>>> maintenance? I see a lot of boilerplate in that .mvn/srcdeps.yaml.
>> Which parts are boilerpate?
>>
>>> If
>>> that gets out of date or something is the only effect that using a
>>> src dependency for the affected item doesn't work?
>> Yes, I think so. As long as the .mvn/srcdeps.yaml file is syntactically
>> correct, any misconfiguration there should not have any other effect
>> than eventually breaking an embedded build.
>>
>> Generally, the things configured in .mvn/srcdeps.yaml tend to be quite
>> stable - it is basically just mapping from GAVs to their respective git
>> URLs. Git URLs do not change often. It is true that dependency artifacts
>> come and go, but as long as their groupIds are selected reasonably (one
>> groupId occurs in not more than one git repo) the mapping itself can be
>> quite stable over time too.
>>
>> Thanks,
>>
>> Peter
>>
>>>> On Jan 25, 2017, at 3:45 PM, Peter Palaga <ppalaga at redhat.com>
>>>> wrote:
>>>>
>>>> Hi *,
>>>>
>>>> this is not new to those of you who attended my talk on the F2F
>>>> 2016 in Brno. Let me explain the idea here again for all others who
>>>> did not have a chance to be there.
>>>>
>>>> Srcdeps [1] is a tool to build Maven dependencies from their
>>>> sources. With srcdeps, wildfly-core can depend on a specific commit
>>>> of, e.g., undertow:
>>>>
>>>> <version.io.undertow>1.4.8.Final-SRC-revision-aabbccd</version.io.undertow>
>>>>
>>>>
>>>>
>> where aabbccd is the git commit id to build when any undertow artifact
>>>> is requested during the build of wildfly-core.
>>>>
>>>> [1] describes in detail, how it works.
>>>>
>>>> The main advantage of srcdeps is that changes in components can be
>>>>   integrated and tested in wildfly-core immediately after they are
>>>> committed to a public component branch. There is no need to wait
>>>> for the component release.
>>>>
>>>> Here in the WildFly family of projects, it is often the case that
>>>> something needs to be fixed in a component, but the verification
>>>> (using bug reproducer, or integration test) is possible only at the
>>>> level of wildfly or wildfly-core. Engineers typically work with
>>>> snapshots locally, but when their changes need to get shared (CI,
>>>> reviews) in a reproducible manner, snapshots cannot be used
>>>> anymore. In such situations a source dependency come in handy: it
>>>> is very easy to share and it is as reproducible as a Maven build
>>>> from a specific commit can be. All CIs and reviewers can work with
>>>> it, because all source dependency compilation is done under the
>>>> hood by Maven.
>>>>
>>>> Developers working on changes that span over multiple
>>>> interdependent git repos can thus get feedback (i-tests, reviews)
>>>> quickly without waiting for releases of components.
>>>>
>>>> Srcdeps emerged in the Hawkular family of projects to solve exactly
>>>> this kind of situation and is in use there since around October
>>>> 2015.
>>>>
>>>> When I said there is no need to wait for releases of components, I
>>>> did not mean that we can get rid of component releases altogether.
>>>> Clearly, we cannot, because i.a. for any tooling uninformed about
>>>> how srcdeps work, those source dependencies would simply be
>>>> non-resolvable from public Maven repositories. So, before releasing
>>>> the dependent component (such as wildfly-core) all its dependencies
>>>> need to be released. To enforce this, srcdeps is by default
>>>> configured to make the release fail, as long as there are source
>>>> dependencies.
>>>>
>>>> I have sent a PR introducing srcdeps to wildfly-core:
>>>> https://github.com/wildfly/wildfly-core/pull/2122 To get a feeling
>>>> how it works, checkout the branch, switch to e.g.
>>>> <version.io.undertow>1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd</version.io.undertow>
>>>>   (that happens to be the commit id of the 1.4.8.Final tag) and
>>>> build wildfly-core as usual with "mvn clean install". You'll see in
>>>>   the build log that undertow is being cloned to
>>>> ~/.m2/srcdeps/io/undertow and that it is built there. After the
>>>> build, check that the
>>>> 1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd
>>>> version of Undertow got installed to your local Maven repo (usually
>>>>   ~/m2/repository/io/undertow/undertow-core )
>>>>
>>>> Are there any questions or comments?
>>>>
>>>> [1] https://github.com/srcdeps/srcdeps-maven#srcdeps-maven
>>>>
>>>> Thanks,
>>>>
>>>> Peter
>>>>
>>>> P.S.: I will be talking about srcdeps on Saturday 2017-01-28 at
>>>> 14:30 at DevConf Brno.
>>>> _______________________________________________ wildfly-dev mailing
>>>> list wildfly-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>> _______________________________________________
>> wildfly-dev mailing list
>> wildfly-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>