[wildfly-dev] Srcdeps in WildFly and WildFly Core

Peter Palaga ppalaga at redhat.com
Fri May 26 05:52:40 EDT 2017 

Hi *,

I was able to finish the missing parts (partly thanks to Carlo and Fedor 
recently asking me about running CI against a WildFly branch with a 
custom, non-released WildFly Core).

* The failWithout configuration option promised in (3) below was 
implemented in srcdeps 3.1.0.

* I have updated the PR for WildFly Core 
https://github.com/wildfly/wildfly-core/pull/2122 accordingly.

* Have submitted a new PR for WildFly 
https://github.com/wildfly/wildfly/pull/10116

* In both named PRs, the source dependencies are opt-in: a build that 
contains source dependencies will only succeed if mvn is invoked with 
-Dsrcdeps.enabled .

* Once the PRs mentioned above get accepted, the CI jobs that build PRs 
should be modified to contain the -Dsrcdeps.enabled mvn parameter.

Is there anything more I need to do to get the above PRs merged?

* As for the task (4) "create a Maven Mojo to figure out for a given 
Maven project if it uses source dependencies", I have not done that yet 
and I'd like to discuss how to implement it. One option that comes to my 
mind is that the mojo would simply make the Maven invocation fail with a 
non zero return code. E.g.

   mvn srcdeps:assert-none

would print something like

   [Error] srcdeps:assert-none detected a source dependency
   org.groupId:artifactId:1.2.3-SRC-revision in module myModule

and exit with a non-zero return code in case there are source 
dependencies; otherwise, the build would succeed with no output.
I hope the mojo would be useful for the pull processor like this?

Thanks,

Peter


On 01/30/2017 04:37 PM, Peter Palaga wrote:
> Thanks everybody for the feedback!
>
> Let me try to sum up the results of the discussion so far:
>
> (1) Nobody except for me wants commits with source dependencies in
> stable branches (such as master or 2.x in wildfly-core). I fully accept
> that the ability to release anytime would go away with source dependency
> merges and I thus give up in this point as long as the "release anytime"
> requirement will be there.
>
> (2) There seems to exist some (enough?) agreement that source
> dependencies could be allowed in pull requests. Such pull requests would
> be there to allow fast feedback from the CI and reviewers, even before
> the release of the dependency. But given (1), such PRs would have to be
> upgraded to a proper dependency release before merging.
>
> (3) If nobody vetoes (2), I am going to implement a failWithout
> configuration option in srcdeps.yaml that will allow for making srcdeps
> resolution opt-in (e.g. via -Dsrcdeps.enabled) for those CI jobs that
> build from PRs but will keep all other srcdeps builds failing.
>
> Having (3) will not prevent merges of PRs with source dependencies
> directly, but will at least make the after-merge CI job fail so that the
> maintainer of the branch is informed very quickly that something bad
> happened.
>
> (4) I'd also add some sort of code (probably a Maven Mojo) to srcdeps
> that would allow to figure out for a given Maven project if it uses
> source dependencies. This could be used to label PRs so that the
> gatekeeper sees clearly that the given PR has source dependencies.
>
> Thanks,
>
> Peter
>
>
> On 2017-01-26 19:11, Brian Stansberry wrote:
>>
>>> On Jan 26, 2017, at 10:28 AM, Peter Palaga <ppalaga at redhat.com> wrote:
>>>
>>> On 2017-01-26 15:44, Brian Stansberry wrote:
>>>> There’s been a lot of discussion overnight, but I’ll reply to this
>>>> one directly since my answers better align with your questions here. :)
>>>>
>>>>> On Jan 26, 2017, at 2:54 AM, Peter Palaga <ppalaga at redhat.com> wrote:
>>>>>
>>>>> Hi Brian, thanks for your comments, more inline...
>>>>>
>>>>> On 2017-01-26 02:02, Brian Stansberry wrote:
>>>>>> My only concerns with this would relate to comitting this kind of src
>>>>>> dependency to the poms in the main branches in the widlfly/wildfly
>>>>>> and wildfly/wildfly-core repos. We’ve managed to survive up to now
>>>>>> with little or no need for that kind of thing, so until we get used
>>>>>> to using this in other ways IMHO we should follow the KISS principle
>>>>>> and forbid that.
>>>>>
>>>>> Maybe I overestimate the amount of changes that span over multiple
>>>>> git repos. Maybe you in the Core team do not do this often. But for
>>>>> us in the Sustaining Engineering Team, this is quite a typical
>>>>> situation. A substantial part of the reports from customers come
>>>>> with a description how to reproduce on the whole server, but they
>>>>> need to be fixed in a component. Having srcdeps would make the CP
>>>>> process simpler and faster, allowing us to uncover the conflicts
>>>>> and regressions earlier.
>>>>
>>>> I don’t see how merging to the main branches is required to get this
>>>> benefit. Git topic branches are fully sharable and CI jobs against
>>>> them are easily done. All CI tests of pull requests are tests of
>>>> topic branches.
>>>
>>> Yes, for me as the submitter of the PR, it is nice to get the
>>> feedback from the CI and a review early, even before the component is
>>> released, but it is quite bothersome to have to revisit the PR again
>>> once the component gets released and rebase (in case there there are
>>> conflicts) and either upgrade to the released component version or
>>> remove the upgrade change (if the upgrade was merged separately).
>>>
>>> As long as my PR is not merged, my changes are not binding for the
>>> rest of the team. I want my PR to get merged as fast as possible and
>>> make others care that their changes are compatible with mine. I want
>>> to happily forget about the PR as soon as possible and pick a new
>>> task :)
>>>
>>
>> You’ve convinced me! Convinced me that we shouldn’t allow this. :D
>>
>> We don’t have a role analogous to the “release coordinator” used with
>> EAP CPs, i.e. someone whose primary responsibility is coordinating to
>> make sure that the untidy pieces get tidied. Most of our non-CR/Final
>> releases are done as side tasks by people who are stealing time from
>> other tasks. They need to be simple and mechanical. We also have a far
>> greater volume of changes to manage than EAP CPs do. A process based
>> on merging half the necessary change and then letting the issue owner
>> walk away and assume someone else is going to come tidy up is a recipe
>> for disaster.
>>
>>>> But, in any case perhaps you’ve seen clear need for merging to the
>>>> main branches with the EAP CP branches. I haven’t seen it in WildFly
>>>> / WildFly Core. I deliberately used specific repo names in my last
>>>> comment to try and scope it. ;)
>>>
>>> My reasons for merging there in EAP CP branches are the same as here
>>> in the community branches: it is better for PR submitters to merge as
>>> early as possible to avoid conflicts, subsequent PR edits and to keep
>>> the list of open tasks short.
>>>
>>>> Note I’m not saying we should disallow PRs with src deps in the pom.
>>>> We should just disallow merging until those are replaced.
>>>
>>> Yes, I understand that and I appreciate that. That would be a
>>> progress too.
>>>
>>>>>> A trick is avoiding doing that by mistake; i.e. a PR is sent up with
>>>>>> a SRC dependency to get CI or review and accidentally gets merged.
>>>>>
>>>>> Oh, I am just realizing I have not said anything about merging. I
>>>>> actually do want to propose that commits with source dependencies
>>>>> get merged to e.g. wildfly-core master as early as possible. Those
>>>>> are the key points of Continuous Integration: get feedback quickly,
>>>>> and merge as soon as possible. This is exactly what Hawkular is
>>>>> doing since more than a year.
>>>>
>>>> We regularly produce releases (ideally weekly for WildFly Core),
>>>> often at short notice under pressure.  Allowing merging of changes
>>>> that are not acceptable for release increases the risk and effort
>>>> required to do that, since now we have to scan for src deps and
>>>> figure out how to get them out of the build. Perhaps needing
>>>> assistance from whoever added the src dep and the lead of relevant
>>>> component, both of whom are on the other side of the world asleep.
>>>> (This is a real issue since we often do releases on Friday afternoon
>>>> US time or Monday morning European time.) We already have too much
>>>> risk and effort doing releases so adding more will need a really
>>>> strong justification.
>>>
>>> This sounds as a valid concern. I must admit I know little about how
>>> you plan and perform the releases of wildfly-core, wildfly and of the
>>> components in the community. Knowing how complex the graph of WF
>>> components is, I am far from underestimating any manual release
>>> efforts or efforts to setup a CI jobs to do that automagically. I'll
>>> have to gather more info about how you work.
>>>
>>>>>> But I suppose that’s not the end of the world, so long as the release
>>>>>> process will eventually detect it and fail.
>>>>>
>>>>> Yes, source dependencies on a stable branch do not harm. They just
>>>>> need to be avoided in releases (for which srcdeps offers technical
>>>>> means).
>>>>
>>>> They do do harm as they mean the branch is no longer releasable.
>>>> It’s not end-of-the-world harm but it’s harm.
>>>
>>> Well, I naivelly thought, that the components are obligated to
>>> provide a release, say, one day before a planned wildfly-core release
>>> and send a PRs that would then sweep out all source dependencies. And
>>> TBH, I did not think "releasable at any time" is important in
>>> wildfly-core. "Releasable once a week" still sounds good enough to me :)
>>
>> Unfortunately, it’s not.
>>
>>>
>>>>>> Can making srcdeps fail (or just disabling it) be turned on via a
>>>>>> maven profile? With that we could set up such a profile and turn it
>>>>>> on in CI jobs that are testing branches where it’s forbidden (e.g.
>>>>>> the nightly builds of master.)
>>>>>
>>>>> Yes, the feature is called "failWith profiles" and can be
>>>>> configured in .mvn/srcdeps.yaml, like here in this srcdeps
>>>>> quickstart:
>>>>> https://github.com/srcdeps/srcdeps-maven/blob/master/srcdeps-maven-quickstarts/srcdeps-mvn-git-profile-quickstart/.mvn/srcdeps.yaml#L33
>>>>>
>>>>> There is also "failWith properties" and "failWith goals". It is
>>>>> documented here:
>>>>> https://github.com/srcdeps/srcdeps-core/blob/master/doc/srcdeps.yaml#L130
>>>>>
>>>>> By default there is failWith: {goals: release:prepare,
>>>>> release:perform}. Projects that do not use the release plugin can
>>>>> set e.g. failWith: {goals: deploy:deploy} or whatever else
>>>>> distinguishes their releases.
>>>>>
>>>>
>>>> Thanks.
>>>>
>>>>>> Oh, one other concern — how robust is this in the face of poor
>>>>>> maintenance? I see a lot of boilerplate in that .mvn/srcdeps.yaml.
>>>>>
>>>>> Which parts are boilerpate?
>>>>
>>>> All of it. :)
>>>>
>>>> I’m not using that word as an attack. I’m just saying it’s extra
>>>> text that needs to be maintained, and since it’s separate from the
>>>> usual place similar text occurs (the poms) it is more likely to
>>>> diverge.
>>>
>>> OK, now I know what you mean :) You are right that poms can diverge
>>> from srcdeps.yaml.
>>>
>>>>>> If
>>>>>> that gets out of date or something is the only effect that using a
>>>>>> src dependency for the affected item doesn't work?
>>>>>
>>>>> Yes, I think so. As long as the .mvn/srcdeps.yaml file is
>>>>> syntactically correct, any misconfiguration there should not have
>>>>> any other effect than eventually breaking an embedded build.
>>>>>
>>>>> Generally, the things configured in .mvn/srcdeps.yaml tend to be
>>>>> quite stable - it is basically just mapping from GAVs to their
>>>>> respective git URLs. Git URLs do not change often. It is true that
>>>>> dependency artifacts come and go, but as long as their groupIds are
>>>>> selected reasonably (one groupId occurs in not more than one git
>>>>> repo) the mapping itself can be quite stable over time too.
>>>>
>>>> Yeah, that’s true. Where this file would be more likely to go
>>>> unmaintained is adding new entries or cleaning out old ones. But the
>>>> latter is just noise and if the only harm of the former is a srcdep
>>>> can’t be used for that lib, then that will naturally get handled by
>>>> whoever wants to use the srcdep.
>>>
>>> Yes, exactly.
>>>
>>> Thanks,
>>>
>>> Peter
>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Peter
>>>>>
>>>>>>
>>>>>>> On Jan 25, 2017, at 3:45 PM, Peter Palaga <ppalaga at redhat.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi *,
>>>>>>>
>>>>>>> this is not new to those of you who attended my talk on the F2F
>>>>>>> 2016 in Brno. Let me explain the idea here again for all others who
>>>>>>> did not have a chance to be there.
>>>>>>>
>>>>>>> Srcdeps [1] is a tool to build Maven dependencies from their
>>>>>>> sources. With srcdeps, wildfly-core can depend on a specific commit
>>>>>>> of, e.g., undertow:
>>>>>>>
>>>>>>> <version.io.undertow>1.4.8.Final-SRC-revision-aabbccd</version.io.undertow>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>> where aabbccd is the git commit id to build when any undertow artifact
>>>>>>> is requested during the build of wildfly-core.
>>>>>>>
>>>>>>> [1] describes in detail, how it works.
>>>>>>>
>>>>>>> The main advantage of srcdeps is that changes in components can be
>>>>>>> integrated and tested in wildfly-core immediately after they are
>>>>>>> committed to a public component branch. There is no need to wait
>>>>>>> for the component release.
>>>>>>>
>>>>>>> Here in the WildFly family of projects, it is often the case that
>>>>>>> something needs to be fixed in a component, but the verification
>>>>>>> (using bug reproducer, or integration test) is possible only at the
>>>>>>> level of wildfly or wildfly-core. Engineers typically work with
>>>>>>> snapshots locally, but when their changes need to get shared (CI,
>>>>>>> reviews) in a reproducible manner, snapshots cannot be used
>>>>>>> anymore. In such situations a source dependency come in handy: it
>>>>>>> is very easy to share and it is as reproducible as a Maven build
>>>>>>> from a specific commit can be. All CIs and reviewers can work with
>>>>>>> it, because all source dependency compilation is done under the
>>>>>>> hood by Maven.
>>>>>>>
>>>>>>> Developers working on changes that span over multiple
>>>>>>> interdependent git repos can thus get feedback (i-tests, reviews)
>>>>>>> quickly without waiting for releases of components.
>>>>>>>
>>>>>>> Srcdeps emerged in the Hawkular family of projects to solve exactly
>>>>>>> this kind of situation and is in use there since around October
>>>>>>> 2015.
>>>>>>>
>>>>>>> When I said there is no need to wait for releases of components, I
>>>>>>> did not mean that we can get rid of component releases altogether.
>>>>>>> Clearly, we cannot, because i.a. for any tooling uninformed about
>>>>>>> how srcdeps work, those source dependencies would simply be
>>>>>>> non-resolvable from public Maven repositories. So, before releasing
>>>>>>> the dependent component (such as wildfly-core) all its dependencies
>>>>>>> need to be released. To enforce this, srcdeps is by default
>>>>>>> configured to make the release fail, as long as there are source
>>>>>>> dependencies.
>>>>>>>
>>>>>>> I have sent a PR introducing srcdeps to wildfly-core:
>>>>>>> https://github.com/wildfly/wildfly-core/pull/2122 To get a feeling
>>>>>>> how it works, checkout the branch, switch to e.g.
>>>>>>> <version.io.undertow>1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd</version.io.undertow>
>>>>>>>
>>>>>>> (that happens to be the commit id of the 1.4.8.Final tag) and
>>>>>>> build wildfly-core as usual with "mvn clean install". You'll see in
>>>>>>> the build log that undertow is being cloned to
>>>>>>> ~/.m2/srcdeps/io/undertow and that it is built there. After the
>>>>>>> build, check that the
>>>>>>> 1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd
>>>>>>> version of Undertow got installed to your local Maven repo (usually
>>>>>>> ~/m2/repository/io/undertow/undertow-core )
>>>>>>>
>>>>>>> Are there any questions or comments?
>>>>>>>
>>>>>>> [1] https://github.com/srcdeps/srcdeps-maven#srcdeps-maven
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Peter
>>>>>>>
>>>>>>> P.S.: I will be talking about srcdeps on Saturday 2017-01-28 at
>>>>>>> 14:30 at DevConf Brno.
>>>>>>> _______________________________________________ wildfly-dev mailing
>>>>>>> list wildfly-dev at lists.jboss.org
>>>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>>>>>
>>>>>
>>>>
>>>
>>
>

More information about the wildfly-dev mailing list