[wildfly-dev] Srcdeps in WildFly and WildFly Core

Peter Palaga ppalaga at redhat.com
Mon Jan 30 10:36:18 EST 2017 

Hi Darran, inline...

On 2017-01-26 13:56, Darran Lofthouse wrote:
>
>
> On 26/01/17 12:42, Peter Palaga wrote:
>> Thanks for the comments, Darran, more inline...
>>
>> On 2017-01-26 12:40, Darran Lofthouse wrote:
>>> On 26/01/17 01:02, Brian Stansberry wrote:
>>>> My only concerns with this would relate to comitting this kind of src
>>>> dependency to the poms in the main branches in the widlfly/wildfly
>>>> and wildfly/wildfly-core repos. We’ve managed to survive up to now
>>>> with little or no need for that kind of thing, so until we get used
>>>> to using this in other ways IMHO we should follow the KISS principle
>>>> and forbid that.
>>>
>>> I like the idea of this being available for teams working on a topic
>>> branch but also think we should avoid this kind of dependency in master
>>> - engineers are continually running builds so I don't think we should
>>> add additional source checkouts and builds.
>>
>> So you also vote for having the support for source dependencies when the
>> e.g. the wildfly-core CI builds a PR that integrates a change in Elytron
>> or when a reviewer does the same, but such PRs should wait for a proper
>> release of the component, right?
>
> I think by the time we get to the PR stage engineers should have had an
> opportunity to discuss the proposed solution sufficiently that we
> shouldn't need a srcdep and should have been able to tag Elytron.

OK, thanks for explaining!

>>> I know Elytron has caused huge amounts of instability but generally I
>>> think we want these branches in a state where we could tag if we
>>> needed to.
>>
>> "these branches" - you mean topic branches in a component (such as
>> Elytron) or in a consuming project (such as WF Core) or both?
>
> I think WildFly Core / master and WildFly / master are generally not
> supposed to be as unstable as we have made them recently.
>
>>> When these downloaded builds are executed are the test cases within also
>>> executed?
>>
>> The builds of source dependencies run with -DskipTests by default so
>> that they finish faster. An explicit per git repository setting in
>> srcdeps.yaml is needed if somebody wants a dependency build to run
>> without -DskipTests.
>>
>>>> A trick is avoiding doing that by mistake; i.e. a PR is sent up with
>>>> a SRC dependency to get CI or review and accidentally gets merged.
>>>> But I suppose that’s not the end of the world, so long as the release
>>>> process will eventually detect it and fail.
>>>
>>> Merging topic branches these temporary SHAs would be in the history but
>>> maybe the PR CI runs could verify they are not present in the final
>>> merge.
>>>
>>>> Can making srcdeps fail (or just disabling it) be turned on via a
>>>> maven profile? With that we could set up such a profile and turn it
>>>> on in CI jobs that are testing branches where it’s forbidden (e.g.
>>>> the nightly builds of master.)
>>>>
>>>> Oh, one other concern — how robust is this in the face of poor
>>>> maintenance? I see a lot of boilerplate in that .mvn/srcdeps.yaml. If
>>>> that gets out of date or something is the only effect that using a
>>>> src dependency for the affected item doesn't work?
>>>
>>> I think that file would be better almost empty
>>
>> The size of the srcdeps.yaml file depends on how many dependencies the
>> given project has and which of those are selected for the srcdeps
>> support. In the PR https://git.io/vMjUC , I have included only the
>> projects owned by us, excluding non-Maven and non-git projects. I am
>> open to discussion whether we should include or exclude more projects.
>>
>>> but not hidden.
>>
>> The .mvn directory was introduced by Maven 3.3.1. Maven reads
>> extensions.xml file from there and I found it rather natural to have
>> srcdeps.yaml file there too.
>
> What that means is that a file that can directly influence the outcome
> of a local build is hidden from the engineer.  There are already quite a
> few things in out builds where when they go wrong we have to spend time
> tracking them down, having a hidden configuration may not help this.

If I heart this earlier, I'd most probably choose a non-hidden location. 
I perhaps do not see this as a problem, because I configure each of my 
Eclipse workspaces to show all .* resources because .mvn is not the only 
important file that is hidden: there is also .gitignore, .gitattributes, 
.travis.yml, .appveyor.yml and even .git and others that one wants to 
open from time to time.
But now that srcdeps has some users, sorry, I do not think I find this 
important enough to break backwards compatibility.

-- P

>>> Thinking about the topic branch development, within the Elytron team we
>>> are using some SNAPSHOTs from the official repos and at times we have
>>> forked projects into our incubator and developed against SNAPSHOTs from
>>> those
>>
>> Allowing SNAPSHOTs on any remote Maven repository leads to loosing
>> reproducibility of the builds of the dependent project. Srcdeps offers
>> much more reproducibility and portalbility over environments in such
>> situations.
>
> Yes I am saying in the development of the Elytron topic branches this
> could help a lot - but the assumptions as to which git repos we are
> using would be wrong.
>
>>> as as you select the SHA for the build you would also need to
>>> select which repository you really want.
>>
>> Yes, the mapping which dependencies are built from which git
>> repositories is defined in srcdeps.yaml. I see no problem in having one
>> git URL in one branch and having different git URL in another branch.
>
> Which I think as this is a file engineers could be editing to tweak
> their builds means it should not be hidden.
 >
>> Thanks,
>>
>> Peter
>>
>>>>> On Jan 25, 2017, at 3:45 PM, Peter Palaga <ppalaga at redhat.com> wrote:
>>>>>
>>>>> Hi *,
>>>>>
>>>>> this is not new to those of you who attended my talk on the F2F
>>>>> 2016 in
>>>>> Brno. Let me explain the idea here again for all others who did not
>>>>> have
>>>>> a chance to be there.
>>>>>
>>>>> Srcdeps [1] is a tool to build Maven dependencies from their sources.
>>>>> With srcdeps, wildfly-core can depend on a specific commit of, e.g.,
>>>>> undertow:
>>>>>
>>>>> <version.io.undertow>1.4.8.Final-SRC-revision-aabbccd</version.io.undertow>
>>>>>
>>>>>
>>>>>
>>>>> where aabbccd is the git commit id to build when any undertow artifact
>>>>> is requested during the build of wildfly-core.
>>>>>
>>>>> [1] describes in detail, how it works.
>>>>>
>>>>> The main advantage of srcdeps is that changes in components can be
>>>>> integrated and tested in wildfly-core immediately after they are
>>>>> committed to a public component branch. There is no need to wait for
>>>>> the
>>>>> component release.
>>>>>
>>>>> Here in the WildFly family of projects, it is often the case that
>>>>> something needs to be fixed in a component, but the verification
>>>>> (using
>>>>> bug reproducer, or integration test) is possible only at the level of
>>>>> wildfly or wildfly-core. Engineers typically work with snapshots
>>>>> locally, but when their changes need to get shared (CI, reviews) in a
>>>>> reproducible manner, snapshots cannot be used anymore.
>>>>> In such situations a source dependency come in handy: it is very
>>>>> easy to
>>>>> share and it is as reproducible as a Maven build from a specific
>>>>> commit
>>>>> can be. All CIs and reviewers can work with it, because all source
>>>>> dependency compilation is done under the hood by Maven.
>>>>>
>>>>> Developers working on changes that span over multiple interdependent
>>>>> git
>>>>> repos can thus get feedback (i-tests, reviews) quickly without waiting
>>>>> for releases of components.
>>>>>
>>>>> Srcdeps emerged in the Hawkular family of projects to solve exactly
>>>>> this
>>>>> kind of situation and is in use there since around October 2015.
>>>>>
>>>>> When I said there is no need to wait for releases of components, I did
>>>>> not mean that we can get rid of component releases altogether.
>>>>> Clearly,
>>>>> we cannot, because i.a. for any tooling uninformed about how srcdeps
>>>>> work, those source dependencies would simply be non-resolvable from
>>>>> public Maven repositories. So, before releasing the dependent
>>>>> component
>>>>> (such as wildfly-core) all its dependencies need to be released. To
>>>>> enforce this, srcdeps is by default configured to make the release
>>>>> fail,
>>>>> as long as there are source dependencies.
>>>>>
>>>>> I have sent a PR introducing srcdeps to wildfly-core:
>>>>> https://github.com/wildfly/wildfly-core/pull/2122
>>>>> To get a feeling how it works, checkout the branch, switch to e.g.
>>>>> <version.io.undertow>1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd</version.io.undertow>
>>>>>
>>>>>
>>>>> (that happens to be the commit id of the 1.4.8.Final tag)
>>>>> and build wildfly-core as usual with "mvn clean install". You'll
>>>>> see in
>>>>> the build log that undertow is being cloned to
>>>>> ~/.m2/srcdeps/io/undertow
>>>>> and that it is built there. After the build, check that the
>>>>> 1.4.8.Final-SRC-revision-1bff8c32f0eee986e83a7589ae95ebbc1d67d6bd
>>>>> version of Undertow got installed to your local Maven repo (usually
>>>>> ~/m2/repository/io/undertow/undertow-core )
>>>>>
>>>>> Are there any questions or comments?
>>>>>
>>>>> [1] https://github.com/srcdeps/srcdeps-maven#srcdeps-maven
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Peter
>>>>>
>>>>> P.S.: I will be talking about srcdeps on Saturday 2017-01-28 at
>>>>> 14:30 at
>>>>> DevConf Brno.
>>>>> _______________________________________________
>>>>> wildfly-dev mailing list
>>>>> wildfly-dev at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>>>
>>> _______________________________________________
>>> wildfly-dev mailing list
>>> wildfly-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>>
>>

More information about the wildfly-dev mailing list