[jbosstools-dev] Generating Jenkins jobs from template (was Re: Building the server component)

[Nick Boldt]

Sounds reasonable in theory, but not all our jobs do the same thing. 
Consider these 9 different templates:

https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/jbosstools-build.parent_master/

https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/jbosstoolstargetplatforms-matrix/

https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/jbosstools-server_master/ 
(this template is reused for 15 or so component jobs, each which 
different scheduled github checks to ensure that jobs will (in theory) 
fire in the correct sequence rather than all at once)

https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/jbosstools-composite-install_master/

https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/jbosstools-build-sites.aggregate.site_master/ 
(this template is reused for building the webtools agg site and the 
coretests agg site, but each one has different upstream/downstream linkages)

https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/devstudio.product_master/

https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/devstudio.versionwatch_master/

https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/jbosstools-discovery_master/

https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/jbosstools-install-grinder.install-tests.matrix_master/

When it's time to update all the jobs to pull from Alpha2x branch 
instead of Alpha1x branch, sure, you could regenerate the 15 component 
jobs but you'd still have to go find+replace strings in the other 10 
jobs' config.xml files, and push those to the server. And if you have to 
find+replace for 10 jobs, why not do it for all 25, rather than having 
2/3rds generated and 1/3rd manually updated?

---

That said, it might be possible to parameterize a job to be able to push 
in a value of "fork" and "branch" so that rather than always building 
the job from jbosstools/jbosstools-<project> from origin/master or 
origin/jbosstools-4.1.1.Alpha2x, you could instead pass in 
fork=maxandersen&branch=origin/JBIDE-12345, which would run the job in 
an override mode to build using your fork and topic branch. Then of 
course we'd need a third parameter to define where to put this forked 
build results so it wouldn't conflict w/ the main composite site & 
downstream aggregate assemblies.

So instead of the usual /builds/staging/CI/JOB_NAME/BUILD_ID/ we might 
instead publish to /builds/staging/forks/JOB_NAME/BUILD_ID_gituser_branch

We could then run the new composite*.xml generator I've started here [1] 
to composite your forks into the larger CI composite, in order to test 
assembly and installation.

[1] https://issues.jboss.org/browse/JBIDE-15483, 
https://issues.jboss.org/browse/JBIDE-15482

WDYT? Crazytalk? Reasonable?

N

On 09/25/2013 02:20 PM, Max Rydahl Andersen wrote:
> On Wed, Sep 25, 2013 at 08:30:44AM +0200, Mickael Istria wrote:
>> On 09/24/2013 09:47 PM, Denis Golovin wrote:
>>> 1. More jenkins builds
>> We already have many jobs on Jenkins and the reactivity is not always
>> very good. Adding 30 to 40 more jobs will not make it better.
>> Despite of that, there is nothing preventing us from doing this.
>> Please open a Jira.
>
> I really would like to see us use something like https://github.com/maxandersen/buildchow
> or maybe even the literatebuild approach http://jenkins-ci.org/content/literate-builds-wtf which
> would allow for defining builds more 'declaratively' and easier to handle branches and allow
> one to reproduce our jenkins build setup locally and not be 117% tied to our
> jenkins instances.
>
> The number of jobs currently handedited or search/replaced and then sometimes edited on the server
> is making it very hard to share/move around to do things ike this "test run a PR".
>
>>> 2. Have no Idea how to deal with PR's related to each other for
>>> different components
>
> Basically these just won't be able to do with this approach imo.
> But how many jobs do you actually have that needs this ?
>
>> It's concretely impossible to make sure a PR 123 of server builds
>> against what was suggested in PR 57 of base, for the following reason:
>> * PR123 has no knowledge that it depends on another PR from another
>> component. So there is no way to automatically choose a specific build
>> output for another component.
>> * Job triggering is not deterministic since it depends on availability
>> of slaves, number of pull requests in the pipe, job duration... So we
>> can't rely on assumptions such as "PR were pushed at the same time, so
>> they'll build fine".
>
> Even you could be deterministic - the jobs using a PR should not be publsihing
> their result anywhere for downstream usage since it has been merged in yet.
>
>> So in that case of linked PR, the output of automated build for PR
>> suggested above would be: green for PR57, red for PR123. The process
>> would be to merge PR57 first, build it with the regular job, and then
>> re-run build for PR123.
>
> yes.
> _______________________________________________
> jbosstools-dev mailing list
> jbosstools-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/jbosstools-dev
>

-- 
Nick Boldt :: JBoss by Red Hat
Productization Lead :: JBoss Tools & Dev Studio
http://nick.divbyzero.com