[jbosstools-issues] [JBoss JIRA] (JBTIS-498) improve CI/snapshot/staging publishing of discovery jars to allow automatic self-cleaning

[Nick Boldt (JIRA)]

    [ https://issues.jboss.org/browse/JBTIS-498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13110228#comment-13110228 ] 

Nick Boldt commented on JBTIS-498:
----------------------------------

Looking through your https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/JBTIS-aggregate-disc/ job and I'm seeing a lot we can do to streamline and automate. 

0. You're pulling the rsync script directly from github; we now prefer to fetch the zipped versions from nexus so you can see which version you're getting.

I've updated your job here:

https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/JBTIS-aggregate-disc/jobConfigHistory/showDiffFiles?timestamp1=2015-09-15_15-14-37&timestamp2=2015-09-17_17-28-27

1. Why do you have 2 jobs for JBTIS-aggregate-disc and JBDS-aggregate-disc ? why not build them at the same time from the same job? 

2. Why do you rsync the contents of sources/BUILD_KIND/site-final/target/ [1] AND its child folder sources/BUILD_KIND/site-final/target/repository [2] into the same target folder on dl.jb.org [3]? You're essentially publishing the same repo twice, plus you're publishing a lot of maven metadata  that you (probably) will never need.

[1] http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/JBTIS-aggregate-disc/ws/sources/jbosstools/site-final/target/
[2] http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/JBTIS-aggregate-disc/ws/sources/jbosstools/site-final/target/repository
[3] http://download.jboss.org/jbosstools/mars/snapshots/builds/integration-stack/4.3.0.Alpha2/

Do you do anything with category.xml? checkstyle*? local-artifacts? p2agent? p2artifacts? p2content? targetPlatformRepository? or the site*.zip?

3. As to the use of the --del flag... I think the problem is that you're pushing a single folder's contents into a single folder. When I use rsync.sh --del, it's to push a WHOLE folder and replace whatever is already there.

So:

{code:title=http://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/DevStudio/view/DevStudio_9.0.mars/job/jbosstools-discovery_master/configure-readonly/}
. ${WORKSPACE}/sources/publish/rsync.sh --del -s ${WORKSPACE}/sources/jbosstools/org.jboss.tools.central.discovery.earlyaccess/target/discovery-site \
  -t mars/snapshots/updates/discovery.earlyaccess/${jbosstools_site_stream}/
{code}

becomes
{code:title=http://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/DevStudio/view/DevStudio_9.0.mars/job/jbosstools-discovery_master/lastBuild/console}
++ echo '[INFO] rsync -arzq --protocol=28  --del /mnt/hudson_workspace/workspace/jbosstools-discovery_master/sources/jbosstools/org.jboss.tools.central.discovery.earlyaccess/target/discovery-site/* tools at filemgmt.jboss.org:/downloads_htdocs/tools/mars/snapshots/updates/discovery.earlyaccess/master//'
{code}

and results in this:

http://download.jboss.org/jbosstools/mars/snapshots/updates/discovery.earlyaccess/master/

Alternatively, if you use a versioned folder, you'll get an automated composite site and automated cleanup -- no --del flag needed!

{code}
. ${WORKSPACE}/sources/publish/rsync.sh -s ${WORKSPACE}/sources/jbdevstudio/com.jboss.jbds.central.discovery/target/discovery-site \
  -t 9.0/snapshots/builds/jbosstools-discovery.central_${jbosstools_site_stream}/${BUILD_ID}-B${BUILD_NUMBER}/all/repo/ \
  -DESTINATION devstudio at filemgmt.jboss.org:/www_htdocs/devstudio
{code}

results in:

https://devstudio.redhat.com/9.0/snapshots/builds/jbosstools-discovery.central_master/
and
https://devstudio.redhat.com/9.0/snapshots/builds/jbosstools-discovery.central_master/compositeContent.xml
and
https://devstudio.redhat.com/9.0/snapshots/builds/jbosstools-discovery.central_master/latest/all/repo/

So... if you can change your method of publishing such that the site you produce in Jenkins is the same as the site you put on dl.b.o or ds.jb.c (without needing a dozen individual rsync/copy/move steps) not only will your local builds == your CI builds, but your release process will be simpler.

4. Why does your CI build sometimes ALSO push bits to /staging/ as well as /snapshots/ ? Surely that promotion step should ONLY ever happen by hand when it's time to do a QE staging release? 

5. Why are there two copies of the same directory.xml file?

https://devstudio.redhat.com/9.0/staging/updates/integration-stack/discovery/9.0.0.Alpha2/plugins/devstudio-integration-stack-directory.xml (doesn't work, as it refers to a child plugins/ folder)
and
https://devstudio.redhat.com/9.0/staging/updates/integration-stack/discovery/9.0.0.Alpha2/devstudio-integration-stack-directory.xml (works, as it refers to the child plugins/ folder)

Hope that helps! 

> improve CI/snapshot/staging publishing of discovery jars to allow automatic self-cleaning
> -----------------------------------------------------------------------------------------
>
>                 Key: JBTIS-498
>                 URL: https://issues.jboss.org/browse/JBTIS-498
>             Project: JBoss Tools Integration Stack
>          Issue Type: Feature Request
>          Components: distribution
>            Reporter: Nick Boldt
>            Assignee: Paul Leacu
>
> Would it be possible to change your publishing process so that you're not dumping dozens of old jars into the same folder? Surely you aren't staging ALL these? (We only stage to QE once or twice per milestone... CR1, CR1a.)
> Have a look here:
> http://download.jboss.org/jbosstools/mars/staging/updates/integration-stack/discovery/4.3.0.Alpha1/plugins/
> https://devstudio.redhat.com/9.0/staging/updates/integration-stack/discovery/9.0.0.Alpha1/plugins/
> And contrast that with the way the JBT & JBDS discovery jars are done:
> http://download.jboss.org/jbosstools/mars/snapshots/builds/jbosstools-discovery.central_master/
> https://devstudio.redhat.com/9.0/snapshots/builds/jbosstools-discovery.central_master/
> (using rsync.sh, you get automated build folder cleanup every time you publish a new build, or you can force cleanup it with this job [1]).
> [1] https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/jbosstools-cleanup/



--
This message was sent by Atlassian JIRA
(v6.4.11#64026)