[jbosstools-issues] [JBoss JIRA] (JBIDE-18772) Include publish.sh in parent pom as versioned maven dependency

[Nick Boldt (JIRA)]

    [ https://issues.jboss.org/browse/JBIDE-18772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052723#comment-13052723 ] 

Nick Boldt commented on JBIDE-18772:
------------------------------------

Please escape your \{ (with a \ ) so they don't break & become illegible in JIRA. 

-----

Recursive rsync is a nice idea but to allow renaming from source folder (repository) to target folder ($\{JOB_NAME}/$\{BUILD_ID}-B$\{BUILD_NUMBER}/all/repo/), we'd have to copy the output in the workspace FIRST, then push the relative path to the remote server:

{code}
mkdir -p $tmpdir/mars/snapshots/builds/$\{JOB_NAME}/$\{BUILD_ID}-B$\{BUILD_NUMBER}/all/repo
rsync -a ${project.build.directory}/repository/* $tmpdir/mars/snapshots/builds/$\{JOB_NAME}/$\{BUILD_ID}-B$\{BUILD_NUMBER}/all/repo/
pushd $tmpdir
rsync -a --relative mars/snapshots/builds/$\{JOB_NAME}/$\{BUILD_ID}-B$\{BUILD_NUMBER}/all/repo/ user at remote:/jbosstools/'
popd
{code}

Recursive sftp does the same thing but doesn't require lots of space on disk. I suppose we could move instead of copying, but then the expected build output in /target/repository/ won't be there so it's harder to diagnose build problems... much like you don't like having Jenkins assume all the github sources are in $\{WORKSPACE}/sources, I don't like having a workspace that's organized differently for Jenkins vs. local builds.

Aside: in your example you had both "updates" and "builds" in the same path. That should never happen -- we are no longer mixing concerns with JBDS-3208 reorg. I'm sure that was just a copy&paste error, but I wanted to make sure that was clearly stated.

-----

re: copy of the build log, I will poll jbosstools-dev and ask if anyone wants that kept. This step is skipped if there's no JOB_NAME (eg., for local builds)

-----

* create a symlink from this latest $\{JOB_NAME}/$\{BUILD_ID}-B$\{BUILD_NUMBER} folder to $\{JOB_NAME}/latest/
* create a composite site in $\{JOB_NAME}/ of all the */all/repo/ folders, and purge old published folders older than 2 days old

The first step is just one line of code:

{code}
# if TARGET_PATH contains a BUILD_ID-B# folder,
# create symlink: jbosstools-build-sites.aggregate.earlyaccess-site_master/latest -> jbosstools-build-sites.aggregate.earlyaccess-site_master/${BUILD_ID}-B${BUILD_NUMBER}
if [[ ${TARGET_PATH/${BUILD_ID}-B${BUILD_NUMBER}} != ${TARGET_PATH} ]]; then
  pushd $tmpdir >/dev/null; ln -s ${BUILD_ID}-B${BUILD_NUMBER} latest; rsync --protocol=28 -l latest ${DESTINATION}/${PARENT_PATH}/; rm -f latest; popd >/dev/null
fi
{code}

The second step is just a simple call to jbosstools-cleanup.sh with a specific folder path to clean.

Are you suggesting that we should have a deploy phase mojo that JUST does the copy of bits, and all these other steps would be a SECOND step in the job?

If so... what do we gain by splitting rsync.sh into a simply rsync call + some additional publishing/linking/cleanup? Surely the act of publishing should include the creation of composite*.xml and purging of week-only nigthlies, too?

-----

*    create a link to the published content from within the Jenkins job's BUILD_DESCRIPTION [not required but nice to have]

You want a separate shell script to just do this? All it does is produce a single link to the output build (if $\{URL} is specified):

{code}
# given TARGET_PATH=/downloads_htdocs/tools/mars/snapshots/builds/jbosstools-build-sites.aggregate.earlyaccess-site_master/2015-03-06_17-58-07-B13/all/repo/
PARENT_PATH=$(echo $TARGET_PATH | sed -e "s#/\?downloads_htdocs/tools/##" -e "s#/\?all/repo/\?##" -e "s#/\$##" -e "s#^/##" -e "s#\(.\+\)/[^/]\+#\1#")
LAST_SEGMENT=$(echo $TARGET_PATH | sed -e "s#/\?downloads_htdocs/tools/##" -e "s#/\?all/repo/\?##" -e "s#/\$##" -e "s#^/##" -e "s#\(.\+\)/\([^/]\+\)#\2#")
BUILD_DESCRIPTION='<li><a href='${URL}'/'${PARENT_PATH}'/'${LAST_SEGMENT}'>'${LAST_SEGMENT}'</a></li>'
{code}

We used to have a much more complex BUILD_DESCRIPTION with links to TPs used in the build, test coverage results, etc. I stripped it down to a single link to the build output folder. Is that STILL too much information?

-----

What I'm hearing is that Max feels that the deploy phase should ONLY copy bits from the Jenkins workspace to somewhere on some server (sftp path entirely configurable). But to make those jobs Just Work for publishing into the composites/aggregates/downstream, we at least need the latest/ symlinks to be created during the same build (so that they don't point to outdated content) and we need automated cleanup (so that we don't blow up download.jboss.org w/ 1000s of CI builds). 

-----

Re: the Take 2 solution... that looks OK but ONLY for project builds. Aggregate builds need TWO pushes [one into builds, one into updates], and the matrix job that handles webtools/central/EA must make sure it's passing in the correct paths, since the JOB_NAME doesn't include the name of the matrix config (eg., webtools-site, central-site, earlyaccess-site). 

> Include publish.sh in parent pom as versioned maven dependency
> --------------------------------------------------------------
>
>                 Key: JBIDE-18772
>                 URL: https://issues.jboss.org/browse/JBIDE-18772
>             Project: Tools (JBoss Tools)
>          Issue Type: Feature Request
>          Components: build
>            Reporter: Max Rydahl Andersen
>            Assignee: Mickael Istria
>             Fix For: 4.3.0.Alpha2
>
>
> instead of relying to publish.sh being on master, we should use a versioned publish.sh (or maybe even mojo) that the build then uses.
> suggestion:
> publish.sh (or mojo) gets released to our maven repo, use it in the pom.xml to perform publishing.
> What this helps with is:
> a) can do changes to publish mechanism without affecting every past builds.
> b) more movable build system
> c) isolated testing possible 
>  



--
This message was sent by Atlassian JIRA
(v6.3.11#6341)