[jbosstools-issues] [JBoss JIRA] (JBIDE-14017) cache downloaded zips fetched as part of building aggregate site for reuse across Jenkins slaves (rather than in each slave's workspace)

Wed Apr 17 05:31:53 EDT 2013

    [ https://issues.jboss.org/browse/JBIDE-14017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767949#comment-12767949 ] 

Mickael Istria commented on JBIDE-14017:
----------------------------------------

[~maxandersen] I think you're missing the big picture of this mojo. Here is what it does and how it does.
The goal is to provide along with the aggregator the set of sites that we used to aggregate (as zips). These zipped repo are currently expected to be in the zips/ folder for further publication.
To do so, Mojo does the following:
* Looks at a component site (based on our convention) to check which zip to get, and the md5 of the zip
* looks inside the ouputFolder (zips/ folder) to check whether the right zip is already here.
* If not found, look inside the zipCacheFolder (when set) to see whether zip exists with the right signature and copies it to the outputFolder if found.
* If not found, fetches the zip and then install it in cache for further usage (if zipCacheFolder is specified)

> cache downloaded zips fetched as part of building aggregate site for reuse across Jenkins slaves (rather than in each slave's workspace)
> ----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: JBIDE-14017
>                 URL: https://issues.jboss.org/browse/JBIDE-14017
>             Project: Tools (JBoss Tools)
>          Issue Type: Bug
>          Components: Build/Releng, updatesite
>    Affects Versions: 4.1.0.Alpha2
>            Reporter: Nick Boldt
>            Assignee: Mickael Istria
>             Fix For: 4.1.0.Beta1
>
>
> {quote}
> (2013-04-12 12:19:29) mistria: nickboldt: aggregate could compare logs
> (2013-04-12 12:19:35) nickboldt: mistria: which logs?
> (2013-04-12 12:20:58) mistria: nickboldt: the one we generate when using "fetch-zips"
> (2013-04-12 12:22:40) nickboldt: so you want the aggregate job to fetch the zips BEFORE it builds the update site to determine if it should build the update site?
> (2013-04-12 12:24:05) nickboldt: whole point of a "should I build?" check is that it happens before the build... eg., check SVN, check git, or run jbosstools-composite-install_master to check the installed IUs installed from the composite to determine if contents or versions have changed
> (2013-04-12 12:24:46) nickboldt: not adverse to the idea of splitting the "fetch zips" step out of the "build aggregate" job, but ... then we have 2 more jobs to worry about 
> (2013-04-12 12:24:53) mistria: nickboldt: actually, the new version of the fetch mojo won't re-fetch zips if it finds out the the signature of the to-be-downloaded zip is the same as the one it has in cache
> (2013-04-12 12:25:05) nickboldt: cache in workspace?
> (2013-04-12 12:25:06) mistria: nickboldt: so no change, not fetch
> (2013-04-12 12:25:23) nickboldt: or in static jenkins share folder?
> (2013-04-12 12:25:23) mistria: nickboldt: in aggregate/site/zips
> (2013-04-12 12:25:36) nickboldt: great, so it's one fetch per slave :(
> (2013-04-12 12:26:14) nickboldt: would be better to cache that outside the workspace, if building in Jenkins
> (2013-04-12 12:26:16) mistria: nickboldt: right, we should provide a way to override this directory and put in in a shared folder
> (2013-04-12 12:27:12) mistria: nickboldt: there is not system property for that yet... https://github.com/jbosstools/jbosstools-maven-plugins/blob/master/tycho-plugins/repository-utils/src/main/java/org/jboss/tools/tycho/sitegenerator/FetchZipsFromRepo.java#L60
> (2013-04-12 12:28:21) nickboldt: Old way: https://github.com/jbosstools/jbosstools-build-sites/blob/jbosstools-4.1.0.Alpha2x/aggregate/build.xml#L510 and https://github.com/jbosstools/jbosstools-build-sites/blob/jbosstools-4.1.0.Alpha2x/aggregate/build.xml#L33 (but we still dumped to $\{output.dir\}/zips, not to \{WORKINGDIR\}/zips )
> (2013-04-12 12:28:58) nickboldt: mistria: should allow caching files in Jenkins's static file tree for use across slaves
> {quote}
> So, I guess all we need to do is expose *outputFolder* to be overridden when -Phudson is enabled.
> {code:title=https://github.com/jbosstools/jbosstools-maven-plugins/blob/master/tycho-plugins/repository-utils/src/main/java/org/jboss/tools/tycho/sitegenerator/FetchZipsFromRepo.java#L60}
> 	/**
> * Location where to put zips
> * @parameter default-value="${basedir}/zips"
> */
> private File outputFolder;
> {code}
> We could even set the correct path in the parent pom's hudson profile so that it need not be set in all the various jobs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira