[jbosstools-issues] [JBoss JIRA] (JBIDE-14017) cache downloaded zips fetched as part of building aggregate site for reuse across Jenkins slaves (rather than in each slave's workspace)

[Mickael Istria (JIRA)]

    [ https://issues.jboss.org/browse/JBIDE-14017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767945#comment-12767945 ] 

Mickael Istria commented on JBIDE-14017:
----------------------------------------

> how about jobs that are run in matrix ? 
This zip cache is only meant for aggregation jobs. Those job don't run in a matrix, and use zips from different streams.

> how about jobs run locally (for those of us that don't consider jenkins where you test things 
Caching is totally optional, if the zipCacheFolder is not set, it's simply not used at all, and plugins performs directly a get instead of looking in cache first.

> relying on MD5 check does not mean much if there are concurrent jobs, the file could be being written to while you are checking MD5.
That's true.
                
> cache downloaded zips fetched as part of building aggregate site for reuse across Jenkins slaves (rather than in each slave's workspace)
> ----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: JBIDE-14017
>                 URL: https://issues.jboss.org/browse/JBIDE-14017
>             Project: Tools (JBoss Tools)
>          Issue Type: Bug
>          Components: Build/Releng, updatesite
>    Affects Versions: 4.1.0.Alpha2
>            Reporter: Nick Boldt
>            Assignee: Mickael Istria
>             Fix For: 4.1.0.Beta1
>
>
> {quote}
> (2013-04-12 12:19:29) mistria: nickboldt: aggregate could compare logs
> (2013-04-12 12:19:35) nickboldt: mistria: which logs?
> (2013-04-12 12:20:58) mistria: nickboldt: the one we generate when using "fetch-zips"
> (2013-04-12 12:22:40) nickboldt: so you want the aggregate job to fetch the zips BEFORE it builds the update site to determine if it should build the update site?
> (2013-04-12 12:24:05) nickboldt: whole point of a "should I build?" check is that it happens before the build... eg., check SVN, check git, or run jbosstools-composite-install_master to check the installed IUs installed from the composite to determine if contents or versions have changed
> (2013-04-12 12:24:46) nickboldt: not adverse to the idea of splitting the "fetch zips" step out of the "build aggregate" job, but ... then we have 2 more jobs to worry about 
> (2013-04-12 12:24:53) mistria: nickboldt: actually, the new version of the fetch mojo won't re-fetch zips if it finds out the the signature of the to-be-downloaded zip is the same as the one it has in cache
> (2013-04-12 12:25:05) nickboldt: cache in workspace?
> (2013-04-12 12:25:06) mistria: nickboldt: so no change, not fetch
> (2013-04-12 12:25:23) nickboldt: or in static jenkins share folder?
> (2013-04-12 12:25:23) mistria: nickboldt: in aggregate/site/zips
> (2013-04-12 12:25:36) nickboldt: great, so it's one fetch per slave :(
> (2013-04-12 12:26:14) nickboldt: would be better to cache that outside the workspace, if building in Jenkins
> (2013-04-12 12:26:16) mistria: nickboldt: right, we should provide a way to override this directory and put in in a shared folder
> (2013-04-12 12:27:12) mistria: nickboldt: there is not system property for that yet... https://github.com/jbosstools/jbosstools-maven-plugins/blob/master/tycho-plugins/repository-utils/src/main/java/org/jboss/tools/tycho/sitegenerator/FetchZipsFromRepo.java#L60
> (2013-04-12 12:28:21) nickboldt: Old way: https://github.com/jbosstools/jbosstools-build-sites/blob/jbosstools-4.1.0.Alpha2x/aggregate/build.xml#L510 and https://github.com/jbosstools/jbosstools-build-sites/blob/jbosstools-4.1.0.Alpha2x/aggregate/build.xml#L33 (but we still dumped to $\{output.dir\}/zips, not to \{WORKINGDIR\}/zips )
> (2013-04-12 12:28:58) nickboldt: mistria: should allow caching files in Jenkins's static file tree for use across slaves
> {quote}
> So, I guess all we need to do is expose *outputFolder* to be overridden when -Phudson is enabled.
> {code:title=https://github.com/jbosstools/jbosstools-maven-plugins/blob/master/tycho-plugins/repository-utils/src/main/java/org/jboss/tools/tycho/sitegenerator/FetchZipsFromRepo.java#L60}
> 	/**
> * Location where to put zips
> * @parameter default-value="${basedir}/zips"
> */
> private File outputFolder;
> {code}
> We could even set the correct path in the parent pom's hudson profile so that it need not be set in all the various jobs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira