[jbosstools-issues] [JBoss JIRA] Issue Comment Edited: (JBIDE-8667) improve publishing of component sites using composite metadata so that content doesn't vanish and break downstream jobs

[Nick Boldt (JIRA)]

    [ https://issues.jboss.org/browse/JBIDE-8667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615785#comment-12615785 ] 

Nick Boldt edited comment on JBIDE-8667 at 7/21/11 6:06 PM:
------------------------------------------------------------

We now have two previous component builds available, stored outside the /staging/ dir [0] to keep that folder less cluttered:

[0] http://download.jboss.org/jbosstools/builds/staging.previous/

Once these are populated, I can update the composite metadata here [1] to reference the collection of 3 sites (current, previous, previous-previous) and be able to better resolve & recover when builds are in progress.

(Note that the solution requires a mix of rsync --delete and sftp to clean out dirs and remove them / rename them. See http://anonsvn.jboss.org/repos/jbosstools/trunk/build/publish.sh and look for the lines starting with JBIDE-8667.)

[1] http://download.jboss.org/jbosstools/builds/staging/_composite_/trunk/compositeArtifacts.xml

----

We still have another related problem, which is that if the agg site for JBT or JBDS is being built and upstream VPE or JSF or JST spin and finish while the downstream is resolving, you can get into this situation:

{code}
[INFO] [Software being installed: 
org.jboss.tools.site.aggregate raw:0.0.1.'SNAPSHOT'/format(n[.n=0;[.n=0;[-S]]]):0.0.1-SNAPSHOT, 
Only one of the following can be installed at once: [
  org.jboss.tools.xulrunner.initializer 3.1.0.v20110720-1516-H397-M3, 
  org.jboss.tools.xulrunner.initializer 3.1.0.v20110721-0807-H399-M3
], 

Cannot satisfy dependency: org.jboss.tools.richfaces.feature.feature.group 3.3.0.v20110720-1633-H328-M3 depends on: 
org.jboss.tools.xulrunner.initializer [3.1.0.v20110720-1516-H397-M3], 

Cannot satisfy dependency: org.jboss.tools.site.aggregate raw:0.0.1.'SNAPSHOT'/format(n[.n=0;[.n=0;[-S]]]):0.0.1-SNAPSHOT depends on:
org.jboss.tools.richfaces.feature.feature.group 0.0.0, 
Cannot satisfy dependency: org.jboss.tools.site.aggregate raw:0.0.1.'SNAPSHOT'/format(n[.n=0;[.n=0;[-S]]]):0.0.1-SNAPSHOT depends on:
org.jboss.tools.xulrunner.feature.feature.group 0.0.0, 
Cannot satisfy dependency: org.jboss.tools.xulrunner.feature.feature.group 1.9.216.v20110721-0807-H399-M3 depends on: 
org.jboss.tools.xulrunner.initializer [3.1.0.v20110721-0807-H399-M3]]
{code}

I could set select upstream components (vpe, jst, jsf) to block while their downstream are running, but that will delay builds from starting if ANY downstream are running. Not sure that's ideal since jbt agg, jbds update, and jbds product can take over an hour each which would prevent CI builds of vpe/jst/jsf from spinning for upwards of 3-4hrs.

      was (Author: nickboldt):
    We now have two previous component builds available, stored outside the /staging/ dir [0] to keep that folder less cluttered:

[0] http://download.jboss.org/jbosstools/builds/staging.previous/

Once these are populated, I can update the composite metadata here [1] to reference the collection of 3 sites (current, previous, previous-previous) and be able to better resolve & recover when builds are in progress.

(Note that the solution requires a mix of rsync --delete and sftp to clean out dirs and remove them / rename them. See http://anonsvn.jboss.org/repos/jbosstools/trunk/build/publish.sh and look for the lines starting with JBIDE-8667.)

[1] http://download.jboss.org/jbosstools/builds/staging/_composite_/trunk/compositeArtifacts.xml

----

We still have another related problem, which is that if the agg site for JBT or JBDS is being built and upstream VPE or JSF or JST spin and finish while the downstream is resolving, you can get into this situation:

{code}
[INFO] [Software being installed: 
org.jboss.tools.site.aggregate raw:0.0.1.'SNAPSHOT'/format(n[.n=0;[.n=0;[-S]]]):0.0.1-SNAPSHOT, 
Only one of the following can be installed at once: [
  org.jboss.tools.xulrunner.initializer 3.1.0.v20110720-1516-H397-M3, 
  org.jboss.tools.xulrunner.initializer 3.1.0.v20110721-0807-H399-M3
], 

Cannot satisfy dependency: org.jboss.tools.richfaces.feature.feature.group 3.3.0.v20110720-1633-H328-M3 depends on: 
org.jboss.tools.xulrunner.initializer [3.1.0.v20110720-1516-H397-M3], 

Cannot satisfy dependency: org.jboss.tools.site.aggregate raw:0.0.1.'SNAPSHOT'/format(n[.n=0;[.n=0;[-S]]]):0.0.1-SNAPSHOT depends on:
org.jboss.tools.richfaces.feature.feature.group 0.0.0, 
Cannot satisfy dependency: org.jboss.tools.site.aggregate raw:0.0.1.'SNAPSHOT'/format(n[.n=0;[.n=0;[-S]]]):0.0.1-SNAPSHOT depends on:
org.jboss.tools.xulrunner.feature.feature.group 0.0.0, 
Cannot satisfy dependency: org.jboss.tools.xulrunner.feature.feature.group 1.9.216.v20110721-0807-H399-M3 depends on: 
org.jboss.tools.xulrunner.initializer [3.1.0.v20110721-0807-H399-M3]]
{code}

  
> improve publishing of component sites using composite metadata so that content doesn't vanish and break downstream jobs
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: JBIDE-8667
>                 URL: https://issues.jboss.org/browse/JBIDE-8667
>             Project: Tools (JBoss Tools)
>          Issue Type: Bug
>          Components: Build/Releng
>    Affects Versions: 3.3.0.M1
>            Reporter: Nick Boldt
>            Assignee: Nick Boldt
>             Fix For: 3.3.0.M3
>
>
> (2011-04-01 10:11:04) nickboldt: and while I'd love to do:
> (2011-04-01 10:11:12) nickboldt: mkdir whatever-new
> (2011-04-01 10:11:22) nickboldt: rsync newstuff whatever-new
> (2011-04-01 10:11:25) nickboldt: rm -fr whatever
> (2011-04-01 10:11:29) nickboldt: mv whatever-new whatever
> (2011-04-01 10:11:43) nickboldt: so that the time between "it's there" and "it's there and updated" is less
> (2011-04-01 10:11:52) nickboldt: I CANNOT with the current infra we have
> (2011-04-01 10:12:03) nickboldt: instead I can do an rsync --delete and push into the same folder on top
> (2011-04-01 10:12:16) nickboldt: but that means that the metadata will oft be out of whack w.r.t the site content
> (2011-04-01 10:12:17) maxandersen: hmmm
> (2011-04-01 10:12:41) nickboldt: if you can think of a way to do a "push then rename" instead of "push into existing and clean out cruft"
> (2011-04-01 10:12:51) nickboldt: over sftp via a script (non-interactive) then I'm all ears
> (2011-04-01 10:13:26) nickboldt: I can't push a script there and run it, or simply connect over ssh and run a script directly
> (2011-04-01 10:13:35) maxandersen: well what I was thinking was to do a push of new site into new dir, push site.xml/compositewhatever pointing to new dir and then rsync delete the existing one..
> (2011-04-01 10:14:10) maxandersen: sftp rm should work?
> (2011-04-01 10:14:13) nickboldt: MIGHT be able to do sftp delete
> (2011-04-01 10:14:39) maxandersen: if you control it with a composite artifact you dont need to rename anything
> (2011-04-01 10:14:51) maxandersen: except the content pointer in there which you know "locally"
> (2011-04-01 10:15:01) nickboldt: true... so I'd need a tool to do a one line swap in the composite*.xml files
> (2011-04-01 10:15:13) nickboldt: remove ref to old site, replace w/ ref to new site
> (2011-04-01 10:15:17) nickboldt: and then delete the old one
> 1. generate composite metadata files on the fly
> 2. publish that into root dir, all/repo/
> 3. purge previous build's folder over scripted sftp?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira