[jbosstools-issues] [JBoss JIRA] (JBIDE-20904) automate publishing latest CI build to staging, then from staging to development (or stable)

[Nick Boldt (JIRA)]

    [ https://issues.jboss.org/browse/JBIDE-20904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191422#comment-13191422 ] 

Nick Boldt edited comment on JBIDE-20904 at 4/13/16 12:29 PM:
--------------------------------------------------------------

Agreed, but since Jenkins is slow to run (finding a slave on which to run sometimes blocked by other jobs) & slow to config (https = painful), for now I'm just running the script as hudson at dev01 to synthesize the effect of running in jenkins but with more interactivity. 

Once it's working for not just ci -> stage but also stage -> dev (will have to fake this up with the master branch jobs, I suspect), I can wrap a jenkins job around it.

Here are the current staging steps:

0. run quick verification tests:
a. does o.j.t.foundation.core have the correct version in it? [Aside: this should be moved to the Central CI job]
b. are all the branches correct?
c. are all the SHAs in github == SHAs built in Jenkins?

1. disable jobs (now scripted via curl)
2. fire discovery job w/ specific param (again, curl), then disable once the build has started
3a. wget staging script; run with some params passed in (this is 5 + 6 steps for JBT and for JBDS, but the 11 steps can be run in parallel processes)
3b. verify that all the new pages are created correctly (11+ URLs to review, some which need additional time to refresh as devstudio.redhat.com can be fickle about showing new content) - verify content (JBIDE-22173)
4. update composite sites for JBT and JBDS [scripted w/ sed] - verify content (JBIDE-22173)
5. merge in IS content to discovery sites (scripted but needs to know the latest IS site URLs to use as source) - verify content (JBIDE-22173)
6. update ide-config.properties (manual for now but could eventually be scripted w/ sed?) - verify content (JBIDE-22173)
7. re-enable jobs (scripted w/ curl)

Then we could fire the smoke tests automatically (p2director, install-grinder) and when those were done, generate the 2 template-based email notifications that the staging was done. Currently we run BYOE tests for JBT and JBDS. 

I don't believe we have an automated test of the installer + installing all the EA content from Central. - JBIDE-22172

Only thing I can't see a way to do is push the changes on download.jboss.org and devstudio.redhat.com into github. So that would need to be some sort of rsync fetch script + manual review and check-in.

Bonus there is that it could be done while the smoke test was underway, and should a bug happen (eg., wrong URL for Central TP site) the fix could be done both in github and pushed back to the server.



was (Author: nickboldt):
Agreed, but since Jenkins is slow to run (finding a slave on which to run sometimes blocked by other jobs) & slow to config (https = painful), for now I'm just running the script as hudson at dev01 to synthesize the effect of running in jenkins but with more interactivity. 

Once it's working for not just ci -> stage but also stage -> dev (will have to fake this up with the master branch jobs, I suspect), I can wrap a jenkins job around it.

Here are the current staging steps:

0. run quick verification tests:
a. does o.j.t.foundation.core have the correct version in it? [Aside: this should be moved to the Central CI job]
b. are all the branches correct?
c. are all the SHAs in github == SHAs built in Jenkins?

1. disable jobs (now scripted via curl)
2. fire discovery job w/ specific param (again, curl), then disable once the build has started
3a. wget staging script; run with some params passed in (this is 5 + 6 steps for JBT and for JBDS, but the 11 steps can be run in parallel processes)
3b. verify that all the new pages are created correctly (11+ URLs to review, some which need additional time to refresh as devstudio.redhat.com can be fickle about showing new content)
4. update composite sites for JBT and JBDS [scripted w/ sed]
5. merge in IS content to discovery sites (scripted but needs to know the latest IS site URLs to use as source)
6. update ide-config.properties (manual for now but could eventually be scripted w/ sed?)
7. re-enable jobs (scripted w/ curl)

Then we could fire the smoke tests automatically (p2director, install-grinder) and when those were done, generate the 2 template-based email notifications that the staging was done. Currently we run BYOE tests for JBT and JBDS. 

I don't believe we have an automated test of the installer + installing all the EA content from Central. - JBIDE-22172

Only thing I can't see a way to do is push the changes on download.jboss.org and devstudio.redhat.com into github. So that would need to be some sort of rsync script + manual review and check-in.

Bonus there is that it could be done while the smoke test was underway, and should a bug happen (eg., wrong URL for Central TP site) the fix could be done both in github and pushed back to the server.


> automate publishing latest CI build to staging, then from staging to development (or stable)
> --------------------------------------------------------------------------------------------
>
>                 Key: JBIDE-20904
>                 URL: https://issues.jboss.org/browse/JBIDE-20904
>             Project: Tools (JBoss Tools)
>          Issue Type: Enhancement
>          Components: build
>    Affects Versions: 4.4.0.Alpha1
>            Reporter: Nick Boldt
>            Assignee: Nick Boldt
>             Fix For: 4.4.0.Alpha1
>
>
> Suggestion:
> rather than opening 10 bash terminals to perform the copy-from-one-place-on-disk-to-local, copy-from-local-to-another-place steps required to clone CI bits to Stage or from Stage to release, [~mickael_istria] and I discovered today that we could use `wait` or `parallel` to orchestrate these steps via a bash script so they run in parallel (as quickly as possible), but still return feedback when all parallel steps are completed.
> So, where today we run these steps sorta-by-hand (copy script into a console and wait until it's done) [1], in future we could simply kick a job and wait for the job to notify its completion.
> [1] https://github.com/jbdevstudio/jbdevstudio-devdoc/tree/master/release_guide/9.x
> Examples of using a series of commands in parallel w/ a wait at the end:
> http://stackoverflow.com/questions/19543139/bash-script-processing-commands-in-parallel
> {code:title=spawns the 3 parallel steps, waits until #3 is done (2 seconds later) and returns the PID of the last one + its return code}
> echo "1" & echo 2 & (sleep 2;echo 3) & wait && echo $! $#
> {code}
> More discussion:
> {quote}
> [12:44:46 PM] Mickael Istria: I believe some parts would have to be turned into functions
> [12:54:41 PM] Mickael Istria: so, to hack the script, it could be just:
> * add && after the 1st rsync in each loop
> * add & after the last one
> * put a wait after the last loop
> * give the big piece of code to procede directly to bash
> {quote}
> After this job is done, releng would still have to "wire up" the new bits by updating composite*.xml and index.html pages, but that's considerably easier to do locally in a terminal, or even to script too. Rather than updating these files w/ sed, we could generate them from a template. 
> And if we don't care about committing those changes back to github, we could even push them to the dl.jb.o and ds.jb.c servers directly as part of the above job. 
> Scary, but much faster!



--
This message was sent by Atlassian JIRA
(v6.4.11#64026)