[JBoss JIRA] (JBIDE-20904) create single bash script (and Jenkins job wrapper) to perform the first 10 steps of CI -> stage or stage -> development/stable
by Nick Boldt (JIRA)
[ https://issues.jboss.org/browse/JBIDE-20904?page=com.atlassian.jira.plugi... ]
Nick Boldt edited comment on JBIDE-20904 at 4/13/16 12:21 PM:
--------------------------------------------------------------
Agreed, but since Jenkins is slow to run (finding a slave on which to run sometimes blocked by other jobs) & slow to config (https = painful), for now I'm just running the script as hudson@dev01 to synthesize the effect of running in jenkins but with more interactivity.
Once it's working for not just ci -> stage but also stage -> dev (will have to fake this up with the master branch jobs, I suspect), I can wrap a jenkins job around it.
Here are the current staging steps:
0. run quick verification tests:
a. does o.j.t.foundation.core have the correct version in it? [Aside: this should be moved to the Central CI job]
b. are all the branches correct?
c. are all the SHAs in github == SHAs built in Jenkins?
1. disable jobs (now scripted via curl)
2. fire discovery job w/ specific param (again, curl), then disable once the build has started
3a. wget staging script; run with some params passed in (this is 5 + 6 steps for JBT and for JBDS, but the 11 steps can be run in parallel processes)
3b. verify that all the new pages are created correctly (11+ URLs to review, some which need additional time to refresh as devstudio.redhat.com can be fickle about showing new content)
4. update composite sites for JBT and JBDS [scripted w/ sed]
5. merge in IS content to discovery sites (scripted but needs to know the latest IS site URLs to use as source)
6. update ide-config.properties (manual for now but could eventually be scripted w/ sed?)
7. re-enable jobs (scripted w/ curl)
Then we could fire the smoke tests automatically (p2director, install-grinder) and when those were done, generate the 2 template-based email notifications that the staging was done. Currently we run BYOE tests for JBT and JBDS.
I don't believe we have an automated test of the installer + installing all the EA content from Central. - JBIDE-22172
Only thing I can't see a way to do is push the changes on download.jboss.org and devstudio.redhat.com into github. So that would need to be some sort of rsync script + manual review and check-in.
Bonus there is that it could be done while the smoke test was underway, and should a bug happen (eg., wrong URL for Central TP site) the fix could be done both in github and pushed back to the server.
was (Author: nickboldt):
Agreed, but since Jenkins is slow to run (finding a slave on which to run sometimes blocked by other jobs) & slow to config (https = painful), for now I'm just running the script as hudson@dev01 to synthesize the effect of running in jenkins but with more interactivity.
Once it's working for not just ci -> stage but also stage -> dev (will have to fake this up with the master branch jobs, I suspect), I can wrap a jenkins job around it.
Here are the current staging steps:
0. run quick verification tests:
a. does o.j.t.foundation.core have the correct version in it? [Aside: this should be moved to the Central CI job]
b. are all the branches correct?
c. are all the SHAs in github == SHAs built in Jenkins?
1. disable jobs (now scripted via curl)
2. fire discovery job w/ specific param (again, curl), then disable once the build has started
3a. wget staging script; run with some params passed in (this is 5 + 6 steps for JBT and for JBDS, but the 11 steps can be run in parallel processes)
3b. verify that all the new pages are created correctly (11+ URLs to review, some which need additional time to refresh as devstudio.redhat.com can be fickle about showing new content)
4. update composite sites for JBT and JBDS [scripted w/ sed]
5. merge in IS content to discovery sites (scripted but needs to know the latest IS site URLs to use as source)
6. update ide-config.properties (manual for now but could eventually be scripted w/ sed?)
7. re-enable jobs (scripted w/ curl)
Then we could fire the smoke tests automatically (p2director, install-grinder) and when those were done, generate the 2 template-based email notifications that the staging was done. Currently we run BYOE tests for JBT and JBDS.
I don't believe we have an automated test of the installer + installing all the EA content from Central.
Only thing I can't see a way to do is push the changes on download.jboss.org and devstudio.redhat.com into github. So that would need to be some sort of rsync script + manual review and check-in.
Bonus there is that it could be done while the smoke test was underway, and should a bug happen (eg., wrong URL for Central TP site) the fix could be done both in github and pushed back to the server.
> create single bash script (and Jenkins job wrapper) to perform the first 10 steps of CI -> stage or stage -> development/stable
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: JBIDE-20904
> URL: https://issues.jboss.org/browse/JBIDE-20904
> Project: Tools (JBoss Tools)
> Issue Type: Enhancement
> Components: build
> Affects Versions: 4.4.0.Alpha1
> Reporter: Nick Boldt
> Assignee: Nick Boldt
> Fix For: 4.4.0.Alpha1
>
>
> Suggestion:
> rather than opening 10 bash terminals to perform the copy-from-one-place-on-disk-to-local, copy-from-local-to-another-place steps required to clone CI bits to Stage or from Stage to release, [~mickael_istria] and I discovered today that we could use `wait` or `parallel` to orchestrate these steps via a bash script so they run in parallel (as quickly as possible), but still return feedback when all parallel steps are completed.
> So, where today we run these steps sorta-by-hand (copy script into a console and wait until it's done) [1], in future we could simply kick a job and wait for the job to notify its completion.
> [1] https://github.com/jbdevstudio/jbdevstudio-devdoc/tree/master/release_gui...
> Examples of using a series of commands in parallel w/ a wait at the end:
> http://stackoverflow.com/questions/19543139/bash-script-processing-comman...
> {code:title=spawns the 3 parallel steps, waits until #3 is done (2 seconds later) and returns the PID of the last one + its return code}
> echo "1" & echo 2 & (sleep 2;echo 3) & wait && echo $! $#
> {code}
> More discussion:
> {quote}
> [12:44:46 PM] Mickael Istria: I believe some parts would have to be turned into functions
> [12:54:41 PM] Mickael Istria: so, to hack the script, it could be just:
> * add && after the 1st rsync in each loop
> * add & after the last one
> * put a wait after the last loop
> * give the big piece of code to procede directly to bash
> {quote}
> After this job is done, releng would still have to "wire up" the new bits by updating composite*.xml and index.html pages, but that's considerably easier to do locally in a terminal, or even to script too. Rather than updating these files w/ sed, we could generate them from a template.
> And if we don't care about committing those changes back to github, we could even push them to the dl.jb.o and ds.jb.c servers directly as part of the above job.
> Scary, but much faster!
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years
[JBoss JIRA] (JBIDE-22172) create automated install test for JBDS installer jar + all early access content in Central
by Nick Boldt (JIRA)
Nick Boldt created JBIDE-22172:
----------------------------------
Summary: create automated install test for JBDS installer jar + all early access content in Central
Key: JBIDE-22172
URL: https://issues.jboss.org/browse/JBIDE-22172
Project: Tools (JBoss Tools)
Issue Type: Sub-task
Components: build, install-tests
Affects Versions: 4.3.1.CR1
Reporter: Nick Boldt
To better automate staging & release, we need a way to run a smoke test of a given installer jar, then to install all the GA & EA content in Central.
Steps to script:
a. wget installer jar
b. headless install (via -console or install.xml script)
c. install grinder or p2director install all the things in Central.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years
[JBoss JIRA] (JBIDE-20904) create single bash script (and Jenkins job wrapper) to perform the first 10 steps of CI -> stage or stage -> development/stable
by Nick Boldt (JIRA)
[ https://issues.jboss.org/browse/JBIDE-20904?page=com.atlassian.jira.plugi... ]
Nick Boldt edited comment on JBIDE-20904 at 4/13/16 12:17 PM:
--------------------------------------------------------------
Agreed, but since Jenkins is slow to run (finding a slave on which to run sometimes blocked by other jobs) & slow to config (https = painful), for now I'm just running the script as hudson@dev01 to synthesize the effect of running in jenkins but with more interactivity.
Once it's working for not just ci -> stage but also stage -> dev (will have to fake this up with the master branch jobs, I suspect), I can wrap a jenkins job around it.
Here are the current staging steps:
0. run quick verification tests:
a. does o.j.t.foundation.core have the correct version in it? [Aside: this should be moved to the Central CI job]
b. are all the branches correct?
c. are all the SHAs in github == SHAs built in Jenkins?
1. disable jobs (now scripted via curl)
2. fire discovery job w/ specific param (again, curl), then disable once the build has started
3a. wget staging script; run with some params passed in (this is 5 + 6 steps for JBT and for JBDS, but the 11 steps can be run in parallel processes)
3b. verify that all the new pages are created correctly (11+ URLs to review, some which need additional time to refresh as devstudio.redhat.com can be fickle about showing new content)
4. update composite sites for JBT and JBDS [scripted w/ sed]
5. merge in IS content to discovery sites (scripted but needs to know the latest IS site URLs to use as source)
6. update ide-config.properties (manual for now but could eventually be scripted w/ sed?)
7. re-enable jobs (scripted w/ curl)
Then we could fire the smoke tests automatically (p2director, install-grinder) and when those were done, generate the 2 template-based email notifications that the staging was done. Currently we run BYOE tests for JBT and JBDS.
I don't believe we have an automated test of the installer + installing all the EA content from Central.
Only thing I can't see a way to do is push the changes on download.jboss.org and devstudio.redhat.com into github. So that would need to be some sort of rsync script + manual review and check-in.
Bonus there is that it could be done while the smoke test was underway, and should a bug happen (eg., wrong URL for Central TP site) the fix could be done both in github and pushed back to the server.
was (Author: nickboldt):
Agreed, but since Jenkins is slow to run (finding a slave on which to run sometimes blocked by other jobs) & slow to config (https = painful), for now I'm just running the script as hudson@dev01 to synthesize the effect of running in jenkins but with more interactivity.
Once it's working for not just ci -> stage but also stage -> dev (will have to fake this up with the master branch jobs, I suspect), I can wrap a jenkins job around it.
Here are the current staging steps:
0. run quick verification tests:
a. does o.j.t.foundation.core have the correct version in it? [Aside: this should be moved to the Central CI job]
b. are all the branches correct?
c. are all the SHAs in github == SHAs built in Jenkins?
1. disable jobs (now scripted via curl)
2. fire discovery job w/ specific param (again, curl), then disable once the build has started
3a. wget staging script; run with some params passed in (this is 5 + 6 steps for JBT and for JBDS, but the 11 steps can be run in parallel processes)
3b. verify that all the new pages are created correctly (11+ URLs to review, some which need additional time to refresh as devstudio.redhat.com can be fickle about showing new content)
4. update composite sites for JBT and JBDS [scripted w/ sed]
5. merge in IS content to discovery sites (scripted but needs to know the latest IS site URLs to use as source)
6. update ide-config.properties (manual for now but could eventually be scripted w/ sed?)
7. re-enable jobs (scripted w/ curl)
Then we could fire the smoke tests automatically (p2director, install-grinder) and when those were done, generate the 2 template-based email notifications that the staging was done.
Only thing I can't see a way to do is push the changes on download.jboss.org and devstudio.redhat.com into github. So that would need to be some sort of rsync script + manual review and check-in.
Bonus there is that it could be done while the smoke test was underway, and should a bug happen (eg., wrong URL for Central TP site) the fix could be done both in github and pushed back to the server.
> create single bash script (and Jenkins job wrapper) to perform the first 10 steps of CI -> stage or stage -> development/stable
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: JBIDE-20904
> URL: https://issues.jboss.org/browse/JBIDE-20904
> Project: Tools (JBoss Tools)
> Issue Type: Enhancement
> Components: build
> Affects Versions: 4.4.0.Alpha1
> Reporter: Nick Boldt
> Assignee: Nick Boldt
> Fix For: 4.4.0.Alpha1
>
>
> Suggestion:
> rather than opening 10 bash terminals to perform the copy-from-one-place-on-disk-to-local, copy-from-local-to-another-place steps required to clone CI bits to Stage or from Stage to release, [~mickael_istria] and I discovered today that we could use `wait` or `parallel` to orchestrate these steps via a bash script so they run in parallel (as quickly as possible), but still return feedback when all parallel steps are completed.
> So, where today we run these steps sorta-by-hand (copy script into a console and wait until it's done) [1], in future we could simply kick a job and wait for the job to notify its completion.
> [1] https://github.com/jbdevstudio/jbdevstudio-devdoc/tree/master/release_gui...
> Examples of using a series of commands in parallel w/ a wait at the end:
> http://stackoverflow.com/questions/19543139/bash-script-processing-comman...
> {code:title=spawns the 3 parallel steps, waits until #3 is done (2 seconds later) and returns the PID of the last one + its return code}
> echo "1" & echo 2 & (sleep 2;echo 3) & wait && echo $! $#
> {code}
> More discussion:
> {quote}
> [12:44:46 PM] Mickael Istria: I believe some parts would have to be turned into functions
> [12:54:41 PM] Mickael Istria: so, to hack the script, it could be just:
> * add && after the 1st rsync in each loop
> * add & after the last one
> * put a wait after the last loop
> * give the big piece of code to procede directly to bash
> {quote}
> After this job is done, releng would still have to "wire up" the new bits by updating composite*.xml and index.html pages, but that's considerably easier to do locally in a terminal, or even to script too. Rather than updating these files w/ sed, we could generate them from a template.
> And if we don't care about committing those changes back to github, we could even push them to the dl.jb.o and ds.jb.c servers directly as part of the above job.
> Scary, but much faster!
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years
[JBoss JIRA] (JBIDE-20904) create single bash script (and Jenkins job wrapper) to perform the first 10 steps of CI -> stage or stage -> development/stable
by Nick Boldt (JIRA)
[ https://issues.jboss.org/browse/JBIDE-20904?page=com.atlassian.jira.plugi... ]
Nick Boldt commented on JBIDE-20904:
------------------------------------
Agreed, but since Jenkins is slow to run (finding a slave on which to run sometimes blocked by other jobs) & slow to config (https = painful), for now I'm just running the script as hudson@dev01 to synthesize the effect of running in jenkins but with more interactivity.
Once it's working for not just ci -> stage but also stage -> dev (will have to fake this up with the master branch jobs, I suspect), I can wrap a jenkins job around it.
Here are the current staging steps:
0. run quick verification tests:
a. does o.j.t.foundation.core have the correct version in it? [Aside: this should be moved to the Central CI job]
b. are all the branches correct?
c. are all the SHAs in github == SHAs built in Jenkins?
1. disable jobs (now scripted via curl)
2. fire discovery job w/ specific param (again, curl), then disable once the build has started
3a. wget staging script; run with some params passed in (this is 5 + 6 steps for JBT and for JBDS, but the 11 steps can be run in parallel processes)
3b. verify that all the new pages are created correctly (11+ URLs to review, some which need additional time to refresh as devstudio.redhat.com can be fickle about showing new content)
4. update composite sites for JBT and JBDS [scripted w/ sed]
5. merge in IS content to discovery sites (scripted but needs to know the latest IS site URLs to use as source)
6. update ide-config.properties (manual for now but could eventually be scripted w/ sed?)
7. re-enable jobs (scripted w/ curl)
Then we could fire the smoke tests automatically (p2director, install-grinder) and when those were done, generate the 2 template-based email notifications that the staging was done.
Only thing I can't see a way to do is push the changes on download.jboss.org and devstudio.redhat.com into github. So that would need to be some sort of rsync script + manual review and check-in.
Bonus there is that it could be done while the smoke test was underway, and should a bug happen (eg., wrong URL for Central TP site) the fix could be done both in github and pushed back to the server.
> create single bash script (and Jenkins job wrapper) to perform the first 10 steps of CI -> stage or stage -> development/stable
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: JBIDE-20904
> URL: https://issues.jboss.org/browse/JBIDE-20904
> Project: Tools (JBoss Tools)
> Issue Type: Enhancement
> Components: build
> Affects Versions: 4.4.0.Alpha1
> Reporter: Nick Boldt
> Assignee: Nick Boldt
> Fix For: 4.4.0.Alpha1
>
>
> Suggestion:
> rather than opening 10 bash terminals to perform the copy-from-one-place-on-disk-to-local, copy-from-local-to-another-place steps required to clone CI bits to Stage or from Stage to release, [~mickael_istria] and I discovered today that we could use `wait` or `parallel` to orchestrate these steps via a bash script so they run in parallel (as quickly as possible), but still return feedback when all parallel steps are completed.
> So, where today we run these steps sorta-by-hand (copy script into a console and wait until it's done) [1], in future we could simply kick a job and wait for the job to notify its completion.
> [1] https://github.com/jbdevstudio/jbdevstudio-devdoc/tree/master/release_gui...
> Examples of using a series of commands in parallel w/ a wait at the end:
> http://stackoverflow.com/questions/19543139/bash-script-processing-comman...
> {code:title=spawns the 3 parallel steps, waits until #3 is done (2 seconds later) and returns the PID of the last one + its return code}
> echo "1" & echo 2 & (sleep 2;echo 3) & wait && echo $! $#
> {code}
> More discussion:
> {quote}
> [12:44:46 PM] Mickael Istria: I believe some parts would have to be turned into functions
> [12:54:41 PM] Mickael Istria: so, to hack the script, it could be just:
> * add && after the 1st rsync in each loop
> * add & after the last one
> * put a wait after the last loop
> * give the big piece of code to procede directly to bash
> {quote}
> After this job is done, releng would still have to "wire up" the new bits by updating composite*.xml and index.html pages, but that's considerably easier to do locally in a terminal, or even to script too. Rather than updating these files w/ sed, we could generate them from a template.
> And if we don't care about committing those changes back to github, we could even push them to the dl.jb.o and ds.jb.c servers directly as part of the above job.
> Scary, but much faster!
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years
[JBoss JIRA] (JBIDE-20904) create single bash script (and Jenkins job wrapper) to perform the first 10 steps of CI -> stage or stage -> development/stable
by Nick Boldt (JIRA)
[ https://issues.jboss.org/browse/JBIDE-20904?page=com.atlassian.jira.plugi... ]
Nick Boldt edited comment on JBIDE-20904 at 4/13/16 11:54 AM:
--------------------------------------------------------------
I've now successfully tested this script for use with the JBDS 9.1.0.CR1b stage -- it currently runs a series of steps in parallel with less console cruft & in fewer console windows. Also migrated it from using /tmp to $HOME so there's more disk space available.
https://github.com/jbosstools/jbosstools-build-ci/blob/jbosstools-4.3.x/p...
I'll run it again for CR1c tonight/tomorrow.
Would also like to script these steps:
* to push target platform zips from www.qa -> devstudio
* to update composite sites & verify content
* to merge in JBDS IS and JBT IS content to discovery sites
* to run headless install test (smoke tests)
And then need to wrap a Jenkins job around it, so it can be done automatically when a discovery site is done, perhaps using an extra param such as "staging=true" to invoke the downstream job automatically.
Also want to switch to using sshfs mounts instead of the current "copy to temp them copy from temp" approach I'm using now.
was (Author: nickboldt):
I've now successfully tested this script for use with the JBDS 9.1.0.CR1b stage.
https://github.com/jbosstools/jbosstools-build-ci/blob/jbosstools-4.3.x/p...
I'll run it again for CR1c tonight/tomorrow.
Would also like to script these steps:
* to push target platform zips from www.qa -> devstudio
* to update composite sites & verify content
* to merge in JBDS IS and JBT IS content to discovery sites
* to run headless install test (smoke tests)
> create single bash script (and Jenkins job wrapper) to perform the first 10 steps of CI -> stage or stage -> development/stable
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: JBIDE-20904
> URL: https://issues.jboss.org/browse/JBIDE-20904
> Project: Tools (JBoss Tools)
> Issue Type: Enhancement
> Components: build
> Affects Versions: 4.4.0.Alpha1
> Reporter: Nick Boldt
> Assignee: Nick Boldt
> Fix For: 4.4.0.Alpha1
>
>
> Suggestion:
> rather than opening 10 bash terminals to perform the copy-from-one-place-on-disk-to-local, copy-from-local-to-another-place steps required to clone CI bits to Stage or from Stage to release, [~mickael_istria] and I discovered today that we could use `wait` or `parallel` to orchestrate these steps via a bash script so they run in parallel (as quickly as possible), but still return feedback when all parallel steps are completed.
> So, where today we run these steps sorta-by-hand (copy script into a console and wait until it's done) [1], in future we could simply kick a job and wait for the job to notify its completion.
> [1] https://github.com/jbdevstudio/jbdevstudio-devdoc/tree/master/release_gui...
> Examples of using a series of commands in parallel w/ a wait at the end:
> http://stackoverflow.com/questions/19543139/bash-script-processing-comman...
> {code:title=spawns the 3 parallel steps, waits until #3 is done (2 seconds later) and returns the PID of the last one + its return code}
> echo "1" & echo 2 & (sleep 2;echo 3) & wait && echo $! $#
> {code}
> More discussion:
> {quote}
> [12:44:46 PM] Mickael Istria: I believe some parts would have to be turned into functions
> [12:54:41 PM] Mickael Istria: so, to hack the script, it could be just:
> * add && after the 1st rsync in each loop
> * add & after the last one
> * put a wait after the last loop
> * give the big piece of code to procede directly to bash
> {quote}
> After this job is done, releng would still have to "wire up" the new bits by updating composite*.xml and index.html pages, but that's considerably easier to do locally in a terminal, or even to script too. Rather than updating these files w/ sed, we could generate them from a template.
> And if we don't care about committing those changes back to github, we could even push them to the dl.jb.o and ds.jb.c servers directly as part of the above job.
> Scary, but much faster!
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years
[JBoss JIRA] (JBIDE-20904) create single bash script (and Jenkins job wrapper) to perform the first 10 steps of CI -> stage or stage -> development/stable
by Mickael Istria (JIRA)
[ https://issues.jboss.org/browse/JBIDE-20904?page=com.atlassian.jira.plugi... ]
Mickael Istria commented on JBIDE-20904:
----------------------------------------
I believe most of the value of that Jira is actually in making CI jobs. The benefit with CI jobs is that they are kind of "self-documenting" in a way that anyone looking at the configuration will be able to reproduce it locally -if they have the credentials-, and that anyone can click a button to stage/release. It would allow to better share the work.
> create single bash script (and Jenkins job wrapper) to perform the first 10 steps of CI -> stage or stage -> development/stable
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: JBIDE-20904
> URL: https://issues.jboss.org/browse/JBIDE-20904
> Project: Tools (JBoss Tools)
> Issue Type: Enhancement
> Components: build
> Affects Versions: 4.4.0.Alpha1
> Reporter: Nick Boldt
> Assignee: Nick Boldt
> Fix For: 4.4.0.Alpha1
>
>
> Suggestion:
> rather than opening 10 bash terminals to perform the copy-from-one-place-on-disk-to-local, copy-from-local-to-another-place steps required to clone CI bits to Stage or from Stage to release, [~mickael_istria] and I discovered today that we could use `wait` or `parallel` to orchestrate these steps via a bash script so they run in parallel (as quickly as possible), but still return feedback when all parallel steps are completed.
> So, where today we run these steps sorta-by-hand (copy script into a console and wait until it's done) [1], in future we could simply kick a job and wait for the job to notify its completion.
> [1] https://github.com/jbdevstudio/jbdevstudio-devdoc/tree/master/release_gui...
> Examples of using a series of commands in parallel w/ a wait at the end:
> http://stackoverflow.com/questions/19543139/bash-script-processing-comman...
> {code:title=spawns the 3 parallel steps, waits until #3 is done (2 seconds later) and returns the PID of the last one + its return code}
> echo "1" & echo 2 & (sleep 2;echo 3) & wait && echo $! $#
> {code}
> More discussion:
> {quote}
> [12:44:46 PM] Mickael Istria: I believe some parts would have to be turned into functions
> [12:54:41 PM] Mickael Istria: so, to hack the script, it could be just:
> * add && after the 1st rsync in each loop
> * add & after the last one
> * put a wait after the last loop
> * give the big piece of code to procede directly to bash
> {quote}
> After this job is done, releng would still have to "wire up" the new bits by updating composite*.xml and index.html pages, but that's considerably easier to do locally in a terminal, or even to script too. Rather than updating these files w/ sed, we could generate them from a template.
> And if we don't care about committing those changes back to github, we could even push them to the dl.jb.o and ds.jb.c servers directly as part of the above job.
> Scary, but much faster!
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years
[JBoss JIRA] (JBDS-3799) Installer - rsync failures seen during post-install JBDS startup of CDK as server runtime
by Len DiMaggio (JIRA)
Len DiMaggio created JBDS-3799:
----------------------------------
Summary: Installer - rsync failures seen during post-install JBDS startup of CDK as server runtime
Key: JBDS-3799
URL: https://issues.jboss.org/browse/JBDS-3799
Project: Developer Studio (JBoss Developer Studio)
Issue Type: Bug
Components: installer
Affects Versions: 9.1.0.CR1
Reporter: Len DiMaggio
Assignee: Fred Bricon
Priority: Critical
This unresolved issue: https://github.com/projectatomic/adb-atomic-developer-bundle/issues/272
Is causing rsync issues such as:
default: Rsyncing folder: /c/DeveloperPlatform/cdk/components/rhel/rhel-ose/ => /vagrant
There was an error when attempting to rsync a synced folder.
Please inspect the error message below for more info.
Host path: /c/DeveloperPlatform/cdk/components/rhel/rhel-ose/
Guest path: /vagrant
Command: rsync --verbose --archive --delete -z --copy-links --chmod=ugo=rwX --no-perms --no-owner --no-group --rsync-path sudo rsync -e ssh -p 2222 -o StrictHostKeyChecking=no -o IdentitiesOnly=true -o UserKnownHostsFile=/dev/null -i 'C:/DeveloperPlatform/cdk/components/rhel/rhel-ose/.vagrant/machines/default/virtualbox/private_key' --exclude .vagrant/ /c/DeveloperPlatform/cdk/components/rhel/rhel-ose/ vagrant@127.0.0.1:/vagrant
Error: Warning: Permanently added '[127.0.0.1]:2222' (ECDSA) to the list of known hosts.
dup() in/out/err failed
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [sender=3.1.2]
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years
[JBoss JIRA] (JBIDE-20904) create single bash script (and Jenkins job wrapper) to perform the first 10 steps of CI -> stage or stage -> development/stable
by Nick Boldt (JIRA)
[ https://issues.jboss.org/browse/JBIDE-20904?page=com.atlassian.jira.plugi... ]
Nick Boldt commented on JBIDE-20904:
------------------------------------
I've now successfully tested this script for use with the JBDS 9.1.0.CR1b stage.
https://github.com/jbosstools/jbosstools-build-ci/blob/jbosstools-4.3.x/p...
I'll run it again for CR1c tonight/tomorrow.
Would also like to script these steps:
* to push target platform zips from www.qa -> devstudio
* to update composite sites & verify content
* to merge in JBDS IS and JBT IS content to discovery sites
* to run headless install test (smoke tests)
> create single bash script (and Jenkins job wrapper) to perform the first 10 steps of CI -> stage or stage -> development/stable
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: JBIDE-20904
> URL: https://issues.jboss.org/browse/JBIDE-20904
> Project: Tools (JBoss Tools)
> Issue Type: Enhancement
> Components: build
> Affects Versions: 4.4.0.Alpha1
> Reporter: Nick Boldt
> Assignee: Nick Boldt
> Fix For: 4.4.0.Alpha1
>
>
> Suggestion:
> rather than opening 10 bash terminals to perform the copy-from-one-place-on-disk-to-local, copy-from-local-to-another-place steps required to clone CI bits to Stage or from Stage to release, [~mickael_istria] and I discovered today that we could use `wait` or `parallel` to orchestrate these steps via a bash script so they run in parallel (as quickly as possible), but still return feedback when all parallel steps are completed.
> So, where today we run these steps sorta-by-hand (copy script into a console and wait until it's done) [1], in future we could simply kick a job and wait for the job to notify its completion.
> [1] https://github.com/jbdevstudio/jbdevstudio-devdoc/tree/master/release_gui...
> Examples of using a series of commands in parallel w/ a wait at the end:
> http://stackoverflow.com/questions/19543139/bash-script-processing-comman...
> {code:title=spawns the 3 parallel steps, waits until #3 is done (2 seconds later) and returns the PID of the last one + its return code}
> echo "1" & echo 2 & (sleep 2;echo 3) & wait && echo $! $#
> {code}
> More discussion:
> {quote}
> [12:44:46 PM] Mickael Istria: I believe some parts would have to be turned into functions
> [12:54:41 PM] Mickael Istria: so, to hack the script, it could be just:
> * add && after the 1st rsync in each loop
> * add & after the last one
> * put a wait after the last loop
> * give the big piece of code to procede directly to bash
> {quote}
> After this job is done, releng would still have to "wire up" the new bits by updating composite*.xml and index.html pages, but that's considerably easier to do locally in a terminal, or even to script too. Rather than updating these files w/ sed, we could generate them from a template.
> And if we don't care about committing those changes back to github, we could even push them to the dl.jb.o and ds.jb.c servers directly as part of the above job.
> Scary, but much faster!
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years
[JBoss JIRA] (JBDS-3799) Installer - rsync failures seen during post-install JBDS startup of CDK as server runtime
by Len DiMaggio (JIRA)
[ https://issues.jboss.org/browse/JBDS-3799?page=com.atlassian.jira.plugin.... ]
Len DiMaggio updated JBDS-3799:
-------------------------------
Labels: havoc (was: )
> Installer - rsync failures seen during post-install JBDS startup of CDK as server runtime
> -----------------------------------------------------------------------------------------
>
> Key: JBDS-3799
> URL: https://issues.jboss.org/browse/JBDS-3799
> Project: Developer Studio (JBoss Developer Studio)
> Issue Type: Bug
> Components: installer
> Affects Versions: 9.1.0.CR1
> Reporter: Len DiMaggio
> Assignee: Fred Bricon
> Priority: Critical
> Labels: havoc
>
> This unresolved issue: https://github.com/projectatomic/adb-atomic-developer-bundle/issues/272
> Is causing rsync issues such as:
> default: Rsyncing folder: /c/DeveloperPlatform/cdk/components/rhel/rhel-ose/ => /vagrant
> There was an error when attempting to rsync a synced folder.
> Please inspect the error message below for more info.
> Host path: /c/DeveloperPlatform/cdk/components/rhel/rhel-ose/
> Guest path: /vagrant
> Command: rsync --verbose --archive --delete -z --copy-links --chmod=ugo=rwX --no-perms --no-owner --no-group --rsync-path sudo rsync -e ssh -p 2222 -o StrictHostKeyChecking=no -o IdentitiesOnly=true -o UserKnownHostsFile=/dev/null -i 'C:/DeveloperPlatform/cdk/components/rhel/rhel-ose/.vagrant/machines/default/virtualbox/private_key' --exclude .vagrant/ /c/DeveloperPlatform/cdk/components/rhel/rhel-ose/ vagrant@127.0.0.1:/vagrant
> Error: Warning: Permanently added '[127.0.0.1]:2222' (ECDSA) to the list of known hosts.
> dup() in/out/err failed
> rsync: connection unexpectedly closed (0 bytes received so far) [sender]
> rsync error: error in rsync protocol data stream (code 12) at io.c(226) [sender=3.1.2]
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years