<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix"><br>
On 08/13/2013 05:49 AM, Nick Boldt wrote:<br>
</div>
<blockquote cite="mid:520A2B71.1080409@redhat.com" type="cite">The
stated workflow is already being done [0], except that today:
<br>
<br>
* the current (N) and previous (N-1) are composited together into
a single site [1]
<br>
<br>
* the project's previous build is NOT then removed from the
composite & deleted once the new build is done publishing
<br>
</blockquote>
<br>
What I am trying to tell for a while is having names
staging/staging.previous in update site and physically move content
of staging to staging.previous after every build is not working. In
fact with this approach local/jenkins build failures are
unpredictable, when dev/jenkins runs a long build lets say javaee
and suddenly on remote server publis.sh L585 moves bits like that<br>
<br>
<span style="color: rgb(51, 51, 51); font-family: Consolas,
'Liberation Mono', Courier, monospace; font-size: 12px;
font-style: normal; font-variant: normal; font-weight: normal;
letter-spacing: normal; line-height: 18px; orphans: auto;
text-align: start; text-indent: 0px; text-transform: none;
white-space: pre; widows: auto; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255, 255,
255); display: inline !important; float: none;"> mv </span><span
class="nv" style="color: rgb(0, 128, 128); font-family: Consolas,
'Liberation Mono', Courier, monospace; font-size: 12px;
font-style: normal; font-variant: normal; font-weight: normal;
letter-spacing: normal; line-height: 18px; orphans: auto;
text-align: start; text-indent: 0px; text-transform: none;
white-space: pre; widows: auto; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255, 255,
255);">$DESTINATION</span><span style="color: rgb(51, 51, 51);
font-family: Consolas, 'Liberation Mono', Courier, monospace;
font-size: 12px; font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal; line-height: 18px;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: pre; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
background-color: rgb(255, 255, 255); display: inline !important;
float: none;">/builds/staging/</span><span class="k"
style="font-weight: bold; color: rgb(51, 51, 51); font-family:
Consolas, 'Liberation Mono', Courier, monospace; font-size: 12px;
font-style: normal; font-variant: normal; letter-spacing: normal;
line-height: 18px; orphans: auto; text-align: start; text-indent:
0px; text-transform: none; white-space: pre; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
background-color: rgb(255, 255, 255);">${</span><span class="nv"
style="color: rgb(0, 128, 128); font-family: Consolas, 'Liberation
Mono', Courier, monospace; font-size: 12px; font-style: normal;
font-variant: normal; font-weight: normal; letter-spacing: normal;
line-height: 18px; orphans: auto; text-align: start; text-indent:
0px; text-transform: none; white-space: pre; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
background-color: rgb(255, 255, 255);">JOB_NAME</span><span
class="k" style="font-weight: bold; color: rgb(51, 51, 51);
font-family: Consolas, 'Liberation Mono', Courier, monospace;
font-size: 12px; font-style: normal; font-variant: normal;
letter-spacing: normal; line-height: 18px; orphans: auto;
text-align: start; text-indent: 0px; text-transform: none;
white-space: pre; widows: auto; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255, 255,
255);">}</span><span style="color: rgb(51, 51, 51); font-family:
Consolas, 'Liberation Mono', Courier, monospace; font-size: 12px;
font-style: normal; font-variant: normal; font-weight: normal;
letter-spacing: normal; line-height: 18px; orphans: auto;
text-align: start; text-indent: 0px; text-transform: none;
white-space: pre; widows: auto; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255, 255,
255); display: inline !important; float: none;"> </span><span
class="nv" style="color: rgb(0, 128, 128); font-family: Consolas,
'Liberation Mono', Courier, monospace; font-size: 12px;
font-style: normal; font-variant: normal; font-weight: normal;
letter-spacing: normal; line-height: 18px; orphans: auto;
text-align: start; text-indent: 0px; text-transform: none;
white-space: pre; widows: auto; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255, 255,
255);">$DESTINATION</span><span style="color: rgb(51, 51, 51);
font-family: Consolas, 'Liberation Mono', Courier, monospace;
font-size: 12px; font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal; line-height: 18px;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: pre; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
background-color: rgb(255, 255, 255); display: inline !important;
float: none;">/builds/staging.previous/</span><span class="k"
style="font-weight: bold; color: rgb(51, 51, 51); font-family:
Consolas, 'Liberation Mono', Courier, monospace; font-size: 12px;
font-style: normal; font-variant: normal; letter-spacing: normal;
line-height: 18px; orphans: auto; text-align: start; text-indent:
0px; text-transform: none; white-space: pre; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
background-color: rgb(255, 255, 255);">${</span><span class="nv"
style="color: rgb(0, 128, 128); font-family: Consolas, 'Liberation
Mono', Courier, monospace; font-size: 12px; font-style: normal;
font-variant: normal; font-weight: normal; letter-spacing: normal;
line-height: 18px; orphans: auto; text-align: start; text-indent:
0px; text-transform: none; white-space: pre; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
background-color: rgb(255, 255, 255);">JOB_NAME</span><span
class="k" style="font-weight: bold; color: rgb(51, 51, 51);
font-family: Consolas, 'Liberation Mono', Courier, monospace;
font-size: 12px; font-style: normal; font-variant: normal;
letter-spacing: normal; line-height: 18px; orphans: auto;
text-align: start; text-indent: 0px; text-transform: none;
white-space: pre; widows: auto; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255, 255,
255);">}<br>
<br>
</span>build fails. Considering it takes couple hours to execute and
you have to do that again and sometimes again until it works, it is
not really convenient. <br>
<br>
<blockquote cite="mid:520A2B71.1080409@redhat.com" type="cite">
<br>
* the composite*.xml metadata files are not regenerated when
publishing, but are *manually* updated if/when a new component is
added or job is renamed
<br>
</blockquote>
<br>
well, they should be updated because:<br>
1. It would let to avoid moving huge p2repos around and wait while
they are in sync with download.jboss.org<br>
2. It would take much less time for *.xml files to be available on
download.jboss.org<br>
<br>
bing! builds would stop failing so often.<br>
<br>
In fact for the future we can even save space by publishing only
changed bits (need to fix qualifier for features/plugins) and it
would let to:<br>
1. Decrease time for syncing with download.jboss.org;<br>
2. Have installation history actually working and let people use
nightly updates and roll back to previous version if something is
wrong;<br>
3. Increase speed of local/jenkins builds, because no more full
downloads for dependencies<br>
<br>
Denis<br>
<br>
<blockquote cite="mid:520A2B71.1080409@redhat.com" type="cite">
<br>
[0]
<a class="moz-txt-link-freetext" href="https://github.com/jbosstools/jbosstools-build-ci/blob/master/publish/publish.sh#L541">https://github.com/jbosstools/jbosstools-build-ci/blob/master/publish/publish.sh#L541</a><br>
<br>
[1]
<a class="moz-txt-link-freetext" href="http://download.jboss.org/jbosstools/builds/staging/_composite_/core/trunk/compositeArtifacts.xml">http://download.jboss.org/jbosstools/builds/staging/_composite_/core/trunk/compositeArtifacts.xml</a><br>
<br>
This is done because:
<br>
<br>
* after the publish of the N build is done, before the bits are
visible on download.jboss.org, the lag seems to vary from seconds
to minutes.
<br>
<br>
* a build-in-progress which resolved against the N-1 bits will
fail if those bits suddenly vanish.
<br>
<br>
---
<br>
<br>
For the stable branch, the same push-then-rename approach [0] is
used (rather than destructively pushing on top of an existing
build, as we did a few years ago), but we only composite [2] the
latest (N) because:
<br>
<br>
* stable branch builds change less often and
<br>
* we want to guarantee that we're using the latest.
<br>
<br>
[2]
<a class="moz-txt-link-freetext" href="http://download.jboss.org/jbosstools/builds/staging/_composite_/core/4.1.kepler/compositeArtifacts.xml">http://download.jboss.org/jbosstools/builds/staging/_composite_/core/4.1.kepler/compositeArtifacts.xml</a><br>
<br>
Hope that makes sense. If you think something can be improved w/o
causing new problems (such as updating the timestamp in the
composite*.xml files dynamically w/ every published build, so p2
sees it as "new" more often), please don't hesitate to open a JIRA
w/ details on what you'd change, and how.
<br>
<br>
Cheers,
<br>
<br>
Nick
<br>
<br>
On 05/21/2013 02:12 PM, Denis Golovin wrote:
<br>
<blockquote type="cite">On 05/21/2013 10:32 AM, Mickael Istria
wrote:
<br>
<blockquote type="cite">On 05/21/2013 06:56 PM, Denis Golovin
wrote:
<br>
<blockquote type="cite">The problem is org.jboss.tools.tests
is not part of
<br>
<br>
<a class="moz-txt-link-freetext" href="http://download.jboss.org/jbosstools/updates/nightly/core/trunk/plugins/">http://download.jboss.org/jbosstools/updates/nightly/core/trunk/plugins/</a>
<br>
</blockquote>
That's right. However it's part of
<br>
<a class="moz-txt-link-freetext" href="http://download.jboss.org/jbosstools/updates/nightly/integrationtests/trunk">http://download.jboss.org/jbosstools/updates/nightly/integrationtests/trunk</a>
<br>
site, so when enabling no profile, the default nightly sites
(core and
<br>
integration tests) are used, so it should resolve this bundle.
<br>
Is this something you can reproduce at every build? It could
happen
<br>
when your build tries to get content at the same time
aggregation is
<br>
getting published.
<br>
</blockquote>
<br>
Pushing process can be implemented different way to avoid this
issue, I
<br>
saw it many times and it is a bit annoying. I remember we spend
some
<br>
time with Nick to tackle it, but problem seems still here.
<br>
<br>
Idea behind this fix is really simple:
<br>
1. Leave old published bits as is on the server side and just
upload new
<br>
ones
<br>
2. Update compositeArtifacts.xml first and include uploaded
update site
<br>
from step 1
<br>
3. Update compositeContent.xml: include new uploaded update site
from
<br>
step 1 and remove oldest one
<br>
4. Update compositeArtifacts.xml and remove oldest one
<br>
5. Remove oldest update site folder
<br>
<br>
Note there are no operations related to renaming/moving
previously
<br>
uploaded update sites and that the key point to have previously
uploaded
<br>
sites available while new one is in uploading stage.
<br>
<br>
It should significantly reduce amount of errors because we keep
two
<br>
update sites for each jbosstools-module, so at least one update
site is
<br>
always available through composite update site for module. There
could
<br>
be still problems for builds with slow connection, but
connection should
<br>
be slow enough to live through at least two builds for
jbosstools
<br>
module. This could be implemented as maven plug-n and number of
builds
<br>
to keep in composite could be a good candidate for configuration
parameters.
<br>
<br>
It is not really critical but nice to have.
<br>
<br>
Denis
<br>
<br>
<blockquote type="cite">--
<br>
Mickael Istria
<br>
Eclipse developer at JBoss, by Red Hat
<a class="moz-txt-link-rfc2396E" href="http://www.jboss.org/tools"><http://www.jboss.org/tools></a>
<br>
My blog <a class="moz-txt-link-rfc2396E" href="http://mickaelistria.wordpress.com"><http://mickaelistria.wordpress.com></a> - My Tweets
<br>
<a class="moz-txt-link-rfc2396E" href="http://twitter.com/mickaelistria"><http://twitter.com/mickaelistria></a>
<br>
<br>
<br>
_______________________________________________
<br>
jbosstools-dev mailing list
<br>
<a class="moz-txt-link-abbreviated" href="mailto:jbosstools-dev@lists.jboss.org">jbosstools-dev@lists.jboss.org</a>
<br>
<a class="moz-txt-link-freetext" href="https://lists.jboss.org/mailman/listinfo/jbosstools-dev">https://lists.jboss.org/mailman/listinfo/jbosstools-dev</a>
<br>
</blockquote>
<br>
<br>
<br>
_______________________________________________
<br>
jbosstools-dev mailing list
<br>
<a class="moz-txt-link-abbreviated" href="mailto:jbosstools-dev@lists.jboss.org">jbosstools-dev@lists.jboss.org</a>
<br>
<a class="moz-txt-link-freetext" href="https://lists.jboss.org/mailman/listinfo/jbosstools-dev">https://lists.jboss.org/mailman/listinfo/jbosstools-dev</a>
<br>
<br>
</blockquote>
<br>
</blockquote>
<br>
</body>
</html>