On 08/13/2013 05:49 AM, Nick Boldt wrote:
The stated workflow is already being done [0], except that today:

* the current (N) and previous (N-1) are composited together into a single site [1]

* the project's previous build is NOT then removed from the composite & deleted once the new build is done publishing

What I am trying to tell for a while is having names staging/staging.previous in update site and physically move content of staging to staging.previous after every build is not working. In fact with this approach local/jenkins build failures are unpredictable, when dev/jenkins runs a long build lets say javaee and suddenly on remote server publis.sh L585 moves bits like that

 mv $DESTINATION/builds/staging/${JOB_NAME} $DESTINATION/builds/staging.previous/${JOB_NAME}

build fails. Considering it takes couple hours to execute and you have to do that again and sometimes again until it works, it is not really convenient.


* the composite*.xml metadata files are not regenerated when publishing, but are *manually* updated if/when a new component is added or job is renamed

well, they should be updated because:
1. It would let to avoid moving huge p2repos around and wait while they are in sync with download.jboss.org
2. It would take much less time for *.xml files to be available on download.jboss.org

bing! builds would stop failing so often.

In fact for the future we can even save space by publishing only changed bits (need to fix qualifier for features/plugins) and it would let to:
1. Decrease time for syncing with download.jboss.org;
2. Have installation history actually working and let people use nightly updates and roll back to previous version if something is wrong;
3. Increase speed of local/jenkins builds, because no more full downloads for dependencies

Denis


[0] https://github.com/jbosstools/jbosstools-build-ci/blob/master/publish/publish.sh#L541

[1] http://download.jboss.org/jbosstools/builds/staging/_composite_/core/trunk/compositeArtifacts.xml

This is done because:

* after the publish of the N build is done, before the bits are visible on download.jboss.org, the lag seems to vary from seconds to minutes.

* a build-in-progress which resolved against the N-1 bits will fail if those bits suddenly vanish.

---

For the stable branch, the same push-then-rename approach [0] is used (rather than destructively pushing on top of an existing build, as we did a few years ago), but we only composite [2] the latest (N) because:

* stable branch builds change less often and
* we want to guarantee that we're using the latest.

[2] http://download.jboss.org/jbosstools/builds/staging/_composite_/core/4.1.kepler/compositeArtifacts.xml

Hope that makes sense. If you think something can be improved w/o causing new problems (such as updating the timestamp in the composite*.xml files dynamically w/ every published build, so p2 sees it as "new" more often), please don't hesitate to open a JIRA w/ details on what you'd change, and how.

Cheers,

Nick

On 05/21/2013 02:12 PM, Denis Golovin wrote:
On 05/21/2013 10:32 AM, Mickael Istria wrote:
On 05/21/2013 06:56 PM, Denis Golovin wrote:
The problem is org.jboss.tools.tests is not part of

http://download.jboss.org/jbosstools/updates/nightly/core/trunk/plugins/
That's right. However it's part of
http://download.jboss.org/jbosstools/updates/nightly/integrationtests/trunk
site, so when enabling no profile, the default nightly sites (core and
integration tests) are used, so it should resolve this bundle.
Is this something you can reproduce at every build? It could happen
when your build tries to get content at the same time aggregation is
getting published.

Pushing process can be implemented different way to avoid this issue, I
saw it many times and it is a bit annoying. I remember we spend some
time with Nick to tackle it, but problem seems still here.

Idea behind this fix is really simple:
1. Leave old published bits as is on the server side and just upload new
ones
2. Update compositeArtifacts.xml first and include uploaded update site
from step 1
3. Update compositeContent.xml: include new uploaded update site from
step 1 and remove oldest one
4. Update compositeArtifacts.xml and remove oldest one
5. Remove oldest update site folder

Note there are no operations related to renaming/moving previously
uploaded update sites and that the key point to have previously uploaded
sites available while new one is in uploading stage.

It should significantly reduce amount of errors because we keep two
update sites for each jbosstools-module, so at least one update site is
always available through composite update site for module. There could
be still problems for builds with slow connection, but connection should
be slow enough to live through at least two builds for jbosstools
module. This could be implemented as maven plug-n and number of builds
to keep in composite could be a good candidate for configuration parameters.

It is not really critical but nice to have.

Denis

--
Mickael Istria
Eclipse developer at JBoss, by Red Hat <http://www.jboss.org/tools>
My blog <http://mickaelistria.wordpress.com> - My Tweets
<http://twitter.com/mickaelistria>


_______________________________________________
jbosstools-dev mailing list
jbosstools-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/jbosstools-dev



_______________________________________________
jbosstools-dev mailing list
jbosstools-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/jbosstools-dev