Re: [jbosstools-dev] Tests error during local build

>> The stated workflow is already being done [0], except that today: >> * the current (N) and previous (N-1) are composited together into a >> single site [1] >> * the project's previous build is NOT then removed from the composite >> & deleted once the new build is done publishing > What I am trying to tell for a while is having names > staging/staging.previous in update site and physically move content of > staging to staging.previous after every build is not working. In fact > with this approach local/jenkins build failures are unpredictable, when > dev/jenkins runs a long build lets say javaee and suddenly on remote > server publis.sh L585 moves bits like that > mv \ > $DESTINATION/builds/staging/${JOB_NAME} \ > $DESTINATION/builds/staging.previous/${JOB_NAME} > > build fails. What it actually does is: 1. rsync new build into staging/${JOB_NAME}.next (slow) 2. delete staging.previous/${JOB_NAME} (slow) 3. move staging/${JOB_NAME} to staging.previous/${JOB_NAME} (fast) 4. move staging/${JOB_NAME}.next to staging/${JOB_NAME} (fast) Therefore the impact on the composite site is only a few seconds' outage, unless your build was depending on content in the *staging.previous site*, in which case, yes, you will likely end up broken. This mechanism allows builds in progress, which depend on the CURRENT Nth build to continue to work even after the new N build replaces the previous one.

For example, if you kick a mvn build, which pings the composite site to resolve dependencies and sees that staging = B123 and staging.previous = B122, your build begins fine. But if while you're building/resolving a new B124 gets published, you'll STILL be fine to depend on B123 bits, just not the now-deleted B122 bits. Restarting your job will always cause the dep resolution to restart, whereupon the metadata will be scanned again and you'll then be building against staging = B124 and staging.previous = B123. > Considering it takes couple hours to execute and you have > to do that again and sometimes again until it works, it is not really > convenient. So are you proposing that instead of an in-place move which reuses generic folder names like "staging" and "staging.previous", we composite build output using unique names like 2013-08-09_05-05-26-B7222/ or 2013-08-13_10-05-28-B7255?

If so, we would need: a) to regenerate the composite site each time there's a new build published, in order to remove the oldest and add the newest (keeping only the Nth and N-1rst builds) (I have a script that might already work for this, or would need tweaking.)

b) heuristics to determine when an older (N-2, N-3, ... N-z) build is no longer needed, perhaps simply by assuming no one needs it after 24hrs? (Once we agree, this is trivial.)

c) a cleanup script which can purge all but the builds which are no more than 1 day old, keeping at all times at least two builds (N and N-1) (I have a script that already does this for folders like http://download.jboss.org/jbosstools/builds/nightly/core/trunk/ but might need to be tweaked to work for a new pattern of staging/${JOB_NAME}/<BUILD_ID>/ .)

>> * the composite*.xml metadata files are not regenerated when >> publishing, but are *manually* updated if/when a new component is >> added or job is renamed > well, they should be updated because: > 1. It would let to avoid moving huge p2repos around and wait while they > are in sync with download.jboss.org > 2. It would take much less time for *.xml files to be available on > download.jboss.org Actually, the time to publish the bits from Jenkins to download.jboss.org would not change. We would still need to push bits to the server, and perform cleanup of old builds. Generating the composite*.xml files would add some (negligible) time to the process, but would guarantee that a p2 or tycho user would always get a fresh timestamp. Currently, as the composite*.xml files do not change regularly, there is NO time required to make them available. So, this would actually introduce some delay, but as I said you'd get fresher files, so it's perhaps an improvement. > bing! builds would stop failing so often. We can certainly try this and verify your hypothesis. :) > In fact for the future we can even save space by publishing only changed > bits (need to fix qualifier for features/plugins) and it would let to: > 1. Decrease time for syncing with download.jboss.org; > 2. Have installation history actually working and let people use nightly > updates and roll back to previous version if something is wrong; > 3. Increase speed of local/jenkins builds, because no more full > downloads for dependencies Publishing only changes to a repo would be nice except that it would mean publishing *destructively on top of existing snapshots*, rather than creating new ones. So anyone who was currently building against the N build would probably end up with a broken build as new IUs were pushed into that build's repo, and its metadata overwritten.

That's definitely worse than what we have today. And would, instead of having timestamped folders which are unique and don't overlap, put us back where we are today with reusable, generic folder names like "staging/${JOB_NAME}". So, I'm all for a new approach to compositing with regenerated composite*.xml files, pointing to UNIQUELY VERSIONED folders (rather than generic reusable ones), then doing cleanup of older N-2, N-3 builds, but I disagree that publishing changed bits into existing repos is a good idea (even if you use rsync --delete to purge the orphaned IUs, you'll still end up breaking more builds this way), which is why we moved AWAY from this approach by implementing the staging.next/staging/staging.previous shell game.

N >> >> [0] >> https://github.com/jbosstools/jbosstools-build-ci/blob/master/publish/pub... >> >> >> [1] >> http://download.jboss.org/jbosstools/builds/staging/_composite_/core/trun... >> >> >> This is done because: >> >> * after the publish of the N build is done, before the bits are >> visible on download.jboss.org, the lag seems to vary from seconds to >> minutes. >> >> * a build-in-progress which resolved against the N-1 bits will fail if >> those bits suddenly vanish. >> >> --- >> >> For the stable branch, the same push-then-rename approach [0] is used >> (rather than destructively pushing on top of an existing build, as we >> did a few years ago), but we only composite [2] the latest (N) because: >> >> * stable branch builds change less often and >> * we want to guarantee that we're using the latest. >> >> [2] >> http://download.jboss.org/jbosstools/builds/staging/_composite_/core/4.1.... >> >> >> Hope that makes sense. If you think something can be improved w/o >> causing new problems (such as updating the timestamp in the >> composite*.xml files dynamically w/ every published build, so p2 sees >> it as "new" more often), please don't hesitate to open a JIRA w/ >> details on what you'd change, and how. >> >> Cheers, >> >> Nick >> >> On 05/21/2013 02:12 PM, Denis Golovin wrote: >>> On 05/21/2013 10:32 AM, Mickael Istria wrote: >>>> On 05/21/2013 06:56 PM, Denis Golovin wrote: >>>>> The problem is org.jboss.tools.tests is not part of >>>>> >>>>> http://download.jboss.org/jbosstools/updates/nightly/core/trunk/plugins/ >>>>> >>>>> >>>> That's right. However it's part of >>>> http://download.jboss.org/jbosstools/updates/nightly/integrationtests/trunk >>>> >>>> >>>> site, so when enabling no profile, the default nightly sites (core >>>> and >>>> integration tests) are used, so it should resolve this bundle. >>>> Is this something you can reproduce at every build? It could happen >>>> when your build tries to get content at the same time aggregation is >>>> getting published. >>> >>> Pushing process can be implemented different way to avoid this >>> issue, I >>> saw it many times and it is a bit annoying. I remember we spend some >>> time with Nick to tackle it, but problem seems still here. >>> >>> Idea behind this fix is really simple: >>> 1. Leave old published bits as is on the server side and just >>> upload new >>> ones >>> 2. Update compositeArtifacts.xml first and include uploaded update >>> site >>> from step 1 >>> 3. Update compositeContent.xml: include new uploaded update site from >>> step 1 and remove oldest one >>> 4. Update compositeArtifacts.xml and remove oldest one >>> 5. Remove oldest update site folder >>> >>> Note there are no operations related to renaming/moving previously >>> uploaded update sites and that the key point to have previously >>> uploaded >>> sites available while new one is in uploading stage. >>> >>> It should significantly reduce amount of errors because we keep two >>> update sites for each jbosstools-module, so at least one update >>> site is >>> always available through composite update site for module. There could >>> be still problems for builds with slow connection, but connection >>> should >>> be slow enough to live through at least two builds for jbosstools >>> module. This could be implemented as maven plug-n and number of builds >>> to keep in composite could be a good candidate for configuration >>> parameters. >>> >>> It is not really critical but nice to have. >>> >>> Denis >>> >>>> -- >>>> Mickael Istria >>>> Eclipse developer at JBoss, by Red Hat <http://www.jboss.org/tools> >>>> My blog <http://mickaelistria.wordpress.com> - My Tweets >>>> <http://twitter.com/mickaelistria> >>>> >>>> >>>> _______________________________________________ >>>> jbosstools-dev mailing list >>>> jbosstools-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/jbosstools-dev >>> >>> >>> >>> _______________________________________________ >>> jbosstools-dev mailing list >>> jbosstools-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/jbosstools-dev >>> >> >