On Tue, May 21, 2013 at 03:19:18PM +0300, Konstantin Marmalyukov wrote: > I'm trying to build org.jboss.tools.vpe.browsersim.eclipse.test > locally to reproduce its failure on Hudson. But I got an error* > > */[ERROR] Cannot resolve project dependencies:// > //[ERROR] Software being installed: > org.jboss.tools.vpe.browsersim.eclipse.test////3.5.0.qualifier// > //[ERROR] Missing requirement: > org.jboss.tools.vpe.browsersim.eclipse.test 3.5.0.qualifier requires > 'bundle org.jboss.tools.tests 3.5.0' but it could not be found/ > > During local built all dependencies must be resolved by tycho and > downloaded from our servers, am I right? How this problem can be fixed? Which branch are you building ? Can you pastebin the full log ? My guess is you built just in the time between clean of the staging area was done so the composite site don't have these bits yet - but hard to say without knowing when and what you built exactly. /max

The problem is org.jboss.tools.tests is not part of http://download.jboss.org/jbosstools/updates/nightly/core/trunk/plugins/

On 05/21/2013 10:32 AM, Mickael Istria wrote: > On 05/21/2013 06:56 PM, Denis Golovin wrote: >> The problem is org.jboss.tools.tests is not part of >> >> http://download.jboss.org/jbosstools/updates/nightly/core/trunk/plugins/ > That's right. However it's part of > http://download.jboss.org/jbosstools/updates/nightly/integrationtests/trunk > site, so when enabling no profile, the default nightly sites (core and > integration tests) are used, so it should resolve this bundle. > Is this something you can reproduce at every build? It could happen > when your build tries to get content at the same time aggregation is > getting published. Pushing process can be implemented different way to avoid this issue, I saw it many times and it is a bit annoying. I remember we spend some time with Nick to tackle it, but problem seems still here. Idea behind this fix is really simple: 1. Leave old published bits as is on the server side and just upload new ones 2. Update compositeArtifacts.xml first and include uploaded update site from step 1 3. Update compositeContent.xml: include new uploaded update site from step 1 and remove oldest one 4. Update compositeArtifacts.xml and remove oldest one 5. Remove oldest update site folder Note there are no operations related to renaming/moving previously uploaded update sites and that the key point to have previously uploaded sites available while new one is in uploading stage. It should significantly reduce amount of errors because we keep two update sites for each jbosstools-module, so at least one update site is always available through composite update site for module. There could be still problems for builds with slow connection, but connection should be slow enough to live through at least two builds for jbosstools module. This could be implemented as maven plug-n and number of builds to keep in composite could be a good candidate for configuration parameters. It is not really critical but nice to have. Denis > -- > Mickael Istria > Eclipse developer at JBoss, by Red Hat <http://www.jboss.org/tools> > My blog <http://mickaelistria.wordpress.com> - My Tweets > <http://twitter.com/mickaelistria> > > > _______________________________________________ > jbosstools-dev mailing list > jbosstools-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jbosstools-dev _______________________________________________ jbosstools-dev mailing list jbosstools-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jbosstools-dev

> The stated workflow is already being done [0], except that today: > * the current (N) and previous (N-1) are composited together into a > single site [1] > * the project's previous build is NOT then removed from the composite > & deleted once the new build is done publishing What I am trying to tell for a while is having names staging/staging.previous in update site and physically move content of staging to staging.previous after every build is not working. In fact with this approach local/jenkins build failures are unpredictable, when dev/jenkins runs a long build lets say javaee and suddenly on remote server publis.sh L585 moves bits like that mv \ $DESTINATION/builds/staging/${JOB_NAME} \ $DESTINATION/builds/staging.previous/${JOB_NAME} build fails.

Considering it takes couple hours to execute and you have to do that again and sometimes again until it works, it is not really convenient.

> * the composite*.xml metadata files are not regenerated when > publishing, but are *manually* updated if/when a new component is > added or job is renamed well, they should be updated because: 1. It would let to avoid moving huge p2repos around and wait while they are in sync with download.jboss.org 2. It would take much less time for *.xml files to be available on download.jboss.org

bing! builds would stop failing so often.

In fact for the future we can even save space by publishing only changed bits (need to fix qualifier for features/plugins) and it would let to: 1. Decrease time for syncing with download.jboss.org; 2. Have installation history actually working and let people use nightly updates and roll back to previous version if something is wrong; 3. Increase speed of local/jenkins builds, because no more full downloads for dependencies

> > [0] > https://github.com/jbosstools/jbosstools-build-ci/blob/master/publish/pub... > > [1] > http://download.jboss.org/jbosstools/builds/staging/_composite_/core/trun... > > This is done because: > > * after the publish of the N build is done, before the bits are > visible on download.jboss.org, the lag seems to vary from seconds to > minutes. > > * a build-in-progress which resolved against the N-1 bits will fail if > those bits suddenly vanish. > > --- > > For the stable branch, the same push-then-rename approach [0] is used > (rather than destructively pushing on top of an existing build, as we > did a few years ago), but we only composite [2] the latest (N) because: > > * stable branch builds change less often and > * we want to guarantee that we're using the latest. > > [2] > http://download.jboss.org/jbosstools/builds/staging/_composite_/core/4.1.... > > Hope that makes sense. If you think something can be improved w/o > causing new problems (such as updating the timestamp in the > composite*.xml files dynamically w/ every published build, so p2 sees > it as "new" more often), please don't hesitate to open a JIRA w/ > details on what you'd change, and how. > > Cheers, > > Nick > > On 05/21/2013 02:12 PM, Denis Golovin wrote: >> On 05/21/2013 10:32 AM, Mickael Istria wrote: >>> On 05/21/2013 06:56 PM, Denis Golovin wrote: >>>> The problem is org.jboss.tools.tests is not part of >>>> >>>> http://download.jboss.org/jbosstools/updates/nightly/core/trunk/plugins/ >>>> >>> That's right. However it's part of >>> http://download.jboss.org/jbosstools/updates/nightly/integrationtests/trunk >>> >>> site, so when enabling no profile, the default nightly sites (core and >>> integration tests) are used, so it should resolve this bundle. >>> Is this something you can reproduce at every build? It could happen >>> when your build tries to get content at the same time aggregation is >>> getting published. >> >> Pushing process can be implemented different way to avoid this issue, I >> saw it many times and it is a bit annoying. I remember we spend some >> time with Nick to tackle it, but problem seems still here. >> >> Idea behind this fix is really simple: >> 1. Leave old published bits as is on the server side and just upload new >> ones >> 2. Update compositeArtifacts.xml first and include uploaded update site >> from step 1 >> 3. Update compositeContent.xml: include new uploaded update site from >> step 1 and remove oldest one >> 4. Update compositeArtifacts.xml and remove oldest one >> 5. Remove oldest update site folder >> >> Note there are no operations related to renaming/moving previously >> uploaded update sites and that the key point to have previously uploaded >> sites available while new one is in uploading stage. >> >> It should significantly reduce amount of errors because we keep two >> update sites for each jbosstools-module, so at least one update site is >> always available through composite update site for module. There could >> be still problems for builds with slow connection, but connection should >> be slow enough to live through at least two builds for jbosstools >> module. This could be implemented as maven plug-n and number of builds >> to keep in composite could be a good candidate for configuration >> parameters. >> >> It is not really critical but nice to have. >> >> Denis >> >>> -- >>> Mickael Istria >>> Eclipse developer at JBoss, by Red Hat <http://www.jboss.org/tools> >>> My blog <http://mickaelistria.wordpress.com> - My Tweets >>> <http://twitter.com/mickaelistria> >>> >>> >>> _______________________________________________ >>> jbosstools-dev mailing list >>> jbosstools-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/jbosstools-dev >> >> >> >> _______________________________________________ >> jbosstools-dev mailing list >> jbosstools-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jbosstools-dev >> >

On 14/08/2013, at 02.32, Denis Golovin <dgolovin(a)exadel.com&gt; wrote: On 08/13/2013 12:24 PM, Nick Boldt wrote: >>> The stated workflow is already being done [0], except that today: >>> * the current (N) and previous (N-1) are composited together into a >>> single site [1] >>> * the project's previous build is NOT then removed from the composite >>> & deleted once the new build is done publishing >> What I am trying to tell for a while is having names >> staging/staging.previous in update site and physically move content of >> staging to staging.previous after every build is not working. In fact >> with this approach local/jenkins build failures are unpredictable, when >> dev/jenkins runs a long build lets say javaee and suddenly on remote >> server publis.sh L585 moves bits like that >> mv \ >> $DESTINATION/builds/staging/${JOB_NAME} \ >> $DESTINATION/builds/staging.previous/${JOB_NAME} >> >> build fails. > > What it actually does is: > > 1. rsync new build into staging/${JOB_NAME}.next (slow) > 2. delete staging.previous/${JOB_NAME} (slow) > 3. move staging/${JOB_NAME} to staging.previous/${JOB_NAME} (fast) > 4. move staging/${JOB_NAME}.next to staging/${JOB_NAME} (fast) > > Therefore the impact on the composite site is only a few seconds' > outage, unless your build was depending on content in the > *staging.previous site*, in which case, yes, you will likely end up > broken. > > This mechanism allows builds in progress, which depend on the CURRENT > Nth build to continue to work even after the new N build replaces the > previous one. I am not following you here. While builds in progress is not affected here? If build downloaded metadata from composite update site and picked up latest one (which is staging/${JOB_NAME}) and started downloading artifacts it would fail in case of (3). > > For example, if you kick a mvn build, which pings the composite site > to resolve dependencies and sees that staging = B123 and > staging.previous = B122, your build begins fine. But if while you're > building/resolving a new B124 gets published, you'll STILL be fine to > depend on B123 bits, just not the now-deleted B122 bits. Restarting > your job will always cause the dep resolution to restart, whereupon > the metadata will be scanned again and you'll then be building against > staging = B124 and staging.previous = B123. > >> Considering it takes couple hours to execute and you have >> to do that again and sometimes again until it works, it is not really >> convenient. > > So are you proposing that instead of an in-place move which reuses > generic folder names like "staging" and "staging.previous", we > composite build output using unique names like > 2013-08-09_05-05-26-B7222/ or 2013-08-13_10-05-28-B7255? yes > > If so, we would need: > > a) to regenerate the composite site each time there's a new build > published, in order to remove the oldest and add the newest (keeping > only the Nth and N-1rst builds) > > (I have a script that might already work for this, or would need > tweaking.) yes, would be good to be able pass number builds to keep as a parameter. > > b) heuristics to determine when an older (N-2, N-3, ... N-z) build is > no longer needed, perhaps simply by assuming no one needs it after 24hrs? > > (Once we agree, this is trivial.) 24 ours should be more that enough. > > c) a cleanup script which can purge all but the builds which are no > more than 1 day old, keeping at all times at least two builds (N and N-1) > > (I have a script that already does this for folders like > http://download.jboss.org/jbosstools/builds/nightly/core/trunk/ but > might need to be tweaked to work for a new pattern of > staging/${JOB_NAME}/<BUILD_ID>/ .) yes > > >>> * the composite*.xml metadata files are not regenerated when >>> publishing, but are *manually* updated if/when a new component is >>> added or job is renamed >> well, they should be updated because: >> 1. It would let to avoid moving huge p2repos around and wait while they >> are in sync with download.jboss.org >> 2. It would take much less time for *.xml files to be available on >> download.jboss.org > > Actually, the time to publish the bits from Jenkins to > download.jboss.org would not change. We would still need to push bits > to the server, and perform cleanup of old builds. Generating the > composite*.xml files would add some (negligible) time to the process, > but would guarantee that a p2 or tycho user would always get a fresh > timestamp. > > Currently, as the composite*.xml files do not change regularly, there > is NO time required to make them available. So, this would actually > introduce some delay, but as I said you'd get fresher files, so it's > perhaps an improvement. > >> bing! builds would stop failing so often. > > We can certainly try this and verify your hypothesis. :) > >> In fact for the future we can even save space by publishing only changed >> bits (need to fix qualifier for features/plugins) and it would let to: >> 1. Decrease time for syncing with download.jboss.org; >> 2. Have installation history actually working and let people use nightly >> updates and roll back to previous version if something is wrong; >> 3. Increase speed of local/jenkins builds, because no more full >> downloads for dependencies > > Publishing only changes to a repo would be nice except that it would > mean publishing *destructively on top of existing snapshots*, rather > than creating new ones. So anyone who was currently building against > the N build would probably end up with a broken build as new IUs were > pushed into that build's repo, and its metadata overwritten. publishing on top should not be required. > > That's definitely worse than what we have today. And would, instead of > having timestamped folders which are unique and don't overlap, put us > back where we are today with reusable, generic folder names like > "staging/${JOB_NAME}". > > So, I'm all for a new approach to compositing with regenerated > composite*.xml files, pointing to UNIQUELY VERSIONED folders (rather > than generic reusable ones), then doing cleanup of older N-2, N-3 > builds, but I disagree that publishing changed bits into existing > repos is a good idea (even if you use rsync --delete to purge the > orphaned IUs, you'll still end up breaking more builds this way), > which is why we moved AWAY from this approach by implementing the > staging.next/staging/staging.previous shell game. This incremental update is tricky to implement and not a priority for us IMO. Denis > > N > >>> >>> [0] >>> https://github.com/jbosstools/jbosstools-build-ci/blob/master/publish/pub... >>> >>> >>> [1] >>> http://download.jboss.org/jbosstools/builds/staging/_composite_/core/trun... >>> >>> >>> This is done because: >>> >>> * after the publish of the N build is done, before the bits are >>> visible on download.jboss.org, the lag seems to vary from seconds to >>> minutes. >>> >>> * a build-in-progress which resolved against the N-1 bits will fail if >>> those bits suddenly vanish. >>> >>> --- >>> >>> For the stable branch, the same push-then-rename approach [0] is used >>> (rather than destructively pushing on top of an existing build, as we >>> did a few years ago), but we only composite [2] the latest (N) because: >>> >>> * stable branch builds change less often and >>> * we want to guarantee that we're using the latest. >>> >>> [2] >>> http://download.jboss.org/jbosstools/builds/staging/_composite_/core/4.1.... >>> >>> >>> Hope that makes sense. If you think something can be improved w/o >>> causing new problems (such as updating the timestamp in the >>> composite*.xml files dynamically w/ every published build, so p2 sees >>> it as "new" more often), please don't hesitate to open a JIRA w/ >>> details on what you'd change, and how. >>> >>> Cheers, >>> >>> Nick >>> >>>> On 05/21/2013 02:12 PM, Denis Golovin wrote: >>>>> On 05/21/2013 10:32 AM, Mickael Istria wrote: >>>>>> On 05/21/2013 06:56 PM, Denis Golovin wrote: >>>>>> The problem is org.jboss.tools.tests is not part of >>>>>> >>>>>> http://download.jboss.org/jbosstools/updates/nightly/core/trunk/plugins/ >>>>> That's right. However it's part of >>>>> http://download.jboss.org/jbosstools/updates/nightly/integrationtests/trunk >>>>> >>>>> >>>>> site, so when enabling no profile, the default nightly sites (core >>>>> and >>>>> integration tests) are used, so it should resolve this bundle. >>>>> Is this something you can reproduce at every build? It could happen >>>>> when your build tries to get content at the same time aggregation is >>>>> getting published. >>>> >>>> Pushing process can be implemented different way to avoid this >>>> issue, I >>>> saw it many times and it is a bit annoying. I remember we spend some >>>> time with Nick to tackle it, but problem seems still here. >>>> >>>> Idea behind this fix is really simple: >>>> 1. Leave old published bits as is on the server side and just >>>> upload new >>>> ones >>>> 2. Update compositeArtifacts.xml first and include uploaded update >>>> site >>>> from step 1 >>>> 3. Update compositeContent.xml: include new uploaded update site from >>>> step 1 and remove oldest one >>>> 4. Update compositeArtifacts.xml and remove oldest one >>>> 5. Remove oldest update site folder >>>> >>>> Note there are no operations related to renaming/moving previously >>>> uploaded update sites and that the key point to have previously >>>> uploaded >>>> sites available while new one is in uploading stage. >>>> >>>> It should significantly reduce amount of errors because we keep two >>>> update sites for each jbosstools-module, so at least one update >>>> site is >>>> always available through composite update site for module. There could >>>> be still problems for builds with slow connection, but connection >>>> should >>>> be slow enough to live through at least two builds for jbosstools >>>> module. This could be implemented as maven plug-n and number of builds >>>> to keep in composite could be a good candidate for configuration >>>> parameters. >>>> >>>> It is not really critical but nice to have. >>>> >>>> Denis >>>> >>>>> -- >>>>> Mickael Istria >>>>> Eclipse developer at JBoss, by Red Hat <http://www.jboss.org/tools> >>>>> My blog <http://mickaelistria.wordpress.com> - My Tweets >>>>> <http://twitter.com/mickaelistria> >>>>> >>>>> >>>>> _______________________________________________ >>>>> jbosstools-dev mailing list >>>>> jbosstools-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/jbosstools-dev >>>> >>>> >>>> >>>> _______________________________________________ >>>> jbosstools-dev mailing list >>>> jbosstools-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/jbosstools-dev _______________________________________________ jbosstools-dev mailing list jbosstools-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jbosstools-dev

This sounds like something we should get done during the current "down time" before we're in the middle of a new Alpha1 code freeze cycle. Denis, is there an existing JIRA for this, or would you like me to open one? N On 08/14/2013 12:48 AM, Max Andersen wrote: > Just to say I agree with Denis overall idea and on nicks specifics. > This approach about updating the ci-builds composite to keep ongoing > builds running is what we discussed at the f2f. > > /max (sent from my phone) > N > >> On 14/08/2013, at 02.32, Denis Golovin <dgolovin(a)exadel.com&gt; wrote: >> >> On 08/13/2013 12:24 PM, Nick Boldt wrote: >>>>> The stated workflow is already being done [0], except that today: >>>>> * the current (N) and previous (N-1) are composited together into a >>>>> single site [1] >>>>> * the project's previous build is NOT then removed from the composite >>>>> & deleted once the new build is done publishing >>>> What I am trying to tell for a while is having names >>>> staging/staging.previous in update site and physically move content of >>>> staging to staging.previous after every build is not working. In fact >>>> with this approach local/jenkins build failures are unpredictable, >>>> when >>>> dev/jenkins runs a long build lets say javaee and suddenly on remote >>>> server publis.sh L585 moves bits like that >>>> mv \ >>>> $DESTINATION/builds/staging/${JOB_NAME} \ >>>> $DESTINATION/builds/staging.previous/${JOB_NAME} >>>> >>>> build fails. >>> >>> What it actually does is: >>> >>> 1. rsync new build into staging/${JOB_NAME}.next (slow) >>> 2. delete staging.previous/${JOB_NAME} (slow) >>> 3. move staging/${JOB_NAME} to staging.previous/${JOB_NAME} (fast) >>> 4. move staging/${JOB_NAME}.next to staging/${JOB_NAME} (fast) >>> >>> Therefore the impact on the composite site is only a few seconds' >>> outage, unless your build was depending on content in the >>> *staging.previous site*, in which case, yes, you will likely end up >>> broken. >>> >>> This mechanism allows builds in progress, which depend on the CURRENT >>> Nth build to continue to work even after the new N build replaces the >>> previous one. >> I am not following you here. While builds in progress is not affected >> here? If build downloaded metadata from composite update site and picked >> up latest one (which is staging/${JOB_NAME}) and started downloading >> artifacts it would fail in case of (3). >>> >>> For example, if you kick a mvn build, which pings the composite site >>> to resolve dependencies and sees that staging = B123 and >>> staging.previous = B122, your build begins fine. But if while you're >>> building/resolving a new B124 gets published, you'll STILL be fine to >>> depend on B123 bits, just not the now-deleted B122 bits. Restarting >>> your job will always cause the dep resolution to restart, whereupon >>> the metadata will be scanned again and you'll then be building against >>> staging = B124 and staging.previous = B123. >>> >>>> Considering it takes couple hours to execute and you have >>>> to do that again and sometimes again until it works, it is not really >>>> convenient. >>> >>> So are you proposing that instead of an in-place move which reuses >>> generic folder names like "staging" and "staging.previous", we >>> composite build output using unique names like >>> 2013-08-09_05-05-26-B7222/ or 2013-08-13_10-05-28-B7255? >> yes >>> >>> If so, we would need: >>> >>> a) to regenerate the composite site each time there's a new build >>> published, in order to remove the oldest and add the newest (keeping >>> only the Nth and N-1rst builds) >>> >>> (I have a script that might already work for this, or would need >>> tweaking.) >> yes, would be good to be able pass number builds to keep as a parameter. >>> >>> b) heuristics to determine when an older (N-2, N-3, ... N-z) build is >>> no longer needed, perhaps simply by assuming no one needs it after >>> 24hrs? >>> >>> (Once we agree, this is trivial.) >> 24 ours should be more that enough. >>> >>> c) a cleanup script which can purge all but the builds which are no >>> more than 1 day old, keeping at all times at least two builds (N and >>> N-1) >>> >>> (I have a script that already does this for folders like >>> http://download.jboss.org/jbosstools/builds/nightly/core/trunk/ but >>> might need to be tweaked to work for a new pattern of >>> staging/${JOB_NAME}/<BUILD_ID>/ .) >> >> yes >> >>> >>> >>>>> * the composite*.xml metadata files are not regenerated when >>>>> publishing, but are *manually* updated if/when a new component is >>>>> added or job is renamed >>>> well, they should be updated because: >>>> 1. It would let to avoid moving huge p2repos around and wait while >>>> they >>>> are in sync with download.jboss.org >>>> 2. It would take much less time for *.xml files to be available on >>>> download.jboss.org >>> >>> Actually, the time to publish the bits from Jenkins to >>> download.jboss.org would not change. We would still need to push bits >>> to the server, and perform cleanup of old builds. Generating the >>> composite*.xml files would add some (negligible) time to the process, >>> but would guarantee that a p2 or tycho user would always get a fresh >>> timestamp. >>> >>> Currently, as the composite*.xml files do not change regularly, there >>> is NO time required to make them available. So, this would actually >>> introduce some delay, but as I said you'd get fresher files, so it's >>> perhaps an improvement. >>> >>>> bing! builds would stop failing so often. >>> >>> We can certainly try this and verify your hypothesis. :) >>> >>>> In fact for the future we can even save space by publishing only >>>> changed >>>> bits (need to fix qualifier for features/plugins) and it would let to: >>>> 1. Decrease time for syncing with download.jboss.org; >>>> 2. Have installation history actually working and let people use >>>> nightly >>>> updates and roll back to previous version if something is wrong; >>>> 3. Increase speed of local/jenkins builds, because no more full >>>> downloads for dependencies >>> >>> Publishing only changes to a repo would be nice except that it would >>> mean publishing *destructively on top of existing snapshots*, rather >>> than creating new ones. So anyone who was currently building against >>> the N build would probably end up with a broken build as new IUs were >>> pushed into that build's repo, and its metadata overwritten. >> >> publishing on top should not be required. >> >>> >>> That's definitely worse than what we have today. And would, instead of >>> having timestamped folders which are unique and don't overlap, put us >>> back where we are today with reusable, generic folder names like >>> "staging/${JOB_NAME}". >>> >>> So, I'm all for a new approach to compositing with regenerated >>> composite*.xml files, pointing to UNIQUELY VERSIONED folders (rather >>> than generic reusable ones), then doing cleanup of older N-2, N-3 >>> builds, but I disagree that publishing changed bits into existing >>> repos is a good idea (even if you use rsync --delete to purge the >>> orphaned IUs, you'll still end up breaking more builds this way), >>> which is why we moved AWAY from this approach by implementing the >>> staging.next/staging/staging.previous shell game. >> >> This incremental update is tricky to implement and not a priority for us >> IMO. >> >> Denis >> >>> >>> N >>> >>>>> >>>>> [0] >>>>> https://github.com/jbosstools/jbosstools-build-ci/blob/master/publish/pub... >>>>> >>>>> >>>>> >>>>> [1] >>>>> http://download.jboss.org/jbosstools/builds/staging/_composite_/core/trun... >>>>> >>>>> >>>>> >>>>> This is done because: >>>>> >>>>> * after the publish of the N build is done, before the bits are >>>>> visible on download.jboss.org, the lag seems to vary from seconds to >>>>> minutes. >>>>> >>>>> * a build-in-progress which resolved against the N-1 bits will >>>>> fail if >>>>> those bits suddenly vanish. >>>>> >>>>> --- >>>>> >>>>> For the stable branch, the same push-then-rename approach [0] is used >>>>> (rather than destructively pushing on top of an existing build, as we >>>>> did a few years ago), but we only composite [2] the latest (N) >>>>> because: >>>>> >>>>> * stable branch builds change less often and >>>>> * we want to guarantee that we're using the latest. >>>>> >>>>> [2] >>>>> http://download.jboss.org/jbosstools/builds/staging/_composite_/core/4.1.... >>>>> >>>>> >>>>> >>>>> Hope that makes sense. If you think something can be improved w/o >>>>> causing new problems (such as updating the timestamp in the >>>>> composite*.xml files dynamically w/ every published build, so p2 sees >>>>> it as "new" more often), please don't hesitate to open a JIRA w/ >>>>> details on what you'd change, and how. >>>>> >>>>> Cheers, >>>>> >>>>> Nick >>>>> >>>>>> On 05/21/2013 02:12 PM, Denis Golovin wrote: >>>>>>> On 05/21/2013 10:32 AM, Mickael Istria wrote: >>>>>>>> On 05/21/2013 06:56 PM, Denis Golovin wrote: >>>>>>>> The problem is org.jboss.tools.tests is not part of >>>>>>>> >>>>>>>> http://download.jboss.org/jbosstools/updates/nightly/core/trunk/plugins/ >>>>>>>> >>>>>>> That's right. However it's part of >>>>>>> http://download.jboss.org/jbosstools/updates/nightly/integrationtests/trunk >>>>>>> >>>>>>> >>>>>>> >>>>>>> site, so when enabling no profile, the default nightly sites (core >>>>>>> and >>>>>>> integration tests) are used, so it should resolve this bundle. >>>>>>> Is this something you can reproduce at every build? It could happen >>>>>>> when your build tries to get content at the same time >>>>>>> aggregation is >>>>>>> getting published. >>>>>> >>>>>> Pushing process can be implemented different way to avoid this >>>>>> issue, I >>>>>> saw it many times and it is a bit annoying. I remember we spend some >>>>>> time with Nick to tackle it, but problem seems still here. >>>>>> >>>>>> Idea behind this fix is really simple: >>>>>> 1. Leave old published bits as is on the server side and just >>>>>> upload new >>>>>> ones >>>>>> 2. Update compositeArtifacts.xml first and include uploaded update >>>>>> site >>>>>> from step 1 >>>>>> 3. Update compositeContent.xml: include new uploaded update site >>>>>> from >>>>>> step 1 and remove oldest one >>>>>> 4. Update compositeArtifacts.xml and remove oldest one >>>>>> 5. Remove oldest update site folder >>>>>> >>>>>> Note there are no operations related to renaming/moving previously >>>>>> uploaded update sites and that the key point to have previously >>>>>> uploaded >>>>>> sites available while new one is in uploading stage. >>>>>> >>>>>> It should significantly reduce amount of errors because we keep two >>>>>> update sites for each jbosstools-module, so at least one update >>>>>> site is >>>>>> always available through composite update site for module. There >>>>>> could >>>>>> be still problems for builds with slow connection, but connection >>>>>> should >>>>>> be slow enough to live through at least two builds for jbosstools >>>>>> module. This could be implemented as maven plug-n and number of >>>>>> builds >>>>>> to keep in composite could be a good candidate for configuration >>>>>> parameters. >>>>>> >>>>>> It is not really critical but nice to have. >>>>>> >>>>>> Denis >>>>>> >>>>>>> -- >>>>>>> Mickael Istria >>>>>>> Eclipse developer at JBoss, by Red Hat <http://www.jboss.org/tools> >>>>>>> My blog <http://mickaelistria.wordpress.com> - My Tweets >>>>>>> <http://twitter.com/mickaelistria> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> jbosstools-dev mailing list >>>>>>> jbosstools-dev(a)lists.jboss.org >>>>>>> https://lists.jboss.org/mailman/listinfo/jbosstools-dev >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> jbosstools-dev mailing list >>>>>> jbosstools-dev(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/jbosstools-dev >> >> _______________________________________________ >> jbosstools-dev mailing list >> jbosstools-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jbosstools-dev