[jbosstools-dev] Tests error during local build

Tue Aug 13 20:31:43 EDT 2013

On 08/13/2013 12:24 PM, Nick Boldt wrote:
>>> The stated workflow is already being done [0], except that today:
>>> * the current (N) and previous (N-1) are composited together into a
>>> single site [1]
>>> * the project's previous build is NOT then removed from the composite
>>> & deleted once the new build is done publishing
>> What I am trying to tell for a while is having names
>> staging/staging.previous in update site and physically move content of
>> staging to staging.previous after every build is not working. In fact
>> with this approach local/jenkins build failures are unpredictable, when
>> dev/jenkins runs a long build lets say javaee and suddenly on remote
>> server publis.sh L585 moves bits like that
>>   mv \
>> $DESTINATION/builds/staging/${JOB_NAME} \
> > $DESTINATION/builds/staging.previous/${JOB_NAME}
>>
>> build fails.
>
> What it actually does is:
>
> 1. rsync new build into staging/${JOB_NAME}.next (slow)
> 2. delete staging.previous/${JOB_NAME} (slow)
> 3. move staging/${JOB_NAME} to staging.previous/${JOB_NAME} (fast)
> 4. move staging/${JOB_NAME}.next to staging/${JOB_NAME} (fast)
>
> Therefore the impact on the composite site is only a few seconds' 
> outage, unless your build was depending on content in the 
> *staging.previous site*, in which case, yes, you will likely end up 
> broken.
>
> This mechanism allows builds in progress, which depend on the CURRENT 
> Nth build to continue to work even after the new N build replaces the 
> previous one.
I am not following you here. While builds in progress is not affected 
here? If build downloaded metadata from composite update site and picked 
up latest one (which is staging/${JOB_NAME}) and started downloading 
artifacts it would fail in case of (3).
>
> For example, if you kick a mvn build, which pings the composite site 
> to resolve dependencies and sees that staging = B123 and 
> staging.previous = B122, your build begins fine. But if while you're 
> building/resolving a new B124 gets published, you'll STILL be fine to 
> depend on B123 bits, just not the now-deleted B122 bits. Restarting 
> your job will always cause the dep resolution to restart, whereupon 
> the metadata will be scanned again and you'll then be building against 
> staging = B124 and staging.previous = B123.
>
>> Considering it takes couple hours to execute and you have
>> to do that again and sometimes again until it works, it is not really
>> convenient.
>
> So are you proposing that instead of an in-place move which reuses 
> generic folder names like "staging" and "staging.previous", we 
> composite build output using unique names like 
> 2013-08-09_05-05-26-B7222/ or 2013-08-13_10-05-28-B7255?
yes
>
> If so, we would need:
>
> a) to regenerate the composite site each time there's a new build 
> published, in order to remove the oldest and add the newest (keeping 
> only the Nth and N-1rst builds)
>
> (I have a script that might already work for this, or would need 
> tweaking.)
yes, would be good to be able pass number builds to keep as a parameter.
>
> b) heuristics to determine when an older (N-2, N-3, ... N-z) build is 
> no longer needed, perhaps simply by assuming no one needs it after 24hrs?
>
> (Once we agree, this is trivial.)
24 ours should be more that enough.
>
> c) a cleanup script which can purge all but the builds which are no 
> more than 1 day old, keeping at all times at least two builds (N and N-1)
>
> (I have a script that already does this for folders like 
> http://download.jboss.org/jbosstools/builds/nightly/core/trunk/ but 
> might need to be tweaked to work for a new pattern of 
> staging/${JOB_NAME}/<BUILD_ID>/ .)

yes

>
>
>>> * the composite*.xml metadata files are not regenerated when
>>> publishing, but are *manually* updated if/when a new component is
>>> added or job is renamed
>> well, they should be updated because:
>> 1. It would let to avoid moving huge p2repos around and wait while they
>> are in sync with download.jboss.org
>> 2. It would take much less time for *.xml files to be available on
>> download.jboss.org
>
> Actually, the time to publish the bits from Jenkins to 
> download.jboss.org would not change. We would still need to push bits 
> to the server, and perform cleanup of old builds. Generating the 
> composite*.xml files would add some (negligible) time to the process, 
> but would guarantee that a p2 or tycho user would always get a fresh 
> timestamp.
>
> Currently, as the composite*.xml files do not change regularly, there 
> is NO time required to make them available. So, this would actually 
> introduce some delay, but as I said you'd get fresher files, so it's 
> perhaps an improvement.
>
>> bing! builds would stop failing so often.
>
> We can certainly try this and verify your hypothesis. :)
>
>> In fact for the future we can even save space by publishing only changed
>> bits (need to fix qualifier for features/plugins) and it would let to:
>> 1. Decrease time for syncing with download.jboss.org;
>> 2. Have installation history actually working and let people use nightly
>> updates and roll back to previous version if something is wrong;
>> 3. Increase speed of local/jenkins builds, because no more full
>> downloads for dependencies
>
> Publishing only changes to a repo would be nice except that it would 
> mean publishing *destructively on top of existing snapshots*, rather 
> than creating new ones. So anyone who was currently building against 
> the N build would probably end up with a broken build as new IUs were 
> pushed into that build's repo, and its metadata overwritten.

publishing on top should not be required.

>
> That's definitely worse than what we have today. And would, instead of 
> having timestamped folders which are unique and don't overlap, put us 
> back where we are today with reusable, generic folder names like 
> "staging/${JOB_NAME}".
>
> So, I'm all for a new approach to compositing with regenerated 
> composite*.xml files, pointing to UNIQUELY VERSIONED folders (rather 
> than generic reusable ones), then doing cleanup of older N-2, N-3 
> builds, but I disagree that publishing changed bits into existing 
> repos is a good idea (even if you use rsync --delete to purge the 
> orphaned IUs, you'll still end up breaking more builds this way), 
> which is why we moved AWAY from this approach by implementing the 
> staging.next/staging/staging.previous shell game.

This incremental update is tricky to implement and not a priority for us 
IMO.

Denis

>
> N
>
>>>
>>> [0]
>>> https://github.com/jbosstools/jbosstools-build-ci/blob/master/publish/publish.sh#L541 
>>>
>>>
>>> [1]
>>> http://download.jboss.org/jbosstools/builds/staging/_composite_/core/trunk/compositeArtifacts.xml 
>>>
>>>
>>> This is done because:
>>>
>>> * after the publish of the N build is done, before the bits are
>>> visible on download.jboss.org, the lag seems to vary from seconds to
>>> minutes.
>>>
>>> * a build-in-progress which resolved against the N-1 bits will fail if
>>> those bits suddenly vanish.
>>>
>>> ---
>>>
>>> For the stable branch, the same push-then-rename approach [0] is used
>>> (rather than destructively pushing on top of an existing build, as we
>>> did a few years ago), but we only composite [2] the latest (N) because:
>>>
>>> * stable branch builds change less often and
>>> * we want to guarantee that we're using the latest.
>>>
>>> [2]
>>> http://download.jboss.org/jbosstools/builds/staging/_composite_/core/4.1.kepler/compositeArtifacts.xml 
>>>
>>>
>>> Hope that makes sense. If you think something can be improved w/o
>>> causing new problems (such as updating the timestamp in the
>>> composite*.xml files dynamically w/ every published build, so p2 sees
>>> it as "new" more often), please don't hesitate to open a JIRA w/
>>> details on what you'd change, and how.
>>>
>>> Cheers,
>>>
>>> Nick
>>>
>>> On 05/21/2013 02:12 PM, Denis Golovin wrote:
>>>> On 05/21/2013 10:32 AM, Mickael Istria wrote:
>>>>> On 05/21/2013 06:56 PM, Denis Golovin wrote:
>>>>>> The problem is org.jboss.tools.tests is not part of
>>>>>>
>>>>>> http://download.jboss.org/jbosstools/updates/nightly/core/trunk/plugins/ 
>>>>>>
>>>>>>
>>>>> That's right. However it's part of
>>>>> http://download.jboss.org/jbosstools/updates/nightly/integrationtests/trunk 
>>>>>
>>>>>
>>>>> site, so when enabling no profile, the default nightly sites (core 
>>>>> and
>>>>> integration tests) are used, so it should resolve this bundle.
>>>>> Is this something you can reproduce at every build? It could happen
>>>>> when your build tries to get content at the same time aggregation is
>>>>> getting published.
>>>>
>>>> Pushing process can be implemented different way to avoid this 
>>>> issue, I
>>>> saw it many times and it is a bit annoying. I remember we spend some
>>>> time with Nick to tackle it, but problem seems still here.
>>>>
>>>> Idea behind this fix is really simple:
>>>> 1. Leave old published bits as is on the server side and just 
>>>> upload new
>>>> ones
>>>> 2. Update compositeArtifacts.xml first and include uploaded update 
>>>> site
>>>> from step 1
>>>> 3. Update compositeContent.xml: include new uploaded update site from
>>>> step 1 and remove oldest one
>>>> 4. Update compositeArtifacts.xml and remove oldest one
>>>> 5. Remove oldest update site folder
>>>>
>>>> Note there are no operations related to renaming/moving previously
>>>> uploaded update sites and that the key point to have previously 
>>>> uploaded
>>>> sites available while new one is in uploading stage.
>>>>
>>>> It should significantly reduce amount of errors because we keep two
>>>> update sites for each jbosstools-module, so at least one update 
>>>> site is
>>>> always available through composite update site for module. There could
>>>> be still problems for builds with slow connection, but connection 
>>>> should
>>>> be slow enough to live through at least two builds for jbosstools
>>>> module. This could be implemented as maven plug-n and number of builds
>>>> to keep in composite could be a good candidate for configuration
>>>> parameters.
>>>>
>>>> It is not really critical but nice to have.
>>>>
>>>> Denis
>>>>
>>>>> -- 
>>>>> Mickael Istria
>>>>> Eclipse developer at JBoss, by Red Hat <http://www.jboss.org/tools>
>>>>> My blog <http://mickaelistria.wordpress.com> - My Tweets
>>>>> <http://twitter.com/mickaelistria>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> jbosstools-dev mailing list
>>>>> jbosstools-dev at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/jbosstools-dev
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> jbosstools-dev mailing list
>>>> jbosstools-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/jbosstools-dev
>>>>
>>>
>>
>