[jbosstools-dev] Tests error during local build

Tue Aug 13 15:24:34 EDT 2013

>> The stated workflow is already being done [0], except that today:
>> * the current (N) and previous (N-1) are composited together into a
>> single site [1]
>> * the project's previous build is NOT then removed from the composite
>> & deleted once the new build is done publishing
> What I am trying to tell for a while is having names
> staging/staging.previous in update site and physically move content of
> staging to staging.previous after every build is not working. In fact
> with this approach local/jenkins build failures are unpredictable, when
> dev/jenkins runs a long build lets say javaee and suddenly on remote
> server publis.sh L585 moves bits like that
>   mv \
> $DESTINATION/builds/staging/${JOB_NAME} \
 > $DESTINATION/builds/staging.previous/${JOB_NAME}
>
> build fails.

What it actually does is:

1. rsync new build into staging/${JOB_NAME}.next (slow)
2. delete staging.previous/${JOB_NAME} (slow)
3. move staging/${JOB_NAME} to staging.previous/${JOB_NAME} (fast)
4. move staging/${JOB_NAME}.next to staging/${JOB_NAME} (fast)

Therefore the impact on the composite site is only a few seconds' 
outage, unless your build was depending on content in the 
*staging.previous site*, in which case, yes, you will likely end up broken.

This mechanism allows builds in progress, which depend on the CURRENT 
Nth build to continue to work even after the new N build replaces the 
previous one.

For example, if you kick a mvn build, which pings the composite site to 
resolve dependencies and sees that staging = B123 and staging.previous = 
B122, your build begins fine. But if while you're building/resolving a 
new B124 gets published, you'll STILL be fine to depend on B123 bits, 
just not the now-deleted B122 bits. Restarting your job will always 
cause the dep resolution to restart, whereupon the metadata will be 
scanned again and you'll then be building against staging = B124 and 
staging.previous = B123.

> Considering it takes couple hours to execute and you have
> to do that again and sometimes again until it works, it is not really
> convenient.

So are you proposing that instead of an in-place move which reuses 
generic folder names like "staging" and "staging.previous", we composite 
build output using unique names like 2013-08-09_05-05-26-B7222/ or 
2013-08-13_10-05-28-B7255?

If so, we would need:

a) to regenerate the composite site each time there's a new build 
published, in order to remove the oldest and add the newest (keeping 
only the Nth and N-1rst builds)

(I have a script that might already work for this, or would need tweaking.)

b) heuristics to determine when an older (N-2, N-3, ... N-z) build is no 
longer needed, perhaps simply by assuming no one needs it after 24hrs?

(Once we agree, this is trivial.)

c) a cleanup script which can purge all but the builds which are no more 
than 1 day old, keeping at all times at least two builds (N and N-1)

(I have a script that already does this for folders like 
http://download.jboss.org/jbosstools/builds/nightly/core/trunk/ but 
might need to be tweaked to work for a new pattern of 
staging/${JOB_NAME}/<BUILD_ID>/ .)

>> * the composite*.xml metadata files are not regenerated when
>> publishing, but are *manually* updated if/when a new component is
>> added or job is renamed
> well, they should be updated because:
> 1. It would let to avoid moving huge p2repos around and wait while they
> are in sync with download.jboss.org
> 2. It would take much less time for *.xml files to be available on
> download.jboss.org

Actually, the time to publish the bits from Jenkins to 
download.jboss.org would not change. We would still need to push bits to 
the server, and perform cleanup of old builds. Generating the 
composite*.xml files would add some (negligible) time to the process, 
but would guarantee that a p2 or tycho user would always get a fresh 
timestamp.

Currently, as the composite*.xml files do not change regularly, there is 
NO time required to make them available. So, this would actually 
introduce some delay, but as I said you'd get fresher files, so it's 
perhaps an improvement.

> bing! builds would stop failing so often.

We can certainly try this and verify your hypothesis. :)

> In fact for the future we can even save space by publishing only changed
> bits (need to fix qualifier for features/plugins) and it would let to:
> 1. Decrease time for syncing with download.jboss.org;
> 2. Have installation history actually working and let people use nightly
> updates and roll back to previous version if something is wrong;
> 3. Increase speed of local/jenkins builds, because no more full
> downloads for dependencies

Publishing only changes to a repo would be nice except that it would 
mean publishing *destructively on top of existing snapshots*, rather 
than creating new ones. So anyone who was currently building against the 
N build would probably end up with a broken build as new IUs were pushed 
into that build's repo, and its metadata overwritten.

That's definitely worse than what we have today. And would, instead of 
having timestamped folders which are unique and don't overlap, put us 
back where we are today with reusable, generic folder names like 
"staging/${JOB_NAME}".

So, I'm all for a new approach to compositing with regenerated 
composite*.xml files, pointing to UNIQUELY VERSIONED folders (rather 
than generic reusable ones), then doing cleanup of older N-2, N-3 
builds, but I disagree that publishing changed bits into existing repos 
is a good idea (even if you use rsync --delete to purge the orphaned 
IUs, you'll still end up breaking more builds this way), which is why we 
moved AWAY from this approach by implementing the 
staging.next/staging/staging.previous shell game.

N

>>
>> [0]
>> https://github.com/jbosstools/jbosstools-build-ci/blob/master/publish/publish.sh#L541
>>
>> [1]
>> http://download.jboss.org/jbosstools/builds/staging/_composite_/core/trunk/compositeArtifacts.xml
>>
>> This is done because:
>>
>> * after the publish of the N build is done, before the bits are
>> visible on download.jboss.org, the lag seems to vary from seconds to
>> minutes.
>>
>> * a build-in-progress which resolved against the N-1 bits will fail if
>> those bits suddenly vanish.
>>
>> ---
>>
>> For the stable branch, the same push-then-rename approach [0] is used
>> (rather than destructively pushing on top of an existing build, as we
>> did a few years ago), but we only composite [2] the latest (N) because:
>>
>> * stable branch builds change less often and
>> * we want to guarantee that we're using the latest.
>>
>> [2]
>> http://download.jboss.org/jbosstools/builds/staging/_composite_/core/4.1.kepler/compositeArtifacts.xml
>>
>> Hope that makes sense. If you think something can be improved w/o
>> causing new problems (such as updating the timestamp in the
>> composite*.xml files dynamically w/ every published build, so p2 sees
>> it as "new" more often), please don't hesitate to open a JIRA w/
>> details on what you'd change, and how.
>>
>> Cheers,
>>
>> Nick
>>
>> On 05/21/2013 02:12 PM, Denis Golovin wrote:
>>> On 05/21/2013 10:32 AM, Mickael Istria wrote:
>>>> On 05/21/2013 06:56 PM, Denis Golovin wrote:
>>>>> The problem is org.jboss.tools.tests is not part of
>>>>>
>>>>> http://download.jboss.org/jbosstools/updates/nightly/core/trunk/plugins/
>>>>>
>>>> That's right. However it's part of
>>>> http://download.jboss.org/jbosstools/updates/nightly/integrationtests/trunk
>>>>
>>>> site, so when enabling no profile, the default nightly sites (core and
>>>> integration tests) are used, so it should resolve this bundle.
>>>> Is this something you can reproduce at every build? It could happen
>>>> when your build tries to get content at the same time aggregation is
>>>> getting published.
>>>
>>> Pushing process can be implemented different way to avoid this issue, I
>>> saw it many times and it is a bit annoying. I remember we spend some
>>> time with Nick to tackle it, but problem seems still here.
>>>
>>> Idea behind this fix is really simple:
>>> 1. Leave old published bits as is on the server side and just upload new
>>> ones
>>> 2. Update compositeArtifacts.xml first and include uploaded update site
>>> from step 1
>>> 3. Update compositeContent.xml: include new uploaded update site from
>>> step 1 and remove oldest one
>>> 4. Update compositeArtifacts.xml and remove oldest one
>>> 5. Remove oldest update site folder
>>>
>>> Note there are no operations related to renaming/moving previously
>>> uploaded update sites and that the key point to have previously uploaded
>>> sites available while new one is in uploading stage.
>>>
>>> It should significantly reduce amount of errors because we keep two
>>> update sites for each jbosstools-module, so at least one update site is
>>> always available through composite update site for module. There could
>>> be still problems for builds with slow connection, but connection should
>>> be slow enough to live through at least two builds for jbosstools
>>> module. This could be implemented as maven plug-n and number of builds
>>> to keep in composite could be a good candidate for configuration
>>> parameters.
>>>
>>> It is not really critical but nice to have.
>>>
>>> Denis
>>>
>>>> --
>>>> Mickael Istria
>>>> Eclipse developer at JBoss, by Red Hat <http://www.jboss.org/tools>
>>>> My blog <http://mickaelistria.wordpress.com> - My Tweets
>>>> <http://twitter.com/mickaelistria>
>>>>
>>>>
>>>> _______________________________________________
>>>> jbosstools-dev mailing list
>>>> jbosstools-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/jbosstools-dev
>>>
>>>
>>>
>>> _______________________________________________
>>> jbosstools-dev mailing list
>>> jbosstools-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/jbosstools-dev
>>>
>>
>

-- 
Nick Boldt :: JBoss by Red Hat
Productization Lead :: JBoss Tools & Dev Studio
http://nick.divbyzero.com