[jbosstools-dev] Tests error during local build

[Max Andersen]

Just to say I agree with Denis overall idea and on nicks specifics. 
This approach about updating the ci-builds composite to keep ongoing builds running is what we discussed at the f2f.

/max (sent from my phone)
N

> On 14/08/2013, at 02.32, Denis Golovin <dgolovin at exadel.com> wrote:
> 
> On 08/13/2013 12:24 PM, Nick Boldt wrote:
>>>> The stated workflow is already being done [0], except that today:
>>>> * the current (N) and previous (N-1) are composited together into a
>>>> single site [1]
>>>> * the project's previous build is NOT then removed from the composite
>>>> & deleted once the new build is done publishing
>>> What I am trying to tell for a while is having names
>>> staging/staging.previous in update site and physically move content of
>>> staging to staging.previous after every build is not working. In fact
>>> with this approach local/jenkins build failures are unpredictable, when
>>> dev/jenkins runs a long build lets say javaee and suddenly on remote
>>> server publis.sh L585 moves bits like that
>>>  mv \
>>> $DESTINATION/builds/staging/${JOB_NAME} \
>>> $DESTINATION/builds/staging.previous/${JOB_NAME}
>>> 
>>> build fails.
>> 
>> What it actually does is:
>> 
>> 1. rsync new build into staging/${JOB_NAME}.next (slow)
>> 2. delete staging.previous/${JOB_NAME} (slow)
>> 3. move staging/${JOB_NAME} to staging.previous/${JOB_NAME} (fast)
>> 4. move staging/${JOB_NAME}.next to staging/${JOB_NAME} (fast)
>> 
>> Therefore the impact on the composite site is only a few seconds' 
>> outage, unless your build was depending on content in the 
>> *staging.previous site*, in which case, yes, you will likely end up 
>> broken.
>> 
>> This mechanism allows builds in progress, which depend on the CURRENT 
>> Nth build to continue to work even after the new N build replaces the 
>> previous one.
> I am not following you here. While builds in progress is not affected 
> here? If build downloaded metadata from composite update site and picked 
> up latest one (which is staging/${JOB_NAME}) and started downloading 
> artifacts it would fail in case of (3).
>> 
>> For example, if you kick a mvn build, which pings the composite site 
>> to resolve dependencies and sees that staging = B123 and 
>> staging.previous = B122, your build begins fine. But if while you're 
>> building/resolving a new B124 gets published, you'll STILL be fine to 
>> depend on B123 bits, just not the now-deleted B122 bits. Restarting 
>> your job will always cause the dep resolution to restart, whereupon 
>> the metadata will be scanned again and you'll then be building against 
>> staging = B124 and staging.previous = B123.
>> 
>>> Considering it takes couple hours to execute and you have
>>> to do that again and sometimes again until it works, it is not really
>>> convenient.
>> 
>> So are you proposing that instead of an in-place move which reuses 
>> generic folder names like "staging" and "staging.previous", we 
>> composite build output using unique names like 
>> 2013-08-09_05-05-26-B7222/ or 2013-08-13_10-05-28-B7255?
> yes
>> 
>> If so, we would need:
>> 
>> a) to regenerate the composite site each time there's a new build 
>> published, in order to remove the oldest and add the newest (keeping 
>> only the Nth and N-1rst builds)
>> 
>> (I have a script that might already work for this, or would need 
>> tweaking.)
> yes, would be good to be able pass number builds to keep as a parameter.
>> 
>> b) heuristics to determine when an older (N-2, N-3, ... N-z) build is 
>> no longer needed, perhaps simply by assuming no one needs it after 24hrs?
>> 
>> (Once we agree, this is trivial.)
> 24 ours should be more that enough.
>> 
>> c) a cleanup script which can purge all but the builds which are no 
>> more than 1 day old, keeping at all times at least two builds (N and N-1)
>> 
>> (I have a script that already does this for folders like 
>> http://download.jboss.org/jbosstools/builds/nightly/core/trunk/ but 
>> might need to be tweaked to work for a new pattern of 
>> staging/${JOB_NAME}/<BUILD_ID>/ .)
> 
> yes
> 
>> 
>> 
>>>> * the composite*.xml metadata files are not regenerated when
>>>> publishing, but are *manually* updated if/when a new component is
>>>> added or job is renamed
>>> well, they should be updated because:
>>> 1. It would let to avoid moving huge p2repos around and wait while they
>>> are in sync with download.jboss.org
>>> 2. It would take much less time for *.xml files to be available on
>>> download.jboss.org
>> 
>> Actually, the time to publish the bits from Jenkins to 
>> download.jboss.org would not change. We would still need to push bits 
>> to the server, and perform cleanup of old builds. Generating the 
>> composite*.xml files would add some (negligible) time to the process, 
>> but would guarantee that a p2 or tycho user would always get a fresh 
>> timestamp.
>> 
>> Currently, as the composite*.xml files do not change regularly, there 
>> is NO time required to make them available. So, this would actually 
>> introduce some delay, but as I said you'd get fresher files, so it's 
>> perhaps an improvement.
>> 
>>> bing! builds would stop failing so often.
>> 
>> We can certainly try this and verify your hypothesis. :)
>> 
>>> In fact for the future we can even save space by publishing only changed
>>> bits (need to fix qualifier for features/plugins) and it would let to:
>>> 1. Decrease time for syncing with download.jboss.org;
>>> 2. Have installation history actually working and let people use nightly
>>> updates and roll back to previous version if something is wrong;
>>> 3. Increase speed of local/jenkins builds, because no more full
>>> downloads for dependencies
>> 
>> Publishing only changes to a repo would be nice except that it would 
>> mean publishing *destructively on top of existing snapshots*, rather 
>> than creating new ones. So anyone who was currently building against 
>> the N build would probably end up with a broken build as new IUs were 
>> pushed into that build's repo, and its metadata overwritten.
> 
> publishing on top should not be required.
> 
>> 
>> That's definitely worse than what we have today. And would, instead of 
>> having timestamped folders which are unique and don't overlap, put us 
>> back where we are today with reusable, generic folder names like 
>> "staging/${JOB_NAME}".
>> 
>> So, I'm all for a new approach to compositing with regenerated 
>> composite*.xml files, pointing to UNIQUELY VERSIONED folders (rather 
>> than generic reusable ones), then doing cleanup of older N-2, N-3 
>> builds, but I disagree that publishing changed bits into existing 
>> repos is a good idea (even if you use rsync --delete to purge the 
>> orphaned IUs, you'll still end up breaking more builds this way), 
>> which is why we moved AWAY from this approach by implementing the 
>> staging.next/staging/staging.previous shell game.
> 
> This incremental update is tricky to implement and not a priority for us 
> IMO.
> 
> Denis
> 
>> 
>> N
>> 
>>>> 
>>>> [0]
>>>> https://github.com/jbosstools/jbosstools-build-ci/blob/master/publish/publish.sh#L541 
>>>> 
>>>> 
>>>> [1]
>>>> http://download.jboss.org/jbosstools/builds/staging/_composite_/core/trunk/compositeArtifacts.xml 
>>>> 
>>>> 
>>>> This is done because:
>>>> 
>>>> * after the publish of the N build is done, before the bits are
>>>> visible on download.jboss.org, the lag seems to vary from seconds to
>>>> minutes.
>>>> 
>>>> * a build-in-progress which resolved against the N-1 bits will fail if
>>>> those bits suddenly vanish.
>>>> 
>>>> ---
>>>> 
>>>> For the stable branch, the same push-then-rename approach [0] is used
>>>> (rather than destructively pushing on top of an existing build, as we
>>>> did a few years ago), but we only composite [2] the latest (N) because:
>>>> 
>>>> * stable branch builds change less often and
>>>> * we want to guarantee that we're using the latest.
>>>> 
>>>> [2]
>>>> http://download.jboss.org/jbosstools/builds/staging/_composite_/core/4.1.kepler/compositeArtifacts.xml 
>>>> 
>>>> 
>>>> Hope that makes sense. If you think something can be improved w/o
>>>> causing new problems (such as updating the timestamp in the
>>>> composite*.xml files dynamically w/ every published build, so p2 sees
>>>> it as "new" more often), please don't hesitate to open a JIRA w/
>>>> details on what you'd change, and how.
>>>> 
>>>> Cheers,
>>>> 
>>>> Nick
>>>> 
>>>>> On 05/21/2013 02:12 PM, Denis Golovin wrote:
>>>>>> On 05/21/2013 10:32 AM, Mickael Istria wrote:
>>>>>>> On 05/21/2013 06:56 PM, Denis Golovin wrote:
>>>>>>> The problem is org.jboss.tools.tests is not part of
>>>>>>> 
>>>>>>> http://download.jboss.org/jbosstools/updates/nightly/core/trunk/plugins/
>>>>>> That's right. However it's part of
>>>>>> http://download.jboss.org/jbosstools/updates/nightly/integrationtests/trunk 
>>>>>> 
>>>>>> 
>>>>>> site, so when enabling no profile, the default nightly sites (core 
>>>>>> and
>>>>>> integration tests) are used, so it should resolve this bundle.
>>>>>> Is this something you can reproduce at every build? It could happen
>>>>>> when your build tries to get content at the same time aggregation is
>>>>>> getting published.
>>>>> 
>>>>> Pushing process can be implemented different way to avoid this 
>>>>> issue, I
>>>>> saw it many times and it is a bit annoying. I remember we spend some
>>>>> time with Nick to tackle it, but problem seems still here.
>>>>> 
>>>>> Idea behind this fix is really simple:
>>>>> 1. Leave old published bits as is on the server side and just 
>>>>> upload new
>>>>> ones
>>>>> 2. Update compositeArtifacts.xml first and include uploaded update 
>>>>> site
>>>>> from step 1
>>>>> 3. Update compositeContent.xml: include new uploaded update site from
>>>>> step 1 and remove oldest one
>>>>> 4. Update compositeArtifacts.xml and remove oldest one
>>>>> 5. Remove oldest update site folder
>>>>> 
>>>>> Note there are no operations related to renaming/moving previously
>>>>> uploaded update sites and that the key point to have previously 
>>>>> uploaded
>>>>> sites available while new one is in uploading stage.
>>>>> 
>>>>> It should significantly reduce amount of errors because we keep two
>>>>> update sites for each jbosstools-module, so at least one update 
>>>>> site is
>>>>> always available through composite update site for module. There could
>>>>> be still problems for builds with slow connection, but connection 
>>>>> should
>>>>> be slow enough to live through at least two builds for jbosstools
>>>>> module. This could be implemented as maven plug-n and number of builds
>>>>> to keep in composite could be a good candidate for configuration
>>>>> parameters.
>>>>> 
>>>>> It is not really critical but nice to have.
>>>>> 
>>>>> Denis
>>>>> 
>>>>>> -- 
>>>>>> Mickael Istria
>>>>>> Eclipse developer at JBoss, by Red Hat <http://www.jboss.org/tools>
>>>>>> My blog <http://mickaelistria.wordpress.com> - My Tweets
>>>>>> <http://twitter.com/mickaelistria>
>>>>>> 
>>>>>> 
>>>>>> _______________________________________________
>>>>>> jbosstools-dev mailing list
>>>>>> jbosstools-dev at lists.jboss.org
>>>>>> https://lists.jboss.org/mailman/listinfo/jbosstools-dev
>>>>> 
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> jbosstools-dev mailing list
>>>>> jbosstools-dev at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/jbosstools-dev
> 
> _______________________________________________
> jbosstools-dev mailing list
> jbosstools-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/jbosstools-dev