[jbosstools-dev] Tests error during local build

Fri Aug 16 11:34:19 EDT 2013

This sounds like something we should get done during the current "down 
time" before we're in the middle of a new Alpha1 code freeze cycle.

Denis, is there an existing JIRA for this, or would you like me to open one?

N

On 08/14/2013 12:48 AM, Max Andersen wrote:
> Just to say I agree with Denis overall idea and on nicks specifics.
> This approach about updating the ci-builds composite to keep ongoing builds running is what we discussed at the f2f.
>
> /max (sent from my phone)
> N
>
>> On 14/08/2013, at 02.32, Denis Golovin <dgolovin at exadel.com> wrote:
>>
>> On 08/13/2013 12:24 PM, Nick Boldt wrote:
>>>>> The stated workflow is already being done [0], except that today:
>>>>> * the current (N) and previous (N-1) are composited together into a
>>>>> single site [1]
>>>>> * the project's previous build is NOT then removed from the composite
>>>>> & deleted once the new build is done publishing
>>>> What I am trying to tell for a while is having names
>>>> staging/staging.previous in update site and physically move content of
>>>> staging to staging.previous after every build is not working. In fact
>>>> with this approach local/jenkins build failures are unpredictable, when
>>>> dev/jenkins runs a long build lets say javaee and suddenly on remote
>>>> server publis.sh L585 moves bits like that
>>>>   mv \
>>>> $DESTINATION/builds/staging/${JOB_NAME} \
>>>> $DESTINATION/builds/staging.previous/${JOB_NAME}
>>>>
>>>> build fails.
>>>
>>> What it actually does is:
>>>
>>> 1. rsync new build into staging/${JOB_NAME}.next (slow)
>>> 2. delete staging.previous/${JOB_NAME} (slow)
>>> 3. move staging/${JOB_NAME} to staging.previous/${JOB_NAME} (fast)
>>> 4. move staging/${JOB_NAME}.next to staging/${JOB_NAME} (fast)
>>>
>>> Therefore the impact on the composite site is only a few seconds'
>>> outage, unless your build was depending on content in the
>>> *staging.previous site*, in which case, yes, you will likely end up
>>> broken.
>>>
>>> This mechanism allows builds in progress, which depend on the CURRENT
>>> Nth build to continue to work even after the new N build replaces the
>>> previous one.
>> I am not following you here. While builds in progress is not affected
>> here? If build downloaded metadata from composite update site and picked
>> up latest one (which is staging/${JOB_NAME}) and started downloading
>> artifacts it would fail in case of (3).
>>>
>>> For example, if you kick a mvn build, which pings the composite site
>>> to resolve dependencies and sees that staging = B123 and
>>> staging.previous = B122, your build begins fine. But if while you're
>>> building/resolving a new B124 gets published, you'll STILL be fine to
>>> depend on B123 bits, just not the now-deleted B122 bits. Restarting
>>> your job will always cause the dep resolution to restart, whereupon
>>> the metadata will be scanned again and you'll then be building against
>>> staging = B124 and staging.previous = B123.
>>>
>>>> Considering it takes couple hours to execute and you have
>>>> to do that again and sometimes again until it works, it is not really
>>>> convenient.
>>>
>>> So are you proposing that instead of an in-place move which reuses
>>> generic folder names like "staging" and "staging.previous", we
>>> composite build output using unique names like
>>> 2013-08-09_05-05-26-B7222/ or 2013-08-13_10-05-28-B7255?
>> yes
>>>
>>> If so, we would need:
>>>
>>> a) to regenerate the composite site each time there's a new build
>>> published, in order to remove the oldest and add the newest (keeping
>>> only the Nth and N-1rst builds)
>>>
>>> (I have a script that might already work for this, or would need
>>> tweaking.)
>> yes, would be good to be able pass number builds to keep as a parameter.
>>>
>>> b) heuristics to determine when an older (N-2, N-3, ... N-z) build is
>>> no longer needed, perhaps simply by assuming no one needs it after 24hrs?
>>>
>>> (Once we agree, this is trivial.)
>> 24 ours should be more that enough.
>>>
>>> c) a cleanup script which can purge all but the builds which are no
>>> more than 1 day old, keeping at all times at least two builds (N and N-1)
>>>
>>> (I have a script that already does this for folders like
>>> http://download.jboss.org/jbosstools/builds/nightly/core/trunk/ but
>>> might need to be tweaked to work for a new pattern of
>>> staging/${JOB_NAME}/<BUILD_ID>/ .)
>>
>> yes
>>
>>>
>>>
>>>>> * the composite*.xml metadata files are not regenerated when
>>>>> publishing, but are *manually* updated if/when a new component is
>>>>> added or job is renamed
>>>> well, they should be updated because:
>>>> 1. It would let to avoid moving huge p2repos around and wait while they
>>>> are in sync with download.jboss.org
>>>> 2. It would take much less time for *.xml files to be available on
>>>> download.jboss.org
>>>
>>> Actually, the time to publish the bits from Jenkins to
>>> download.jboss.org would not change. We would still need to push bits
>>> to the server, and perform cleanup of old builds. Generating the
>>> composite*.xml files would add some (negligible) time to the process,
>>> but would guarantee that a p2 or tycho user would always get a fresh
>>> timestamp.
>>>
>>> Currently, as the composite*.xml files do not change regularly, there
>>> is NO time required to make them available. So, this would actually
>>> introduce some delay, but as I said you'd get fresher files, so it's
>>> perhaps an improvement.
>>>
>>>> bing! builds would stop failing so often.
>>>
>>> We can certainly try this and verify your hypothesis. :)
>>>
>>>> In fact for the future we can even save space by publishing only changed
>>>> bits (need to fix qualifier for features/plugins) and it would let to:
>>>> 1. Decrease time for syncing with download.jboss.org;
>>>> 2. Have installation history actually working and let people use nightly
>>>> updates and roll back to previous version if something is wrong;
>>>> 3. Increase speed of local/jenkins builds, because no more full
>>>> downloads for dependencies
>>>
>>> Publishing only changes to a repo would be nice except that it would
>>> mean publishing *destructively on top of existing snapshots*, rather
>>> than creating new ones. So anyone who was currently building against
>>> the N build would probably end up with a broken build as new IUs were
>>> pushed into that build's repo, and its metadata overwritten.
>>
>> publishing on top should not be required.
>>
>>>
>>> That's definitely worse than what we have today. And would, instead of
>>> having timestamped folders which are unique and don't overlap, put us
>>> back where we are today with reusable, generic folder names like
>>> "staging/${JOB_NAME}".
>>>
>>> So, I'm all for a new approach to compositing with regenerated
>>> composite*.xml files, pointing to UNIQUELY VERSIONED folders (rather
>>> than generic reusable ones), then doing cleanup of older N-2, N-3
>>> builds, but I disagree that publishing changed bits into existing
>>> repos is a good idea (even if you use rsync --delete to purge the
>>> orphaned IUs, you'll still end up breaking more builds this way),
>>> which is why we moved AWAY from this approach by implementing the
>>> staging.next/staging/staging.previous shell game.
>>
>> This incremental update is tricky to implement and not a priority for us
>> IMO.
>>
>> Denis
>>
>>>
>>> N
>>>
>>>>>
>>>>> [0]
>>>>> https://github.com/jbosstools/jbosstools-build-ci/blob/master/publish/publish.sh#L541
>>>>>
>>>>>
>>>>> [1]
>>>>> http://download.jboss.org/jbosstools/builds/staging/_composite_/core/trunk/compositeArtifacts.xml
>>>>>
>>>>>
>>>>> This is done because:
>>>>>
>>>>> * after the publish of the N build is done, before the bits are
>>>>> visible on download.jboss.org, the lag seems to vary from seconds to
>>>>> minutes.
>>>>>
>>>>> * a build-in-progress which resolved against the N-1 bits will fail if
>>>>> those bits suddenly vanish.
>>>>>
>>>>> ---
>>>>>
>>>>> For the stable branch, the same push-then-rename approach [0] is used
>>>>> (rather than destructively pushing on top of an existing build, as we
>>>>> did a few years ago), but we only composite [2] the latest (N) because:
>>>>>
>>>>> * stable branch builds change less often and
>>>>> * we want to guarantee that we're using the latest.
>>>>>
>>>>> [2]
>>>>> http://download.jboss.org/jbosstools/builds/staging/_composite_/core/4.1.kepler/compositeArtifacts.xml
>>>>>
>>>>>
>>>>> Hope that makes sense. If you think something can be improved w/o
>>>>> causing new problems (such as updating the timestamp in the
>>>>> composite*.xml files dynamically w/ every published build, so p2 sees
>>>>> it as "new" more often), please don't hesitate to open a JIRA w/
>>>>> details on what you'd change, and how.
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Nick
>>>>>
>>>>>> On 05/21/2013 02:12 PM, Denis Golovin wrote:
>>>>>>> On 05/21/2013 10:32 AM, Mickael Istria wrote:
>>>>>>>> On 05/21/2013 06:56 PM, Denis Golovin wrote:
>>>>>>>> The problem is org.jboss.tools.tests is not part of
>>>>>>>>
>>>>>>>> http://download.jboss.org/jbosstools/updates/nightly/core/trunk/plugins/
>>>>>>> That's right. However it's part of
>>>>>>> http://download.jboss.org/jbosstools/updates/nightly/integrationtests/trunk
>>>>>>>
>>>>>>>
>>>>>>> site, so when enabling no profile, the default nightly sites (core
>>>>>>> and
>>>>>>> integration tests) are used, so it should resolve this bundle.
>>>>>>> Is this something you can reproduce at every build? It could happen
>>>>>>> when your build tries to get content at the same time aggregation is
>>>>>>> getting published.
>>>>>>
>>>>>> Pushing process can be implemented different way to avoid this
>>>>>> issue, I
>>>>>> saw it many times and it is a bit annoying. I remember we spend some
>>>>>> time with Nick to tackle it, but problem seems still here.
>>>>>>
>>>>>> Idea behind this fix is really simple:
>>>>>> 1. Leave old published bits as is on the server side and just
>>>>>> upload new
>>>>>> ones
>>>>>> 2. Update compositeArtifacts.xml first and include uploaded update
>>>>>> site
>>>>>> from step 1
>>>>>> 3. Update compositeContent.xml: include new uploaded update site from
>>>>>> step 1 and remove oldest one
>>>>>> 4. Update compositeArtifacts.xml and remove oldest one
>>>>>> 5. Remove oldest update site folder
>>>>>>
>>>>>> Note there are no operations related to renaming/moving previously
>>>>>> uploaded update sites and that the key point to have previously
>>>>>> uploaded
>>>>>> sites available while new one is in uploading stage.
>>>>>>
>>>>>> It should significantly reduce amount of errors because we keep two
>>>>>> update sites for each jbosstools-module, so at least one update
>>>>>> site is
>>>>>> always available through composite update site for module. There could
>>>>>> be still problems for builds with slow connection, but connection
>>>>>> should
>>>>>> be slow enough to live through at least two builds for jbosstools
>>>>>> module. This could be implemented as maven plug-n and number of builds
>>>>>> to keep in composite could be a good candidate for configuration
>>>>>> parameters.
>>>>>>
>>>>>> It is not really critical but nice to have.
>>>>>>
>>>>>> Denis
>>>>>>
>>>>>>> --
>>>>>>> Mickael Istria
>>>>>>> Eclipse developer at JBoss, by Red Hat <http://www.jboss.org/tools>
>>>>>>> My blog <http://mickaelistria.wordpress.com> - My Tweets
>>>>>>> <http://twitter.com/mickaelistria>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> jbosstools-dev mailing list
>>>>>>> jbosstools-dev at lists.jboss.org
>>>>>>> https://lists.jboss.org/mailman/listinfo/jbosstools-dev
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> jbosstools-dev mailing list
>>>>>> jbosstools-dev at lists.jboss.org
>>>>>> https://lists.jboss.org/mailman/listinfo/jbosstools-dev
>>
>> _______________________________________________
>> jbosstools-dev mailing list
>> jbosstools-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/jbosstools-dev

-- 
Nick Boldt :: JBoss by Red Hat
Productization Lead :: JBoss Tools & Dev Studio
http://nick.divbyzero.com