[jbosstools-dev] Tests error during local build

Nick Boldt nboldt at redhat.com
Mon Aug 19 22:25:50 EDT 2013 

I've opened this JIRA:

https://issues.jboss.org/browse/JBIDE-15356



On 08/16/2013 11:34 AM, Nick Boldt wrote:
> This sounds like something we should get done during the current "down
> time" before we're in the middle of a new Alpha1 code freeze cycle.
>
> Denis, is there an existing JIRA for this, or would you like me to open
> one?
>
> N
>
> On 08/14/2013 12:48 AM, Max Andersen wrote:
>> Just to say I agree with Denis overall idea and on nicks specifics.
>> This approach about updating the ci-builds composite to keep ongoing
>> builds running is what we discussed at the f2f.
>>
>> /max (sent from my phone)
>> N
>>
>>> On 14/08/2013, at 02.32, Denis Golovin <dgolovin at exadel.com> wrote:
>>>
>>> On 08/13/2013 12:24 PM, Nick Boldt wrote:
>>>>>> The stated workflow is already being done [0], except that today:
>>>>>> * the current (N) and previous (N-1) are composited together into a
>>>>>> single site [1]
>>>>>> * the project's previous build is NOT then removed from the composite
>>>>>> & deleted once the new build is done publishing
>>>>> What I am trying to tell for a while is having names
>>>>> staging/staging.previous in update site and physically move content of
>>>>> staging to staging.previous after every build is not working. In fact
>>>>> with this approach local/jenkins build failures are unpredictable,
>>>>> when
>>>>> dev/jenkins runs a long build lets say javaee and suddenly on remote
>>>>> server publis.sh L585 moves bits like that
>>>>>   mv \
>>>>> $DESTINATION/builds/staging/${JOB_NAME} \
>>>>> $DESTINATION/builds/staging.previous/${JOB_NAME}
>>>>>
>>>>> build fails.
>>>>
>>>> What it actually does is:
>>>>
>>>> 1. rsync new build into staging/${JOB_NAME}.next (slow)
>>>> 2. delete staging.previous/${JOB_NAME} (slow)
>>>> 3. move staging/${JOB_NAME} to staging.previous/${JOB_NAME} (fast)
>>>> 4. move staging/${JOB_NAME}.next to staging/${JOB_NAME} (fast)
>>>>
>>>> Therefore the impact on the composite site is only a few seconds'
>>>> outage, unless your build was depending on content in the
>>>> *staging.previous site*, in which case, yes, you will likely end up
>>>> broken.
>>>>
>>>> This mechanism allows builds in progress, which depend on the CURRENT
>>>> Nth build to continue to work even after the new N build replaces the
>>>> previous one.
>>> I am not following you here. While builds in progress is not affected
>>> here? If build downloaded metadata from composite update site and picked
>>> up latest one (which is staging/${JOB_NAME}) and started downloading
>>> artifacts it would fail in case of (3).
>>>>
>>>> For example, if you kick a mvn build, which pings the composite site
>>>> to resolve dependencies and sees that staging = B123 and
>>>> staging.previous = B122, your build begins fine. But if while you're
>>>> building/resolving a new B124 gets published, you'll STILL be fine to
>>>> depend on B123 bits, just not the now-deleted B122 bits. Restarting
>>>> your job will always cause the dep resolution to restart, whereupon
>>>> the metadata will be scanned again and you'll then be building against
>>>> staging = B124 and staging.previous = B123.
>>>>
>>>>> Considering it takes couple hours to execute and you have
>>>>> to do that again and sometimes again until it works, it is not really
>>>>> convenient.
>>>>
>>>> So are you proposing that instead of an in-place move which reuses
>>>> generic folder names like "staging" and "staging.previous", we
>>>> composite build output using unique names like
>>>> 2013-08-09_05-05-26-B7222/ or 2013-08-13_10-05-28-B7255?
>>> yes
>>>>
>>>> If so, we would need:
>>>>
>>>> a) to regenerate the composite site each time there's a new build
>>>> published, in order to remove the oldest and add the newest (keeping
>>>> only the Nth and N-1rst builds)
>>>>
>>>> (I have a script that might already work for this, or would need
>>>> tweaking.)
>>> yes, would be good to be able pass number builds to keep as a parameter.
>>>>
>>>> b) heuristics to determine when an older (N-2, N-3, ... N-z) build is
>>>> no longer needed, perhaps simply by assuming no one needs it after
>>>> 24hrs?
>>>>
>>>> (Once we agree, this is trivial.)
>>> 24 ours should be more that enough.
>>>>
>>>> c) a cleanup script which can purge all but the builds which are no
>>>> more than 1 day old, keeping at all times at least two builds (N and
>>>> N-1)
>>>>
>>>> (I have a script that already does this for folders like
>>>> http://download.jboss.org/jbosstools/builds/nightly/core/trunk/ but
>>>> might need to be tweaked to work for a new pattern of
>>>> staging/${JOB_NAME}/<BUILD_ID>/ .)
>>>
>>> yes
>>>
>>>>
>>>>
>>>>>> * the composite*.xml metadata files are not regenerated when
>>>>>> publishing, but are *manually* updated if/when a new component is
>>>>>> added or job is renamed
>>>>> well, they should be updated because:
>>>>> 1. It would let to avoid moving huge p2repos around and wait while
>>>>> they
>>>>> are in sync with download.jboss.org
>>>>> 2. It would take much less time for *.xml files to be available on
>>>>> download.jboss.org
>>>>
>>>> Actually, the time to publish the bits from Jenkins to
>>>> download.jboss.org would not change. We would still need to push bits
>>>> to the server, and perform cleanup of old builds. Generating the
>>>> composite*.xml files would add some (negligible) time to the process,
>>>> but would guarantee that a p2 or tycho user would always get a fresh
>>>> timestamp.
>>>>
>>>> Currently, as the composite*.xml files do not change regularly, there
>>>> is NO time required to make them available. So, this would actually
>>>> introduce some delay, but as I said you'd get fresher files, so it's
>>>> perhaps an improvement.
>>>>
>>>>> bing! builds would stop failing so often.
>>>>
>>>> We can certainly try this and verify your hypothesis. :)
>>>>
>>>>> In fact for the future we can even save space by publishing only
>>>>> changed
>>>>> bits (need to fix qualifier for features/plugins) and it would let to:
>>>>> 1. Decrease time for syncing with download.jboss.org;
>>>>> 2. Have installation history actually working and let people use
>>>>> nightly
>>>>> updates and roll back to previous version if something is wrong;
>>>>> 3. Increase speed of local/jenkins builds, because no more full
>>>>> downloads for dependencies
>>>>
>>>> Publishing only changes to a repo would be nice except that it would
>>>> mean publishing *destructively on top of existing snapshots*, rather
>>>> than creating new ones. So anyone who was currently building against
>>>> the N build would probably end up with a broken build as new IUs were
>>>> pushed into that build's repo, and its metadata overwritten.
>>>
>>> publishing on top should not be required.
>>>
>>>>
>>>> That's definitely worse than what we have today. And would, instead of
>>>> having timestamped folders which are unique and don't overlap, put us
>>>> back where we are today with reusable, generic folder names like
>>>> "staging/${JOB_NAME}".
>>>>
>>>> So, I'm all for a new approach to compositing with regenerated
>>>> composite*.xml files, pointing to UNIQUELY VERSIONED folders (rather
>>>> than generic reusable ones), then doing cleanup of older N-2, N-3
>>>> builds, but I disagree that publishing changed bits into existing
>>>> repos is a good idea (even if you use rsync --delete to purge the
>>>> orphaned IUs, you'll still end up breaking more builds this way),
>>>> which is why we moved AWAY from this approach by implementing the
>>>> staging.next/staging/staging.previous shell game.
>>>
>>> This incremental update is tricky to implement and not a priority for us
>>> IMO.
>>>
>>> Denis
>>>
>>>>
>>>> N
>>>>
>>>>>>
>>>>>> [0]
>>>>>> https://github.com/jbosstools/jbosstools-build-ci/blob/master/publish/publish.sh#L541
>>>>>>
>>>>>>
>>>>>>
>>>>>> [1]
>>>>>> http://download.jboss.org/jbosstools/builds/staging/_composite_/core/trunk/compositeArtifacts.xml
>>>>>>
>>>>>>
>>>>>>
>>>>>> This is done because:
>>>>>>
>>>>>> * after the publish of the N build is done, before the bits are
>>>>>> visible on download.jboss.org, the lag seems to vary from seconds to
>>>>>> minutes.
>>>>>>
>>>>>> * a build-in-progress which resolved against the N-1 bits will
>>>>>> fail if
>>>>>> those bits suddenly vanish.
>>>>>>
>>>>>> ---
>>>>>>
>>>>>> For the stable branch, the same push-then-rename approach [0] is used
>>>>>> (rather than destructively pushing on top of an existing build, as we
>>>>>> did a few years ago), but we only composite [2] the latest (N)
>>>>>> because:
>>>>>>
>>>>>> * stable branch builds change less often and
>>>>>> * we want to guarantee that we're using the latest.
>>>>>>
>>>>>> [2]
>>>>>> http://download.jboss.org/jbosstools/builds/staging/_composite_/core/4.1.kepler/compositeArtifacts.xml
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hope that makes sense. If you think something can be improved w/o
>>>>>> causing new problems (such as updating the timestamp in the
>>>>>> composite*.xml files dynamically w/ every published build, so p2 sees
>>>>>> it as "new" more often), please don't hesitate to open a JIRA w/
>>>>>> details on what you'd change, and how.
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Nick
>>>>>>
>>>>>>> On 05/21/2013 02:12 PM, Denis Golovin wrote:
>>>>>>>> On 05/21/2013 10:32 AM, Mickael Istria wrote:
>>>>>>>>> On 05/21/2013 06:56 PM, Denis Golovin wrote:
>>>>>>>>> The problem is org.jboss.tools.tests is not part of
>>>>>>>>>
>>>>>>>>> http://download.jboss.org/jbosstools/updates/nightly/core/trunk/plugins/
>>>>>>>>>
>>>>>>>> That's right. However it's part of
>>>>>>>> http://download.jboss.org/jbosstools/updates/nightly/integrationtests/trunk
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> site, so when enabling no profile, the default nightly sites (core
>>>>>>>> and
>>>>>>>> integration tests) are used, so it should resolve this bundle.
>>>>>>>> Is this something you can reproduce at every build? It could happen
>>>>>>>> when your build tries to get content at the same time
>>>>>>>> aggregation is
>>>>>>>> getting published.
>>>>>>>
>>>>>>> Pushing process can be implemented different way to avoid this
>>>>>>> issue, I
>>>>>>> saw it many times and it is a bit annoying. I remember we spend some
>>>>>>> time with Nick to tackle it, but problem seems still here.
>>>>>>>
>>>>>>> Idea behind this fix is really simple:
>>>>>>> 1. Leave old published bits as is on the server side and just
>>>>>>> upload new
>>>>>>> ones
>>>>>>> 2. Update compositeArtifacts.xml first and include uploaded update
>>>>>>> site
>>>>>>> from step 1
>>>>>>> 3. Update compositeContent.xml: include new uploaded update site
>>>>>>> from
>>>>>>> step 1 and remove oldest one
>>>>>>> 4. Update compositeArtifacts.xml and remove oldest one
>>>>>>> 5. Remove oldest update site folder
>>>>>>>
>>>>>>> Note there are no operations related to renaming/moving previously
>>>>>>> uploaded update sites and that the key point to have previously
>>>>>>> uploaded
>>>>>>> sites available while new one is in uploading stage.
>>>>>>>
>>>>>>> It should significantly reduce amount of errors because we keep two
>>>>>>> update sites for each jbosstools-module, so at least one update
>>>>>>> site is
>>>>>>> always available through composite update site for module. There
>>>>>>> could
>>>>>>> be still problems for builds with slow connection, but connection
>>>>>>> should
>>>>>>> be slow enough to live through at least two builds for jbosstools
>>>>>>> module. This could be implemented as maven plug-n and number of
>>>>>>> builds
>>>>>>> to keep in composite could be a good candidate for configuration
>>>>>>> parameters.
>>>>>>>
>>>>>>> It is not really critical but nice to have.
>>>>>>>
>>>>>>> Denis
>>>>>>>
>>>>>>>> --
>>>>>>>> Mickael Istria
>>>>>>>> Eclipse developer at JBoss, by Red Hat <http://www.jboss.org/tools>
>>>>>>>> My blog <http://mickaelistria.wordpress.com> - My Tweets
>>>>>>>> <http://twitter.com/mickaelistria>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> jbosstools-dev mailing list
>>>>>>>> jbosstools-dev at lists.jboss.org
>>>>>>>> https://lists.jboss.org/mailman/listinfo/jbosstools-dev
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> jbosstools-dev mailing list
>>>>>>> jbosstools-dev at lists.jboss.org
>>>>>>> https://lists.jboss.org/mailman/listinfo/jbosstools-dev
>>>
>>> _______________________________________________
>>> jbosstools-dev mailing list
>>> jbosstools-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/jbosstools-dev
>

-- 
Nick Boldt :: JBoss by Red Hat
Productization Lead :: JBoss Tools & Dev Studio
http://nick.divbyzero.com

More information about the jbosstools-dev mailing list