4:18 p.m.
Nick Boldt created JBIDE-11155:
----------------------------------
Summary: consider caching svn sources outside Jenkins workspaces
Key: JBIDE-11155
URL: https://issues.jboss.org/browse/JBIDE-11155
Project: Tools (JBoss Tools)
Issue Type: Bug
Components: Build/Releng
Affects Versions: 3.3.0.Beta2
Reporter: Nick Boldt
Assignee: Nick Boldt
Fix For: 3.3.0.Beta2
From JBIDE-10974:
9. Every change in build component triggers full rebuild for all
components
9. This could be fixed by instead of watching the /build folder in SVN for changes, we
simply fetch it using `svn co` or `svn up` as a prelim step in the job.
Perhaps taking this a step further, we could fetch the whole SVN tree and store it outside
the workspaces so that jobs would simply re-use local shared space instead of
clean-fetching everything every time they run on a new slave.
This would of course make jobs "more tied to infrastructure" and "harder to
build the same way offline" but that's the price you pay for optimizing for
speed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:27 p.m.
[
https://issues.jboss.org/browse/JBIDE-11155?page=com.atlassian.jira.plugi...
]
Nick Boldt commented on JBIDE-11155:
------------------------------------
In target platform build, we could do this to cache the /build/ folder sources (one per
branch):
{code}
rsync -aqrz ${WORKSPACE}/sources/*
/home/hudson/static_build_env/jbds/tools/sources/build/trunk/
{code}
consider caching svn sources outside Jenkins workspaces
-------------------------------------------------------
Key: JBIDE-11155
URL: https://issues.jboss.org/browse/JBIDE-11155
Project: Tools (JBoss Tools)
Issue Type: Bug
Components: Build/Releng
Affects Versions: 3.3.0.Beta2
Reporter: Nick Boldt
Assignee: Nick Boldt
Fix For: 3.3.0.Beta2
From JBIDE-10974:
> 9. Every change in build component triggers full rebuild for all components
9. This could be fixed by instead of watching the /build folder in SVN for changes, we
simply fetch it using `svn co` or `svn up` as a prelim step in the job.
Perhaps taking this a step further, we could fetch the whole SVN tree and store it
outside the workspaces so that jobs would simply re-use local shared space instead of
clean-fetching everything every time they run on a new slave.
This would of course make jobs "more tied to infrastructure" and "harder
to build the same way offline" but that's the price you pay for optimizing for
speed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:24 a.m.
[
https://issues.jboss.org/browse/JBIDE-11155?page=com.atlassian.jira.plugi...
]
Nick Boldt commented on JBIDE-11155:
------------------------------------
http://hudson.qa.jboss.com/hudson/job/jbosstools-3.3_trunk.target-platfor... --
published build folder to the above folder. Now we can start rolling this out to the trunk
jobs.
consider caching svn sources outside Jenkins workspaces
-------------------------------------------------------
Key: JBIDE-11155
URL: https://issues.jboss.org/browse/JBIDE-11155
Project: Tools (JBoss Tools)
Issue Type: Bug
Components: Build/Releng
Affects Versions: 3.3.0.Beta2
Reporter: Nick Boldt
Assignee: Nick Boldt
Fix For: 3.3.0.Beta2
From JBIDE-10974:
> 9. Every change in build component triggers full rebuild for all components
9. This could be fixed by instead of watching the /build folder in SVN for changes, we
simply fetch it using `svn co` or `svn up` as a prelim step in the job.
Perhaps taking this a step further, we could fetch the whole SVN tree and store it
outside the workspaces so that jobs would simply re-use local shared space instead of
clean-fetching everything every time they run on a new slave.
This would of course make jobs "more tied to infrastructure" and "harder
to build the same way offline" but that's the price you pay for optimizing for
speed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:47 p.m.
[
https://issues.jboss.org/browse/JBIDE-11155?page=com.atlassian.jira.plugi...
]
Nick Boldt commented on JBIDE-11155:
------------------------------------
Now doing this to cache a copy of the ~/.m2 repo folder too. I figure a quick NFS copy
will trump fetching everything from maven central every time we build, right?
{code}
# store a copy of this project's sources so it can be usd to bootstrap other projects
mkdir -p /home/hudson/static_build_env/jbds/tools/sources/build/trunk/
rsync -aqrz ${WORKSPACE}/sources/*
/home/hudson/static_build_env/jbds/tools/sources/build/trunk/
# store a copy of this project's ~/.m2 folder so it can be used to bootstrap other
projects
mkdir -p /home/hudson/static_build_env/jbds/tools/sources/m2-repository/trunk/
rsync -aqrz ${WORKSPACE}/m2-repository/*
/home/hudson/static_build_env/jbds/tools/sources/m2-repository/trunk/
{code}
Awaiting respin:
http://hudson.qa.jboss.com/hudson/job/jbosstools-3.3_trunk.target-platform/ >=362
consider caching svn sources outside Jenkins workspaces
-------------------------------------------------------
Key: JBIDE-11155
URL: https://issues.jboss.org/browse/JBIDE-11155
Project: Tools (JBoss Tools)
Issue Type: Bug
Components: Build/Releng
Affects Versions: 3.3.0.Beta3
Reporter: Nick Boldt
Assignee: Nick Boldt
Fix For: 3.3.0.Beta3
From JBIDE-10974:
> 9. Every change in build component triggers full rebuild for all components
9. This could be fixed by instead of watching the /build folder in SVN for changes, we
simply fetch it using `svn co` or `svn up` as a prelim step in the job.
Perhaps taking this a step further, we could fetch the whole SVN tree and store it
outside the workspaces so that jobs would simply re-use local shared space instead of
clean-fetching everything every time they run on a new slave.
This would of course make jobs "more tied to infrastructure" and "harder
to build the same way offline" but that's the price you pay for optimizing for
speed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:31 p.m.
[
https://issues.jboss.org/browse/JBIDE-11155?page=com.atlassian.jira.plugi...
]
Max Rydahl Andersen commented on JBIDE-11155:
---------------------------------------------
how do we ensure that release builds are not using these cached .m2 and source ?
consider caching svn sources outside Jenkins workspaces
-------------------------------------------------------
Key: JBIDE-11155
URL: https://issues.jboss.org/browse/JBIDE-11155
Project: Tools (JBoss Tools)
Issue Type: Bug
Components: Build/Releng
Affects Versions: 3.3.0.Beta3
Reporter: Nick Boldt
Assignee: Nick Boldt
Fix For: 3.3.0.Beta3
From JBIDE-10974:
> 9. Every change in build component triggers full rebuild for all components
9. This could be fixed by instead of watching the /build folder in SVN for changes, we
simply fetch it using `svn co` or `svn up` as a prelim step in the job.
Perhaps taking this a step further, we could fetch the whole SVN tree and store it
outside the workspaces so that jobs would simply re-use local shared space instead of
clean-fetching everything every time they run on a new slave.
This would of course make jobs "more tied to infrastructure" and "harder
to build the same way offline" but that's the price you pay for optimizing for
speed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:13 a.m.
[
https://issues.jboss.org/browse/JBIDE-11155?page=com.atlassian.jira.plugi...
]
Nick Boldt commented on JBIDE-11155:
------------------------------------
How about a parameter of the job, set to RELEASE=true which can be checked before blindly
using the .m2 and ~/trunk/build folder caches? Then again, what's the problem w/
having ONE SVN fetch that feeds all the 40+ jobs in the stream, instead of many?
consider caching svn sources outside Jenkins workspaces
-------------------------------------------------------
Key: JBIDE-11155
URL: https://issues.jboss.org/browse/JBIDE-11155
Project: Tools (JBoss Tools)
Issue Type: Bug
Components: Build/Releng
Affects Versions: 3.3.0.Beta3
Reporter: Nick Boldt
Assignee: Nick Boldt
Fix For: 3.3.0.Beta3
From JBIDE-10974:
> 9. Every change in build component triggers full rebuild for all components
9. This could be fixed by instead of watching the /build folder in SVN for changes, we
simply fetch it using `svn co` or `svn up` as a prelim step in the job.
Perhaps taking this a step further, we could fetch the whole SVN tree and store it
outside the workspaces so that jobs would simply re-use local shared space instead of
clean-fetching everything every time they run on a new slave.
This would of course make jobs "more tied to infrastructure" and "harder
to build the same way offline" but that's the price you pay for optimizing for
speed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:49 a.m.
[
https://issues.jboss.org/browse/JBIDE-11155?page=com.atlassian.jira.plugi...
]
Mickael Istria commented on JBIDE-11155:
----------------------------------------
We should also consider "caching" mvn repo on a NFS folder as well.
consider caching svn sources outside Jenkins workspaces
-------------------------------------------------------
Key: JBIDE-11155
URL: https://issues.jboss.org/browse/JBIDE-11155
Project: Tools (JBoss Tools)
Issue Type: Bug
Components: Build/Releng
Affects Versions: 3.3.0.Beta3
Reporter: Nick Boldt
Assignee: Nick Boldt
Fix For: 3.3.0.Beta3
From JBIDE-10974:
> 9. Every change in build component triggers full rebuild for all components
9. This could be fixed by instead of watching the /build folder in SVN for changes, we
simply fetch it using `svn co` or `svn up` as a prelim step in the job.
Perhaps taking this a step further, we could fetch the whole SVN tree and store it
outside the workspaces so that jobs would simply re-use local shared space instead of
clean-fetching everything every time they run on a new slave.
This would of course make jobs "more tied to infrastructure" and "harder
to build the same way offline" but that's the price you pay for optimizing for
speed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:44 p.m.
[
https://issues.jboss.org/browse/JBIDE-11155?page=com.atlassian.jira.plugi...
]
Nick Boldt commented on JBIDE-11155:
------------------------------------
Problem w/ caching .m2 repo is that sometimes that gets corrupted. Reusing a corrupt repo
would cause subsequent builds to fail. Currently, if a build fails due to corrupt m2 repo,
chances are good the repin will be fine. If we start caching .m2 folders for each job, we
will also need to manually purge these folders if/when they get corrupted. Just something
to consider for downstream maintenance. :)
consider caching svn sources outside Jenkins workspaces
-------------------------------------------------------
Key: JBIDE-11155
URL: https://issues.jboss.org/browse/JBIDE-11155
Project: Tools (JBoss Tools)
Issue Type: Bug
Components: Build/Releng
Affects Versions: 3.3.0.Beta3
Reporter: Nick Boldt
Assignee: Nick Boldt
Fix For: 3.3.0.Beta3
From JBIDE-10974:
> 9. Every change in build component triggers full rebuild for all components
9. This could be fixed by instead of watching the /build folder in SVN for changes, we
simply fetch it using `svn co` or `svn up` as a prelim step in the job.
Perhaps taking this a step further, we could fetch the whole SVN tree and store it
outside the workspaces so that jobs would simply re-use local shared space instead of
clean-fetching everything every time they run on a new slave.
This would of course make jobs "more tied to infrastructure" and "harder
to build the same way offline" but that's the price you pay for optimizing for
speed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:48 p.m.
[
https://issues.jboss.org/browse/JBIDE-11155?page=com.atlassian.jira.plugi...
]
Mickael Istria commented on JBIDE-11155:
----------------------------------------
Agreed. It currently looks like caching will involve a lot of effort and introduce
pitfalls for low gain.
consider caching svn sources outside Jenkins workspaces
-------------------------------------------------------
Key: JBIDE-11155
URL: https://issues.jboss.org/browse/JBIDE-11155
Project: Tools (JBoss Tools)
Issue Type: Bug
Components: Build/Releng
Affects Versions: 3.3.0.Beta3
Reporter: Nick Boldt
Assignee: Nick Boldt
Fix For: 3.3.0.Beta3
From JBIDE-10974:
> 9. Every change in build component triggers full rebuild for all components
9. This could be fixed by instead of watching the /build folder in SVN for changes, we
simply fetch it using `svn co` or `svn up` as a prelim step in the job.
Perhaps taking this a step further, we could fetch the whole SVN tree and store it
outside the workspaces so that jobs would simply re-use local shared space instead of
clean-fetching everything every time they run on a new slave.
This would of course make jobs "more tied to infrastructure" and "harder
to build the same way offline" but that's the price you pay for optimizing for
speed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:10 p.m.
[
https://issues.jboss.org/browse/JBIDE-11155?page=com.atlassian.jira.plugi...
]
Nick Boldt commented on JBIDE-11155:
------------------------------------
The gain is faster CI builds... so maybe we try this in _trunk but not in _stable_branch?
Or just for aggregates and product jobs?
consider caching svn sources outside Jenkins workspaces
-------------------------------------------------------
Key: JBIDE-11155
URL: https://issues.jboss.org/browse/JBIDE-11155
Project: Tools (JBoss Tools)
Issue Type: Bug
Components: Build/Releng
Affects Versions: 3.3.0.Beta3
Reporter: Nick Boldt
Assignee: Nick Boldt
Fix For: 3.3.0.Beta3
From JBIDE-10974:
> 9. Every change in build component triggers full rebuild for all components
9. This could be fixed by instead of watching the /build folder in SVN for changes, we
simply fetch it using `svn co` or `svn up` as a prelim step in the job.
Perhaps taking this a step further, we could fetch the whole SVN tree and store it
outside the workspaces so that jobs would simply re-use local shared space instead of
clean-fetching everything every time they run on a new slave.
This would of course make jobs "more tied to infrastructure" and "harder
to build the same way offline" but that's the price you pay for optimizing for
speed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:52 a.m.
[
https://issues.jboss.org/browse/JBIDE-11155?page=com.atlassian.jira.plugi...
]
Nick Boldt updated JBIDE-11155:
-------------------------------
Fix Version/s: 3.3.0.CR1
(was: 3.3.0.Beta3)
consider caching svn sources outside Jenkins workspaces
-------------------------------------------------------
Key: JBIDE-11155
URL: https://issues.jboss.org/browse/JBIDE-11155
Project: Tools (JBoss Tools)
Issue Type: Bug
Components: Build/Releng
Affects Versions: 3.3.0.Beta3
Reporter: Nick Boldt
Assignee: Nick Boldt
Fix For: 3.3.0.CR1
From JBIDE-10974:
> 9. Every change in build component triggers full rebuild for all components
9. This could be fixed by instead of watching the /build folder in SVN for changes, we
simply fetch it using `svn co` or `svn up` as a prelim step in the job.
Perhaps taking this a step further, we could fetch the whole SVN tree and store it
outside the workspaces so that jobs would simply re-use local shared space instead of
clean-fetching everything every time they run on a new slave.
This would of course make jobs "more tied to infrastructure" and "harder
to build the same way offline" but that's the price you pay for optimizing for
speed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:40 p.m.
[
https://issues.jboss.org/browse/JBIDE-11155?page=com.atlassian.jira.plugi...
]
Max Rydahl Andersen updated JBIDE-11155:
----------------------------------------
Fix Version/s: 3.4.x
(was: 3.3.0.CR1)
consider caching svn sources outside Jenkins workspaces
-------------------------------------------------------
Key: JBIDE-11155
URL: https://issues.jboss.org/browse/JBIDE-11155
Project: Tools (JBoss Tools)
Issue Type: Bug
Components: Build/Releng
Affects Versions: 3.3.0.Beta3
Reporter: Nick Boldt
Assignee: Nick Boldt
Fix For: 3.4.x
From JBIDE-10974:
> 9. Every change in build component triggers full rebuild for all components
9. This could be fixed by instead of watching the /build folder in SVN for changes, we
simply fetch it using `svn co` or `svn up` as a prelim step in the job.
Perhaps taking this a step further, we could fetch the whole SVN tree and store it
outside the workspaces so that jobs would simply re-use local shared space instead of
clean-fetching everything every time they run on a new slave.
This would of course make jobs "more tied to infrastructure" and "harder
to build the same way offline" but that's the price you pay for optimizing for
speed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:04 p.m.
[
https://issues.jboss.org/browse/JBIDE-11155?page=com.atlassian.jira.plugi...
]
Nick Boldt closed JBIDE-11155.
------------------------------
Fix Version/s: 4.0.0.Alpha2
(was: 4.0.x)
Resolution: Rejected
Won't fix. we're moving to git and restructuring build/ folder so it's more
efficient anyway.
consider caching svn sources outside Jenkins workspaces
-------------------------------------------------------
Key: JBIDE-11155
URL: https://issues.jboss.org/browse/JBIDE-11155
Project: Tools (JBoss Tools)
Issue Type: Bug
Components: Build/Releng
Affects Versions: 3.3.0.Beta3
Reporter: Nick Boldt
Assignee: Nick Boldt
Fix For: 4.0.0.Alpha2
From JBIDE-10974:
> 9. Every change in build component triggers full rebuild for all components
9. This could be fixed by instead of watching the /build folder in SVN for changes, we
simply fetch it using `svn co` or `svn up` as a prelim step in the job.
Perhaps taking this a step further, we could fetch the whole SVN tree and store it
outside the workspaces so that jobs would simply re-use local shared space instead of
clean-fetching everything every time they run on a new slave.
This would of course make jobs "more tied to infrastructure" and "harder
to build the same way offline" but that's the price you pay for optimizing for
speed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
4364
days inactive
4577
days old
jbosstools-issues@lists.jboss.org
12 comments
3 participants
participants (3)
-
Max Rydahl Andersen (JIRA) -
Mickael Istria (JIRA)
-
Nick Boldt (JIRA)