[jbosstools-issues] [JBoss JIRA] (JBDS-2254) clean out redundant test runtime caches on in static_build_env

Fri Aug 24 13:01:15 EDT 2012

     [ https://issues.jboss.org/browse/JBDS-2254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nick Boldt updated JBDS-2254:
-----------------------------

    Description: 
Noticed today that we have a lot of duplicate copies of the same runtimes. For example:

{code}
/home/hudson/static_build_env/jbds/download-cache/soa-p-5.3.0.ER2.zip
/home/hudson/static_build_env/jbds/requirements/download/soa-p-5.3.0.ER2/soa-p-5.3.0.ER2.zip
/home/hudson/static_build_env/jbds/requirements/download/soap/soa-p-5.3.0.ER2.zip
{code}

Doing a full search for dupes from the download-cache folder vs. the requirements/download/ folder, I see:

{code}
$ cd /home/hudson/static_build_env/jbds/download-cache; for d in 
$(find . -type f -name "*.zip" | sort); do du -sch 
{/home/hudson/static_build_env/jbds/download-cache/,/home/hudson/static_build_env/jbds/requirements/download/*/}${d};
done

178M    /home/hudson/static_build_env/jbds/download-cache/./GateIn-3.1.0-FINAL-jbossas.zip
178M    /home/hudson/static_build_env/jbds/requirements/download/gatein-3.1.0.Final/./GateIn-3.1.0-FINAL-jbossas.zip
355M    total

8.0K    /home/hudson/static_build_env/jbds/download-cache/./jbds-seamfp-patch.zip
du: cannot access `/home/hudson/static_build_env/jbds/requirements/download/*/./jbds-seamfp-patch.zip': No such file or directory
8.0K    total

54M     /home/hudson/static_build_env/jbds/download-cache/./jboss-3.2.8.SP1.zip
54M     /home/hudson/static_build_env/jbds/requirements/download/jbossas/./jboss-3.2.8.SP1.zip
107M    total

78M     /home/hudson/static_build_env/jbds/download-cache/./jboss-4.0.5.GA.zip
78M     /home/hudson/static_build_env/jbds/requirements/download/jbossas/./jboss-4.0.5.GA.zip
155M    total

105M    /home/hudson/static_build_env/jbds/download-cache/./jboss-5.0.1.GA.zip
105M    /home/hudson/static_build_env/jbds/requirements/download/jbossas/./jboss-5.0.1.GA.zip
209M    total

128M    /home/hudson/static_build_env/jbds/download-cache/./jboss-5.1.0.GA.zip
128M    /home/hudson/static_build_env/jbds/requirements/download/jbossas-5.1.0.GA/./jboss-5.1.0.GA.zip
128M    /home/hudson/static_build_env/jbds/requirements/download/jbossas/./jboss-5.1.0.GA.zip
383M    total

270M    /home/hudson/static_build_env/jbds/download-cache/./jboss-eap-4.3.0.GA_CP10.zip
du: cannot access `/home/hudson/static_build_env/jbds/requirements/download/*/./jboss-eap-4.3.0.GA_CP10.zip': No such file or directory
270M    total

293M    /home/hudson/static_build_env/jbds/download-cache/./jboss-eap-5.1.2.zip
du: cannot access `/home/hudson/static_build_env/jbds/requirements/download/*/./jboss-eap-5.1.2.zip': No such file or directory
293M    total

110M    /home/hudson/static_build_env/jbds/download-cache/./jboss-eap-6.0.0.zip
110M    /home/hudson/static_build_env/jbds/requirements/download/jbeap-6/./jboss-eap-6.0.0.zip
220M    total

78M     /home/hudson/static_build_env/jbds/download-cache/./jboss-eap-fp-4.3.0.CP03-FP01.zip
78M     /home/hudson/static_build_env/jbds/requirements/download/seamfp/./jboss-eap-fp-4.3.0.CP03-FP01.zip
155M    total

193M    /home/hudson/static_build_env/jbds/download-cache/./jboss-eap-noauth-4.3.0.GA_CP03.zip
193M    /home/hudson/static_build_env/jbds/requirements/download/jbeap/./jboss-eap-noauth-4.3.0.GA_CP03.zip
385M    total

293M    /home/hudson/static_build_env/jbds/download-cache/./jboss-eap-noauth-5.1.2.zip
du: cannot access `/home/hudson/static_build_env/jbds/requirements/download/*/./jboss-eap-noauth-5.1.2.zip': No such file or directory
293M    total

189M    /home/hudson/static_build_env/jbds/download-cache/./jbossesb-4.11.zip
189M    /home/hudson/static_build_env/jbds/requirements/download/jbossesb-4.11/./jbossesb-4.11.zip
377M    total

157M    /home/hudson/static_build_env/jbds/download-cache/./jboss-portal-2.7.2-bundled.zip
157M    /home/hudson/static_build_env/jbds/requirements/download/JBoss-Portal-2.7.2.GA/./jboss-portal-2.7.2-bundled.zip
314M    total

26M     /home/hudson/static_build_env/jbds/download-cache/./JBossPortletBridge-2.1.0.FINAL.zip
26M     /home/hudson/static_build_env/jbds/requirements/download/JBossPortletBridge-2.1.0.FINAL/./JBossPortletBridge-2.1.0.FINAL.zip
51M     total

93M     /home/hudson/static_build_env/jbds/download-cache/./jboss-seam-2.0.1.GA.zip
93M     /home/hudson/static_build_env/jbds/requirements/download/seam-2.0.1.GA/./jboss-seam-2.0.1.GA.zip
186M    total

89M     /home/hudson/static_build_env/jbds/download-cache/./jboss-seam-2.0.2.SP1.zip
89M     /home/hudson/static_build_env/jbds/requirements/download/seam-2.0.2.SP1/./jboss-seam-2.0.2.SP1.zip
177M    total

89M     /home/hudson/static_build_env/jbds/download-cache/./jboss-seam-2.0.3.CR1.zip
89M     /home/hudson/static_build_env/jbds/requirements/download/seam-2.0.3.CR1/./jboss-seam-2.0.3.CR1.zip
178M    total

107M    /home/hudson/static_build_env/jbds/download-cache/./jboss-seam-2.1.1.GA.zip
107M    /home/hudson/static_build_env/jbds/requirements/download/seam-2.1.1.GA/./jboss-seam-2.1.1.GA.zip
213M    total

137M    /home/hudson/static_build_env/jbds/download-cache/./jboss-seam-2.1.2.zip
137M    /home/hudson/static_build_env/jbds/requirements/download/seam-2.1.2.GA/./jboss-seam-2.1.2.zip
273M    total

141M    /home/hudson/static_build_env/jbds/download-cache/./jboss-seam-2.2.0.CR1.zip
141M    /home/hudson/static_build_env/jbds/requirements/download/seam-2.2.0.CR1/./jboss-seam-2.2.0.CR1.zip
281M    total

113M    /home/hudson/static_build_env/jbds/download-cache/./jboss-seam-2.2.0.GA.zip
113M    /home/hudson/static_build_env/jbds/requirements/download/seam-2.2.0.GA/./jboss-seam-2.2.0.GA.zip
225M    total

108M    /home/hudson/static_build_env/jbds/download-cache/./jboss-seam-2.2.1.Final.zip
108M    /home/hudson/static_build_env/jbds/requirements/download/seam-2.2.1.Final/./jboss-seam-2.2.1.Final.zip
216M    total

110M    /home/hudson/static_build_env/jbds/download-cache/./jboss-seam-2.3.0.ALPHA-distribution.zip
110M    /home/hudson/static_build_env/jbds/requirements/download/seam-2.3.0/./jboss-seam-2.3.0.ALPHA-distribution.zip
219M    total

24M     /home/hudson/static_build_env/jbds/download-cache/./jbpm-3.1.4.zip
24M     /home/hudson/static_build_env/jbds/requirements/download/jbpm-3.1.4/./jbpm-3.1.4.zip
47M     total

8.2M    /home/hudson/static_build_env/jbds/download-cache/./rhds-seam-gen-patch.zip
8.2M    /home/hudson/static_build_env/jbds/requirements/download/eapseamgen/./rhds-seam-gen-patch.zip
17M     total

60M     /home/hudson/static_build_env/jbds/download-cache/./riftsaw-2.3.0.Final.zip
60M     /home/hudson/static_build_env/jbds/requirements/download/riftsaw-2.3.0.Final/./riftsaw-2.3.0.Final.zip
120M    total

569M    /home/hudson/static_build_env/jbds/download-cache/./soa-4.3.0.GA_CP05.zip
569M    /home/hudson/static_build_env/jbds/requirements/download/soa-4.3.0.GA_CP05/./soa-4.3.0.GA_CP05.zip
569M    /home/hudson/static_build_env/jbds/requirements/download/soap/./soa-4.3.0.GA_CP05.zip
1.7G    total

593M    /home/hudson/static_build_env/jbds/download-cache/./soa-5.0.1.GA-signed.zip
593M    /home/hudson/static_build_env/jbds/requirements/download/soa-5.0.1.GA-signed/./soa-5.0.1.GA-signed.zip
593M    /home/hudson/static_build_env/jbds/requirements/download/soap/./soa-5.0.1.GA-signed.zip
1.8G    total

635M    /home/hudson/static_build_env/jbds/download-cache/./soa-p-5.3.0.ER2.zip
635M    /home/hudson/static_build_env/jbds/requirements/download/soa-p-5.3.0.ER2/./soa-p-5.3.0.ER2.zip
635M    /home/hudson/static_build_env/jbds/requirements/download/soap/./soa-p-5.3.0.ER2.zip
1.9G    total

629M    /home/hudson/static_build_env/jbds/download-cache/./soa-p-5.3.0.ER5.zip
du: cannot access `/home/hudson/static_build_env/jbds/requirements/download/*/./soa-p-5.3.0.ER5.zip': No such file or directory
629M    total

{code}

Full disk footprint between the two locations:

{code}
$ du -sch /home/hudson/static_build_env/jbds/download-cache /home/hudson/static_build_env/jbds/requirements/download/

5.6G    /home/hudson/static_build_env/jbds/download-cache
13G     /home/hudson/static_build_env/jbds/requirements/download/
18G     total
{code}

So... seems like there are two problems:

a) we have a lot of old/obsolete reqs which can be purged
b) we fetch into two places - I suspect one path (download-cache?) is the new approach mistria wrote for JBDS 6, and the old path is the old Ant-based fetch approach (requirements/download/).

Spoke with Len and he said we were good to purge these, but I wanted to double-check w/ mistria and dgolovin that wiping out all 18G and letting the builds re-fetch what they need is an acceptable way to clean up this area. 

Also, can we fix JBDS 5 (and 4?) so it uses the new maven-based fetch instead of the old Ant one, in order to avoid the duplicates and disk usage?

  was:
Noticed today that we have a lot of duplicate copies of the same runtimes. For example:

{code}
    /home/hudson/static_build_env/jbds/download-cache/soa-p-5.3.0.ER2.zip
    /home/hudson/static_build_env/jbds/requirements/download/soa-p-5.3.0.ER2/soa-p-5.3.0.ER2.zip
    /home/hudson/static_build_env/jbds/requirements/download/soap/soa-p-5.3.0.ER2.zip
{code}