[jbosstools-issues] [JBoss JIRA] (JBDS-2254) clean out redundant test runtime caches on in static_build_env

[ https://issues.jboss.org/browse/JBDS-2254?page=com.atlassian.jira.plugin.... ] Nick Boldt updated JBDS-2254: ----------------------------- Description: Noticed today that we have a lot of duplicate copies of the same runtimes. For example: {code} /home/hudson/static_build_env/jbds/download-cache/soa-p-5.3.0.ER2.zip /home/hudson/static_build_env/jbds/requirements/download/soa-p-5.3.0.ER2/soa-p-5.3.0.ER2.zip /home/hudson/static_build_env/jbds/requirements/download/soap/soa-p-5.3.0.ER2.zip {code} Doing a full search for dupes from the download-cache folder vs. the requirements/download/ folder, I see: {code} $ cd /home/hudson/static_build_env/jbds/download-cache; for d in $(find . -type f -name "*.zip" | sort); do du -sch {/home/hudson/static_build_env/jbds/download-cache/,/home/hudson/static_build_env/jbds/requirements/download/*/}${d}; done 178M /home/hudson/static_build_env/jbds/download-cache/./GateIn-3.1.0-FINAL-jbossas.zip 178M /home/hudson/static_build_env/jbds/requirements/download/gatein-3.1.0.Final/./GateIn-3.1.0-FINAL-jbossas.zip 355M total 8.0K /home/hudson/static_build_env/jbds/download-cache/./jbds-seamfp-patch.zip du: cannot access `/home/hudson/static_build_env/jbds/requirements/download/*/./jbds-seamfp-patch.zip': No such file or directory 8.0K total 54M /home/hudson/static_build_env/jbds/download-cache/./jboss-3.2.8.SP1.zip 54M /home/hudson/static_build_env/jbds/requirements/download/jbossas/./jboss-3.2.8.SP1.zip 107M total 78M /home/hudson/static_build_env/jbds/download-cache/./jboss-4.0.5.GA.zip 78M /home/hudson/static_build_env/jbds/requirements/download/jbossas/./jboss-4.0.5.GA.zip 155M total 105M /home/hudson/static_build_env/jbds/download-cache/./jboss-5.0.1.GA.zip 105M /home/hudson/static_build_env/jbds/requirements/download/jbossas/./jboss-5.0.1.GA.zip 209M total 128M /home/hudson/static_build_env/jbds/download-cache/./jboss-5.1.0.GA.zip 128M /home/hudson/static_build_env/jbds/requirements/download/jbossas-5.1.0.GA/./jboss-5.1.0.GA.zip 128M /home/hudson/static_build_env/jbds/requirements/download/jbossas/./jboss-5.1.0.GA.zip 383M total 270M /home/hudson/static_build_env/jbds/download-cache/./jboss-eap-4.3.0.GA_CP10.zip du: cannot access `/home/hudson/static_build_env/jbds/requirements/download/*/./jboss-eap-4.3.0.GA_CP10.zip': No such file or directory 270M total 293M /home/hudson/static_build_env/jbds/download-cache/./jboss-eap-5.1.2.zip du: cannot access `/home/hudson/static_build_env/jbds/requirements/download/*/./jboss-eap-5.1.2.zip': No such file or directory 293M total 110M /home/hudson/static_build_env/jbds/download-cache/./jboss-eap-6.0.0.zip 110M /home/hudson/static_build_env/jbds/requirements/download/jbeap-6/./jboss-eap-6.0.0.zip 220M total 78M /home/hudson/static_build_env/jbds/download-cache/./jboss-eap-fp-4.3.0.CP03-FP01.zip 78M /home/hudson/static_build_env/jbds/requirements/download/seamfp/./jboss-eap-fp-4.3.0.CP03-FP01.zip 155M total 193M /home/hudson/static_build_env/jbds/download-cache/./jboss-eap-noauth-4.3.0.GA_CP03.zip 193M /home/hudson/static_build_env/jbds/requirements/download/jbeap/./jboss-eap-noauth-4.3.0.GA_CP03.zip 385M total 293M /home/hudson/static_build_env/jbds/download-cache/./jboss-eap-noauth-5.1.2.zip du: cannot access `/home/hudson/static_build_env/jbds/requirements/download/*/./jboss-eap-noauth-5.1.2.zip': No such file or directory 293M total 189M /home/hudson/static_build_env/jbds/download-cache/./jbossesb-4.11.zip 189M /home/hudson/static_build_env/jbds/requirements/download/jbossesb-4.11/./jbossesb-4.11.zip 377M total 157M /home/hudson/static_build_env/jbds/download-cache/./jboss-portal-2.7.2-bundled.zip 157M /home/hudson/static_build_env/jbds/requirements/download/JBoss-Portal-2.7.2.GA/./jboss-portal-2.7.2-bundled.zip 314M total 26M /home/hudson/static_build_env/jbds/download-cache/./JBossPortletBridge-2.1.0.FINAL.zip 26M /home/hudson/static_build_env/jbds/requirements/download/JBossPortletBridge-2.1.0.FINAL/./JBossPortletBridge-2.1.0.FINAL.zip 51M total 93M /home/hudson/static_build_env/jbds/download-cache/./jboss-seam-2.0.1.GA.zip 93M /home/hudson/static_build_env/jbds/requirements/download/seam-2.0.1.GA/./jboss-seam-2.0.1.GA.zip 186M total 89M /home/hudson/static_build_env/jbds/download-cache/./jboss-seam-2.0.2.SP1.zip 89M /home/hudson/static_build_env/jbds/requirements/download/seam-2.0.2.SP1/./jboss-seam-2.0.2.SP1.zip 177M total 89M /home/hudson/static_build_env/jbds/download-cache/./jboss-seam-2.0.3.CR1.zip 89M /home/hudson/static_build_env/jbds/requirements/download/seam-2.0.3.CR1/./jboss-seam-2.0.3.CR1.zip 178M total 107M /home/hudson/static_build_env/jbds/download-cache/./jboss-seam-2.1.1.GA.zip 107M /home/hudson/static_build_env/jbds/requirements/download/seam-2.1.1.GA/./jboss-seam-2.1.1.GA.zip 213M total 137M /home/hudson/static_build_env/jbds/download-cache/./jboss-seam-2.1.2.zip 137M /home/hudson/static_build_env/jbds/requirements/download/seam-2.1.2.GA/./jboss-seam-2.1.2.zip 273M total 141M /home/hudson/static_build_env/jbds/download-cache/./jboss-seam-2.2.0.CR1.zip 141M /home/hudson/static_build_env/jbds/requirements/download/seam-2.2.0.CR1/./jboss-seam-2.2.0.CR1.zip 281M total 113M /home/hudson/static_build_env/jbds/download-cache/./jboss-seam-2.2.0.GA.zip 113M /home/hudson/static_build_env/jbds/requirements/download/seam-2.2.0.GA/./jboss-seam-2.2.0.GA.zip 225M total 108M /home/hudson/static_build_env/jbds/download-cache/./jboss-seam-2.2.1.Final.zip 108M /home/hudson/static_build_env/jbds/requirements/download/seam-2.2.1.Final/./jboss-seam-2.2.1.Final.zip 216M total 110M /home/hudson/static_build_env/jbds/download-cache/./jboss-seam-2.3.0.ALPHA-distribution.zip 110M /home/hudson/static_build_env/jbds/requirements/download/seam-2.3.0/./jboss-seam-2.3.0.ALPHA-distribution.zip 219M total 24M /home/hudson/static_build_env/jbds/download-cache/./jbpm-3.1.4.zip 24M /home/hudson/static_build_env/jbds/requirements/download/jbpm-3.1.4/./jbpm-3.1.4.zip 47M total 8.2M /home/hudson/static_build_env/jbds/download-cache/./rhds-seam-gen-patch.zip 8.2M /home/hudson/static_build_env/jbds/requirements/download/eapseamgen/./rhds-seam-gen-patch.zip 17M total 60M /home/hudson/static_build_env/jbds/download-cache/./riftsaw-2.3.0.Final.zip 60M /home/hudson/static_build_env/jbds/requirements/download/riftsaw-2.3.0.Final/./riftsaw-2.3.0.Final.zip 120M total 569M /home/hudson/static_build_env/jbds/download-cache/./soa-4.3.0.GA_CP05.zip 569M /home/hudson/static_build_env/jbds/requirements/download/soa-4.3.0.GA_CP05/./soa-4.3.0.GA_CP05.zip 569M /home/hudson/static_build_env/jbds/requirements/download/soap/./soa-4.3.0.GA_CP05.zip 1.7G total 593M /home/hudson/static_build_env/jbds/download-cache/./soa-5.0.1.GA-signed.zip 593M /home/hudson/static_build_env/jbds/requirements/download/soa-5.0.1.GA-signed/./soa-5.0.1.GA-signed.zip 593M /home/hudson/static_build_env/jbds/requirements/download/soap/./soa-5.0.1.GA-signed.zip 1.8G total 635M /home/hudson/static_build_env/jbds/download-cache/./soa-p-5.3.0.ER2.zip 635M /home/hudson/static_build_env/jbds/requirements/download/soa-p-5.3.0.ER2/./soa-p-5.3.0.ER2.zip 635M /home/hudson/static_build_env/jbds/requirements/download/soap/./soa-p-5.3.0.ER2.zip 1.9G total 629M /home/hudson/static_build_env/jbds/download-cache/./soa-p-5.3.0.ER5.zip du: cannot access `/home/hudson/static_build_env/jbds/requirements/download/*/./soa-p-5.3.0.ER5.zip': No such file or directory 629M total {code} Full disk footprint between the two locations: {code} $ du -sch /home/hudson/static_build_env/jbds/download-cache /home/hudson/static_build_env/jbds/requirements/download/ 5.6G /home/hudson/static_build_env/jbds/download-cache 13G /home/hudson/static_build_env/jbds/requirements/download/ 18G total {code} So... seems like there are two problems: a) we have a lot of old/obsolete reqs which can be purged b) we fetch into two places - I suspect one path (download-cache?) is the new approach mistria wrote for JBDS 6, and the old path is the old Ant-based fetch approach (requirements/download/). Spoke with Len and he said we were good to purge these, but I wanted to double-check w/ mistria and dgolovin that wiping out all 18G and letting the builds re-fetch what they need is an acceptable way to clean up this area. Also, can we fix JBDS 5 (and 4?) so it uses the new maven-based fetch instead of the old Ant one, in order to avoid the duplicates and disk usage? was: Noticed today that we have a lot of duplicate copies of the same runtimes. For example: {code} /home/hudson/static_build_env/jbds/download-cache/soa-p-5.3.0.ER2.zip /home/hudson/static_build_env/jbds/requirements/download/soa-p-5.3.0.ER2/soa-p-5.3.0.ER2.zip /home/hudson/static_build_env/jbds/requirements/download/soap/soa-p-5.3.0.ER2.zip {code} Doing a full search for dupes from the download-cache folder vs. the requirements/download/ folder, I see: {code} $ cd /home/hudson/static_build_env/jbds/download-cache; for d in $(find . -type f -name "*.zip" | sort); do du -sch {/home/hudson/static_build_env/jbds/download-cache/,/home/hudson/static_build_env/jbds/requirements/download/*/}${d} ; done 178M /home/hudson/static_build_env/jbds/download-cache/./GateIn-3.1.0-FINAL-jbossas.zip 178M /home/hudson/static_build_env/jbds/requirements/download/gatein-3.1.0.Final/./GateIn-3.1.0-FINAL-jbossas.zip 355M total 8.0K /home/hudson/static_build_env/jbds/download-cache/./jbds-seamfp-patch.zip du: cannot access `/home/hudson/static_build_env/jbds/requirements/download/*/./jbds-seamfp-patch.zip': No such file or directory 8.0K total 54M /home/hudson/static_build_env/jbds/download-cache/./jboss-3.2.8.SP1.zip 54M /home/hudson/static_build_env/jbds/requirements/download/jbossas/./jboss-3.2.8.SP1.zip 107M total 78M /home/hudson/static_build_env/jbds/download-cache/./jboss-4.0.5.GA.zip 78M /home/hudson/static_build_env/jbds/requirements/download/jbossas/./jboss-4.0.5.GA.zip 155M total 105M /home/hudson/static_build_env/jbds/download-cache/./jboss-5.0.1.GA.zip 105M /home/hudson/static_build_env/jbds/requirements/download/jbossas/./jboss-5.0.1.GA.zip 209M total 128M /home/hudson/static_build_env/jbds/download-cache/./jboss-5.1.0.GA.zip 128M /home/hudson/static_build_env/jbds/requirements/download/jbossas-5.1.0.GA/./jboss-5.1.0.GA.zip 128M /home/hudson/static_build_env/jbds/requirements/download/jbossas/./jboss-5.1.0.GA.zip 383M total 270M /home/hudson/static_build_env/jbds/download-cache/./jboss-eap-4.3.0.GA_CP10.zip du: cannot access `/home/hudson/static_build_env/jbds/requirements/download/*/./jboss-eap-4.3.0.GA_CP10.zip': No such file or directory 270M total 293M /home/hudson/static_build_env/jbds/download-cache/./jboss-eap-5.1.2.zip du: cannot access `/home/hudson/static_build_env/jbds/requirements/download/*/./jboss-eap-5.1.2.zip': No such file or directory 293M total 110M /home/hudson/static_build_env/jbds/download-cache/./jboss-eap-6.0.0.zip 110M /home/hudson/static_build_env/jbds/requirements/download/jbeap-6/./jboss-eap-6.0.0.zip 220M total 78M /home/hudson/static_build_env/jbds/download-cache/./jboss-eap-fp-4.3.0.CP03-FP01.zip 78M /home/hudson/static_build_env/jbds/requirements/download/seamfp/./jboss-eap-fp-4.3.0.CP03-FP01.zip 155M total 193M /home/hudson/static_build_env/jbds/download-cache/./jboss-eap-noauth-4.3.0.GA_CP03.zip 193M /home/hudson/static_build_env/jbds/requirements/download/jbeap/./jboss-eap-noauth-4.3.0.GA_CP03.zip 385M total 293M /home/hudson/static_build_env/jbds/download-cache/./jboss-eap-noauth-5.1.2.zip du: cannot access `/home/hudson/static_build_env/jbds/requirements/download/*/./jboss-eap-noauth-5.1.2.zip': No such file or directory 293M total 189M /home/hudson/static_build_env/jbds/download-cache/./jbossesb-4.11.zip 189M /home/hudson/static_build_env/jbds/requirements/download/jbossesb-4.11/./jbossesb-4.11.zip 377M total 157M /home/hudson/static_build_env/jbds/download-cache/./jboss-portal-2.7.2-bundled.zip 157M /home/hudson/static_build_env/jbds/requirements/download/JBoss-Portal-2.7.2.GA/./jboss-portal-2.7.2-bundled.zip 314M total 26M /home/hudson/static_build_env/jbds/download-cache/./JBossPortletBridge-2.1.0.FINAL.zip 26M /home/hudson/static_build_env/jbds/requirements/download/JBossPortletBridge-2.1.0.FINAL/./JBossPortletBridge-2.1.0.FINAL.zip 51M total 93M /home/hudson/static_build_env/jbds/download-cache/./jboss-seam-2.0.1.GA.zip 93M /home/hudson/static_build_env/jbds/requirements/download/seam-2.0.1.GA/./jboss-seam-2.0.1.GA.zip 186M total 89M /home/hudson/static_build_env/jbds/download-cache/./jboss-seam-2.0.2.SP1.zip 89M /home/hudson/static_build_env/jbds/requirements/download/seam-2.0.2.SP1/./jboss-seam-2.0.2.SP1.zip 177M total 89M /home/hudson/static_build_env/jbds/download-cache/./jboss-seam-2.0.3.CR1.zip 89M /home/hudson/static_build_env/jbds/requirements/download/seam-2.0.3.CR1/./jboss-seam-2.0.3.CR1.zip 178M total 107M /home/hudson/static_build_env/jbds/download-cache/./jboss-seam-2.1.1.GA.zip 107M /home/hudson/static_build_env/jbds/requirements/download/seam-2.1.1.GA/./jboss-seam-2.1.1.GA.zip 213M total 137M /home/hudson/static_build_env/jbds/download-cache/./jboss-seam-2.1.2.zip 137M /home/hudson/static_build_env/jbds/requirements/download/seam-2.1.2.GA/./jboss-seam-2.1.2.zip 273M total 141M /home/hudson/static_build_env/jbds/download-cache/./jboss-seam-2.2.0.CR1.zip 141M /home/hudson/static_build_env/jbds/requirements/download/seam-2.2.0.CR1/./jboss-seam-2.2.0.CR1.zip 281M total 113M /home/hudson/static_build_env/jbds/download-cache/./jboss-seam-2.2.0.GA.zip 113M /home/hudson/static_build_env/jbds/requirements/download/seam-2.2.0.GA/./jboss-seam-2.2.0.GA.zip 225M total 108M /home/hudson/static_build_env/jbds/download-cache/./jboss-seam-2.2.1.Final.zip 108M /home/hudson/static_build_env/jbds/requirements/download/seam-2.2.1.Final/./jboss-seam-2.2.1.Final.zip 216M total 110M /home/hudson/static_build_env/jbds/download-cache/./jboss-seam-2.3.0.ALPHA-distribution.zip 110M /home/hudson/static_build_env/jbds/requirements/download/seam-2.3.0/./jboss-seam-2.3.0.ALPHA-distribution.zip 219M total 24M /home/hudson/static_build_env/jbds/download-cache/./jbpm-3.1.4.zip 24M /home/hudson/static_build_env/jbds/requirements/download/jbpm-3.1.4/./jbpm-3.1.4.zip 47M total 8.2M /home/hudson/static_build_env/jbds/download-cache/./rhds-seam-gen-patch.zip 8.2M /home/hudson/static_build_env/jbds/requirements/download/eapseamgen/./rhds-seam-gen-patch.zip 17M total 60M /home/hudson/static_build_env/jbds/download-cache/./riftsaw-2.3.0.Final.zip 60M /home/hudson/static_build_env/jbds/requirements/download/riftsaw-2.3.0.Final/./riftsaw-2.3.0.Final.zip 120M total 569M /home/hudson/static_build_env/jbds/download-cache/./soa-4.3.0.GA_CP05.zip 569M /home/hudson/static_build_env/jbds/requirements/download/soa-4.3.0.GA_CP05/./soa-4.3.0.GA_CP05.zip 569M /home/hudson/static_build_env/jbds/requirements/download/soap/./soa-4.3.0.GA_CP05.zip 1.7G total 593M /home/hudson/static_build_env/jbds/download-cache/./soa-5.0.1.GA-signed.zip 593M /home/hudson/static_build_env/jbds/requirements/download/soa-5.0.1.GA-signed/./soa-5.0.1.GA-signed.zip 593M /home/hudson/static_build_env/jbds/requirements/download/soap/./soa-5.0.1.GA-signed.zip 1.8G total 635M /home/hudson/static_build_env/jbds/download-cache/./soa-p-5.3.0.ER2.zip 635M /home/hudson/static_build_env/jbds/requirements/download/soa-p-5.3.0.ER2/./soa-p-5.3.0.ER2.zip 635M /home/hudson/static_build_env/jbds/requirements/download/soap/./soa-p-5.3.0.ER2.zip 1.9G total 629M /home/hudson/static_build_env/jbds/download-cache/./soa-p-5.3.0.ER5.zip du: cannot access `/home/hudson/static_build_env/jbds/requirements/download/*/./soa-p-5.3.0.ER5.zip': No such file or directory 629M total {code} Full disk footprint between the two locations: {code} $ du -sch /home/hudson/static_build_env/jbds/download-cache /home/hudson/static_build_env/jbds/requirements/download/ 5.6G /home/hudson/static_build_env/jbds/download-cache 13G /home/hudson/static_build_env/jbds/requirements/download/ 18G total {code} So... seems like there are two problems: a) we have a lot of old/obsolete reqs which can be purged b) we fetch into two places - I suspect one path (download-cache?) is the new approach mistria wrote for JBDS 6, and the old path is the old Ant-based fetch approach (requirements/download/). Spoke with Len and he said we were good to purge these, but I wanted to double-check w/ mistria and dgolovin that wiping out all 18G and letting the builds re-fetch what they need is an acceptable way to clean up this area. Also, can we fix JBDS 5 (and 4?) so it uses the new maven-based fetch instead of the old Ant one, in order to avoid the duplicates and disk usage? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira