[rules-dev] Split-up: the gory details (for future reference)

For future reference, I am copy-pasting the methodology I used during the split-up in our dev mailing list archive. Just ignore it. However, do take a look at the blog about the split-up, which has some interesting numbers: http://blog.athico.com/2011/03/splitting-up-into-smaller-git.html -- With kind regards, Geoffrey De Smet split-up ======== Gather all in one directory --------------------------- Comment out the modules with: <!-- Already split-off to separate repo --> Run a clean build: mvn clean install -DskipTests Commit with message: JBRULES-2869: Split-off preparation: move everything for a separate repository together into a single directory Do split -------- git clone git@github.com:droolsjbpm/droolsjbpm.git droolsjbpm-untouched cd droolsjbpm-untouched/ git fetch git branch git pull --rebase cd .. cp -R droolsjbpm-untouched <repoName>-uncloned cd <repoName>-uncloned git remote rm origin git branch -a git tag -l git tag -d 3.0-GA git tag -d 3.0-GA@4643 git tag -d 3.0.1-GA git tag -d 3.0.2-GA git tag -d 3.0.3-GA git tag -d 3.0.5-GA git tag -d 3.0.6-GA git tag -d 4.0.0.13773GA git tag -d 4.0.1.14754GA git tag -d 4.0.2-SOA_4.2.CP01 git tag -d 4.0.2.15666.GA git tag -d 4.0.2.15830.SOA-4-2.CP01 git tag -d 4.0.3.15993.GA git tag -d 4.0.4.17825GA git tag -d 4.0.5.19064GA git tag -d 4.0.6.19326.GA git tag -d 4.0.7.19894.GA git tag -d 5.0.0.26115.FINAL git tag -d 5.0.1.26597.FINAL git tag -d 5.1.0.34406.FINAL git tag -d 5.1.1.34858.FINAL git tag -d 5.2.0.M1 git tag -d DROOLS_SOA_4_2_18483_GA git tag -d DROOLS_SOA_4_2_19342_CP1 git tag -d DROOLS_SOA_4_2_19937_CP2 git tag -d DROOLS_SOA_4_2_CR3_18258 git tag -d before_merge_factsconstraint git tag -l Depending on the split-off: Canonical: git filter-branch --index-filter 'git ls-tree --name-only --full-tree $GIT_COMMIT | grep -v "^\(\(\.git\)\|...\)$" | xargs -I {} git rm --cached -r {}' --tag-name-filter cat --prune-empty -- --all For planner ----------- git filter-branch --index-filter 'git ls-tree --name-only --full-tree $GIT_COMMIT | grep -v "^\(\(\.git\)\|\(drools\-docs\)\|\(drools-planner\)\|\(drools-solver\)\)$" | xargs -I {} git rm --cached -r {}' --tag-name-filter cat --prune-empty -- --all The list: drools-docs drools-planner drools-solver For tools --------- git filter-branch --index-filter 'git ls-tree --name-only --full-tree $GIT_COMMIT | grep -v "^\(\(\.git\)\|\(droolsjbpm\-tools\)\|\(drools\-ant\)\|\(drools\-eclipse\)\|\(drools\-ide\)\|\(drools\-ide\-feature\)\|\(drools\-ide\-test\)\|\(drools\-eclipse3\.2\)\|\(drools\-eclipse3\.3\)\)$" | xargs -I {} git rm --cached -r {}' --tag-name-filter cat --prune-empty -- --all The list: droolsjbpm-tools drools-ant drools-eclipse drools-ide drools-ide-feature drools-ide-test drools-eclipse3.2 drools-eclipse3.3 For guvnor ---------- git filter-branch --index-filter 'git ls-tree --name-only --full-tree $GIT_COMMIT | grep -v "^\(\(\.git\)\|\(drools\-docs\)\|\(drools\-examples\)\|\(guvnor\)\|\(drools\-guvnor\)\|\(drools\-jbrms\)\|\(drools\-atom\)\|\(drools\-factconstraint\)\|\(drools\-ide\-common\)\|\(drools\-repository\)\|\(drools\-repository\-parent\)\)$" | xargs -I {} git rm --cached -r {}' --tag-name-filter cat --prune-empty -- --all The list: drools-docs drools-examples guvnor drools-guvnor drools-jbrms drools-atom drools-factconstraint drools-ide-common drools-repository drools-repository-parent Also remove the GWT generated binaries from the history: git filter-branch --index-filter 'git ls-tree --name-only --full-tree -r $GIT_COMMIT | grep --regex "^.*/src/main/webapp/.*\(\(\.gwt\.rpc\)\|\(\.cache\.html\)\|\(\.cache\.js\)\|\(\.cache\.xml\)\)$" | xargs -I {} git rm --cached {}' --tag-name-filter cat --prune-empty -- --all The list: src/main/webapp/org.drools.guvnor.Guvnor/**/*.gwt.rpc src/main/webapp/org.drools.guvnor.Guvnor/**/*.cache.html src/main/webapp/org.drools.guvnor.Guvnor/**/*.cache.js src/main/webapp/org.drools.guvnor.Guvnor/**/*.cache.xml For integration --------------- git filter-branch --index-filter 'git ls-tree --name-only --full-tree $GIT_COMMIT | grep -v "^\(\(\.git\)\|\(drools\-docs\)\|\(drools\-grid\)\|\(drools\-container\)\|\(drools\-spring\)\|\(drools\-seam\)\|\(drools\-camel\)\|\(drools\-server\)\|\(drools\-pipeline\)\|\(drools\-rhq\-plugin\)\|\(drools\-simulator\)\)$" | xargs -I {} git rm --cached -r {}' --tag-name-filter cat --prune-empty -- --all The list: drools-docs drools-grid drools-container drools-spring drools-seam drools-camel drools-server drools-pipeline drools-rhq-plugin drools-simulator For drools ---------- git filter-branch --index-filter 'git ls-tree --name-only --full-tree $GIT_COMMIT | grep "^\(\(blog\)\|\(lib\)\|\(m2_repo\)\|\(repository\)\|\(drools\-flow\-compiler\)\|\(drools\-flow\-core\)\|\(drools\-flow\-persistence\-jpa\)\|\(drools\-process\)\|\(drools-planner\)\|\(drools-solver\)\|\(droolsjbpm\-tools\)\|\(drools\-ant\)\|\(drools\-eclipse\)\|\(drools\-ide\)\|\(drools\-ide\-feature\)\|\(drools\-ide\-test\)\|\(drools\-eclipse3\.2\)\|\(drools\-eclipse3\.3\)\|\(guvnor\)\|\(drools\-guvnor\)\|\(drools\-jbrms\)\|\(drools\-atom\)\|\(drools\-factconstraint\)\|\(drools\-ide\-common\)\|\(drools\-repository\)\|\(drools\-repository\-parent\)\|\(drools\-grid\)\|\(drools\-container\)\|\(drools\-spring\)\|\(drools\-seam\)\|\(drools\-camel\)\|\(drools\-server\)\|\(drools\-pipeline\)\|\(drools\-rhq\-plugin\)\|\(drools\-simulator\)\|\(drools\-api\)\|\(.*\.xml\)\|\(.*\.txt\)\|\(.*\.TXT\)\|\(.*\.env\)\|\(.*\.sh\)\|\(.*\.properties\)\(\.gitignore\)\|\(src\)\|\(install\)\|\(osgi\-bundles\)\)$" | xargs -I {} git rm --cached -r {}' --tag-name-filter cat --prune-empty -- --all The list is NOT: blog lib m2_repo repository drools-flow-compiler drools-flow-core drools-flow-persistence-jpa drools-process + rest of other migrations also not except drools-examples and drools-docs For droolsjbpm-knowledge ------------------------ git filter-branch --index-filter 'git ls-tree --name-only --full-tree $GIT_COMMIT | grep -v "^\(\(\.git\)\|\(drools\-docs\)\|\(drools\-api\)\)$" | xargs -I {} git rm --cached -r {}' --tag-name-filter cat --prune-empty -- --all The list: drools-docs drools-api For droolsjbpm-build-bootstrap ---------- git filter-branch --index-filter 'git ls-tree --name-only --full-tree $GIT_COMMIT | grep -v "^\(\(\.git\)\|\(.*\.xml\)\|\(.*\.txt\)\|\(.*\.TXT\)\|\(.*\.env\)\|\(.*\.sh\)\|\(.*\.properties\)\(\.gitignore\)\)$" | xargs -I {} git rm --cached -r {}' --tag-name-filter cat --prune-empty -- --all The list: pom.xml README.txt LICENSE-ASL-2.0.txt eclipse-cleanup.xml eclipse-code-style.xml For droolsjbpm-build-distribution --------------------------------- git filter-branch --index-filter 'git ls-tree --name-only --full-tree $GIT_COMMIT | grep -v "^\(\(\.git\)\|\(src\)\|\(install\)\|\(osgi\-bundles\)\)$" | xargs -I {} git rm --cached -r {}' --tag-name-filter cat --prune-empty -- --all The list: src install osgi-bundles Then continue with: ------------------- cd .. git clone file:///home/gdesmet/projects/jboss/split-up-migration/<repoName>-uncloned/ <repoName> cd <repoName> git remote rm origin git remote add origin git@github.com:droolsjbpm/<repoName>.git git push --all After the migration =================== Remove from monolithic-droolsjbpm --------------------------------- Delete the directories split-off in the droolsjbpm repository. Comment out their <module> reference in the parent pom: <!-- Already split-off to separate repo --> Run: mvn clean install -DskipTests Commit with message: JBRULES-2869: Split-off of module GGG into separate git repository Clean up in new repository -------------------------- Remove non-needed docs and examples dirs. Commit with message: JBRULES-2869: Split-off: remove directories from docs and examples that do not belong in the repository Add license file ---------------- Copy the LICENSE-ASL-2.0.txt file Commit with message: JBRULES-2869: Split-off requires license file per repo Hook in multiproject -------------------- Hook in the new multiproject parent pom. mvn clean install -DskipTests Commit with message: JBRULES-2869: Hook in the new multiproject as parent after split-up dos2unix -------- Run command: find . -type f -regex ".*\.java$" -and -not -path \*/\.git/\* -exec dos2unix {} \; find . -type f -regex ".*\.xml$" -and -not -path \*/\.git/\* -exec dos2unix {} \; find . -type f -not -path \*/\.git/\* -exec dos2unix {} \; Review files that you don't want to dos2unix Commit with message: JBRULES-2855 Dos2unix all source files (except .bat) Copyright javadoc ----------------- Replace in all *.java files regex case sensitive: /\*\*\n \* Copyright with /*\n * Copyright Search in all files regex case sensitive: /\*\*\n \* Copyright Commit with message: JBRULES-2577 License headers in java files are not valid javadocs and should not start with /** Remove all \t suffixes ---------------------- Replace in all *.java files regex case sensitive: ([;\{\}])[\t ]+$ with $1 Commit with message: JBRULES-2855 Remove all trailing whitespace in java files Tabs ---- Replace in all *.java/drl files regex case sensitive: \t with 4 spaces Or replace multiple times in all *.java/drl files regex case sensitive: ^( *)\t with (without quotes) "$1 " Replace in all *.xml files regex case sensitive: \t with 2 spaces Search in all files regex: \t Commit with message: JBRULES-2855 Do not use tabs (\t) for indentation, use spaces instead: 4 for java files, 2 for xml, 4 for drl Strip author ------------ Replace in all *.java files regex case sensitive: ^ *\* +@author .*\n with nothing. Delete empty javadocs. Replace in all *.java files regex case sensitive: ^ */\*\*\n( *\* *\n)* *\*/\n with nothing Delete trailing empty javadoc lines: Replace in all *.java files regex case sensitive: ^( *\* *\n)+( *)\*/ with $2*/ Search in all files: author Commit with message: JBRULES-2895 Strip all author tags from java files - For motivation see https://issues.jboss.org/browse/JBRULES-2895 Rename to new directory structure --------------------------------- Do renames Commit with message: JBRULES-2869: Rename to new directory structure after split-up Add repository jboss public in main pom.xml: <repositories> <!-- Bootstrap repository to locate the parent pom when the parent pom has not been build locally. --> <repository> <id>jboss-public-repository-group</id> <name>JBoss Public Repository Group</name> <url>http://repository.jboss.org/nexus/content/groups/public/</url> <layout>default</layout> <releases> <enabled>true</enabled> <updatePolicy>never</updatePolicy> </releases> <snapshots> <enabled>true</enabled> <updatePolicy>never</updatePolicy> </snapshots> </repository> </repositories> Commit with message: JBRULES-2869: Include bootstrap repository Configure hudson job -------------------- TODO for planner and tools + mutliproject pom per repo # parent repository in it # correct parent pom path Failed alternatives =================== Probably not needed: git checkout --track remotes/origin/3.0.x git checkout --track remotes/origin/4.0.x git checkout --track remotes/origin/4.1.x git checkout --track remotes/origin/5.0.x git checkout --track remotes/origin/5.1.x git checkout --track remotes/origin/5.2.0.M1.x git checkout --track remotes/origin/K200 git checkout --track remotes/origin/lr_unlinking_20101116 git checkout --track remotes/origin/persistence_refactor git checkout --track remotes/origin/rete_using_static_methods_aug2009 git checkout master Alt 1: cp -R droolsjbpm-untouched drools-planner-uncloned cd drools-planner-uncloned git remote rm origin git remote -v git branch -D K200 git branch -D lr_unlinking_20101116 git branch -d persistence_refactor git branch -D rete_using_static_methods_aug2009 git branch -d 3.0.x git filter-branch --subdirectory-filter drools-solver --tag-name-filter cat 4.0.x git filter-branch --subdirectory-filter drools-solver --tag-name-filter cat -f 4.1.x git filter-branch --subdirectory-filter drools-solver --tag-name-filter cat -f 5.0.x git filter-branch --subdirectory-filter drools-planner --tag-name-filter cat -f 5.1.x git filter-branch --subdirectory-filter drools-planner --tag-name-filter cat -f 5.2.0.M1.x git filter-branch --subdirectory-filter drools-planner --tag-name-filter cat -f master cd .. git clone file:///home/gdesmet/projects/jboss/split-up-migration/drools-planner-uncloned drools-planner cd drools-planner git checkout --track remotes/origin/4.0.x git checkout --track remotes/origin/4.1.x git checkout --track remotes/origin/5.0.x git checkout --track remotes/origin/5.1.x git checkout --track remotes/origin/5.2.0.M1.x git checkout master git remote rm origin git remote add origin git@github.com:droolsjbpm/drools-planner.git git push --all Alt 2: git clone --no-hardlinks file:///home/gdesmet/projects/jboss/split-up-migration/droolsjbpm-untouched drools-planner-uncloned cd drools-planner-uncloned git remote rm origin git filter-branch --subdirectory-filter drools-planner -- --all git reset --hard git gc --prune cd .. git clone file:///home/gdesmet/projects/jboss/split-up-migration/drools-planner-uncloned drools-planner cd drools-planner git remote rm origin git remote add origin git@github.com:droolsjbpm/drools-planner.git git push --all Alt 3: git filter-branch --tree-filter 'rm -Rf lib' -- --all for i in $(ls) ; do echo $i done git filter-branch --tree-filter "find . -not -regex '\.' -and -not -regex '\./git.*' -and -not -regex '\./drools-planner.*' -and -not -regex '\./drools-solver.*' -and -not -regex '\./drools-docs/drools-docs-planner.*' -and -not -regex '\./drools-docs/drools-docs-solver.*' -delete" --tag-name-filter cat --prune-empty -- --all Alt 4: git remote rm origin git filter-branch --subdirectory-filter drools-planner --tag-name-filter cat --prune-empty -- --all => anything before containing "drools-planner" stays as-is so it stays 1GB Alt 5: git remote rm origin git filter-branch --index-filter 'git rm -r --cached --ignore-unmatch lib drools-api drools-core drools-compiler' --tag-name-filter cat --prune-empty -- --all Alt 6: git remote rm origin git filter-branch --index-filter 'git ls-tree --name-only --full-tree $GIT_COMMIT | grep -v "^\(\.git\)\|\(drools\-ant\)\|\(drools\-eclipse\)" | xargs git rm -r --cached --ignore-unmatch {}' --prune-empty master -- --all Alt 7: => WINNER git remote rm origin git branch -a git branch -D 3.0.x git branch -D 4.0.x git branch -D 4.1.x git branch -D 5.0.x git branch -D 5.1.x git branch -D 5.2.0.M1.x git branch -D K200 git branch -D lr_unlinking_20101116 git branch -D persistence_refactor git branch -D rete_using_static_methods_aug2009 git branch -a git tag -l git tag -d 3.0-GA git tag -d 3.0-GA@4643 git tag -d 3.0.1-GA git tag -d 3.0.2-GA git tag -d 3.0.3-GA git tag -d 3.0.5-GA git tag -d 3.0.6-GA git tag -d 4.0.0.13773GA git tag -d 4.0.1.14754GA git tag -d 4.0.2-SOA_4.2.CP01 git tag -d 4.0.2.15666.GA git tag -d 4.0.2.15830.SOA-4-2.CP01 git tag -d 4.0.3.15993.GA git tag -d 4.0.4.17825GA git tag -d 4.0.5.19064GA git tag -d 4.0.6.19326.GA git tag -d 4.0.7.19894.GA git tag -d 5.0.0.26115.FINAL git tag -d 5.0.1.26597.FINAL git tag -d 5.1.0.34406.FINAL git tag -d 5.1.1.34858.FINAL git tag -d 5.2.0.M1 git tag -d DROOLS_SOA_4_2_18483_GA git tag -d DROOLS_SOA_4_2_19342_CP1 git tag -d DROOLS_SOA_4_2_19937_CP2 git tag -d DROOLS_SOA_4_2_CR3_18258 git tag -d before_merge_factsconstraint git tag -l git filter-branch --subdirectory-filter drools-planner --prune-empty master # extra guvnor step? cd .. git clone file:///home/gdesmet/projects/jboss/split-up-migration/drools-planner-uncloned/ drools-planner cd drools-planner git remote rm origin git remote add origin git@github.com:droolsjbpm/drools-planner.git git push --all Alt 8: git filter-branch --index-filter 'git ls-tree --name-only --full-tree $GIT_COMMIT | grep -v "^\(\.git\)\|\(drools\-ant\)\|\(drools\-eclipse\)" | xargs -0 -I {} git rm --cached --ignore-unmatch -r {}' --prune-empty master Hickups ------- Redo planner migration by starting from an earlier tree: git reset --hard 73add05c3fd74b3fdf4456055d486b7a7547c28b