Suggestion: Github Cleanup (Branches, tags, some history leftovers) #6835

dimitre · 2022-01-19T04:01:34Z

I think this branches can be safely deleted

ofThreadChannel-trySend
bugfix-hdv

some tags can be cleaned to and kept for releases or intermediate builds

and recently I've cloned the openFrameworks repo to list the top 20 largest files in there, and it seems almost none should be part of the distribution anyway. here is the command

git rev-list --objects --all \
  | grep "$(git verify-pack -v .git/objects/pack/*.idx \
           | sort -k 3 -n \
           | tail -20 \
           | awk '{print$1}')"

and the result

81606070fba217248e20fe33c645557174bd4a48 addons/ofxAssimpModelLoader/libs/assimp/lib/ios/assimp.a
4d3665d57e655419eae40ff2eccbea02f7bbf89e addons/ofxOpenCv/libs/ippicv/lib/vs/Win32/ippicvmt.lib
8bbefa53a3c0c31ff5586154ffa3510eb0bfb3f0 addons/ofxOpenCv/libs/ippicv/lib/vs/x64/ippicvmt.lib
7a7be13d79a11cf05267e7feb26a044b51ba2888 addons/ofxOpenCv/libs/opencv/lib/osx/opencv.a
8cd61fc251a3faca852ebae26bf9d95fda897b43 addons/ofxOpenCv/libs/opencv/lib/vs/x64/Debug/opencv_imgproc310d.lib
f275e6e8254d7816dab253aaa497bffdf296fb91 addons/ofxOpenCv/libs/opencv/lib/osx/opencv.a
54807c5869f95893ed5d4b352190637ce0b4f8a5 libs/poco/lib/vs/Win32/PocoDatamdd.lib
48ebeb5ea46ff50c1e06e0c19a3a3c98f1d42352 libs/poco/lib/vs/x64/PocoDatamdd.lib
d80172553712228ad3503c772e95458b47c440ba libs/poco/lib/vs/x64/PocoDatamdd.lib
47fcea465ac8e6677302c9dc83e6c9098cbf4128 libs/poco/lib/vs/x64/PocoNetmdd.lib
f15c21a8af00bc39b4eeb82a4dbdc822734f183a libs/uri/lib/vs/x64/network-uri_debug.lib
a3475b229231b0fe4befbbb7ceb56e9c11005c82 addons/ofxAssimpModelLoader/libs/assimp/lib/ios/assimp.a
feb0f5a76c03ff71f5959c874b22540ace6cd1dd libs/FreeImage/lib/tvos/freeimage.a
4bb178f4f4e0d47419e44c6023abd9248774b235 addons/ofxOpenCv/libs/opencv/lib/osx/opencv.a
cb016d3a98a1dd50033a9a981f5a6288e64a5960 addons/ofxOpenCv/libs/opencv/lib/osx/opencv.a
2fe0b9acb098c17776b24a63ee3c7410bab87ba7 libs/poco/lib/vs/PocoCryptomdd.lib
233476b88643ec5e6f64d29fa5dcad6ca120635e libs/openFrameworksCompiled/lib/linuxarmv7l/libopenFrameworksDebug.a
5015c0166c0ee3f5e0a3d037de854a6e5427b4fc addons/ofxOpenCv/libs/opencv/lib/iphone/opencv.a
913db6b0f700adad9d421acaad2b791ea917800e libs/FreeImage/lib/iphone/libFreeImage_iphone_universal.a
96b1bf56f47f99b1c476406193aab29eda62ab91 libs/FreeImage/lib/linux64/libfreeimage.a

The text was updated successfully, but these errors were encountered:

dimitre · 2022-04-05T16:20:48Z

some files should be removed from git history, like addons/ofxAssimpModelLoader/libs/assimp/lib/ios/assimp.a

it takes a while but this can make way for a lean git

git filter-branch --index-filter "git rm -rf --cached --ignore-unmatch addons/ofxAssimpModelLoader/libs/assimp/lib/ios/assimp.a" HEAD

#6841 (comment)

dimitre · 2022-05-28T16:12:03Z

now running the top 100 files, we can see none of them are useful now.
this command takes a while to run

git rev-list master | while read rev; do git ls-tree -lr $rev  | cut -c54- | sed -r 's/^ +//g;'; done  | sort -u | perl -e 'while (<>) { chomp; @stuff=split("\t");$sums{$stuff[1]} += $stuff[0];} print "$sums{$_} $_\n" for (keys %sums);' | sort -rn | head -n 100

241986536 addons/ofxOpenCv/libs/opencv/lib/osx/opencv.a
174043100 libs/poco/lib/vs/PocoFoundationmdd.lib
125837804 addons/ofxAssimpModelLoader/libs/assimp/lib/ios/assimp.a
122668796 libs/FreeImage/lib/osx/freeimage.a
111412166 libs/poco/lib/vs/PocoNetmdd.lib
97117984 libs/uri/lib/vs/x64/network-uri_debug.lib
87467074 libs/poco/lib/vs2010/PocoFoundationmdd.lib
86410344 addons/ofxAssimpModelLoader/libs/assimp/lib/osx/assimp.a
74870592 addons/ofxOpenCv/libs/opencv/lib/ios/opencv.a
74665896 libs/FreeImage/lib/ios/freeimage.a
72302584 libs/poco/lib/vs2012/PocoFoundationmdd.lib
67763948 addons/ofxOpenCv/libs/opencv/lib/vs/x64/Debug/opencv_imgproc310d.lib
67446188 libs/poco/lib/vs/PocoXMLmdd.lib
65318614 libs/poco/lib/vs/PocoNetSSLmd.lib
64122226 libs/uri/lib/vs/Win32/network-uri_debug.lib
63164510 libs/poco/lib/vs/x64/PocoDatamdd.lib
62283420 addons/ofxOpenCv/libs/ippicv/lib/vs/x64/ippicvmt.lib
59088148 libs/poco/lib/vs/PocoCryptomd.lib
58488390 libs/poco/lib/vs/x64/PocoFoundationmdd.lib
57405420 libs/poco/lib/vs2010/PocoNetmdd.lib
57047954 libs/poco/lib/vs/PocoUtilmdd.lib
55734986 addons/ofxOpenCv/libs/opencv/lib/vs/x64/Debug/opencv_core310d.lib
55355428 libs/FreeImage/lib/linux64/libfreeimage.a
53815780 libs/poco/lib/ios/PocoFoundation.a
52284164 addons/ofxOpenCv/libs/opencv/lib/ios/opencv_core.a
48893496 libs/poco/lib/vs/PocoFoundationmd.lib
48275328 addons/ofxOpenCv/libs/ippicv/lib/vs/Win32/ippicvmt.lib
48237200 libs/opencv/lib/osx/libOpenCV.a
48079856 addons/ofxOpenCv/libs/opencv/lib/iphone/opencv.a
47930668 addons/ofxOpenCv/libs/opencv/lib/ios/opencv_imgproc.a
47187610 libs/poco/lib/vs2012/PocoNetmdd.lib
46153176 addons/ofxOpenCv/libs/opencv/lib/vs/Win32/Debug/opencv_imgproc310d.lib
45373558 libs/poco/lib/vs/PocoNetSSLmdd.lib
44304008 libs/FreeImage/lib/iphone/libFreeImage_iphone_universal.a
44230000 libs/FreeImage/lib/android/armeabi-v7a/libfreeimage.a
43078084 libs/poco/lib/vs/PocoCryptomdd.lib
42483056 libs/FreeImage/lib/tvos/freeimage.a
42181468 libs/poco/lib/vs/Win32/PocoDatamdd.lib
42068476 addons/ofxAssimpModelLoader/libs/assimp/lib/android/armeabi-v7a/libassimp.a
41686498 libs/poco/lib/vs/Win32/PocoFoundationmdd.lib
41672540 libs/openssl/lib/ios/crypto.a
40680392 addons/ofxOpenCv/libs/opencv/lib/vs/x64/Debug/opencv_calib3d310d.lib
39778848 libs/FreeImage/lib/iphone/freeimage.a
38890308 libs/poco/lib/vs/x64/PocoNetmdd.lib
38069442 addons/ofxOpenCv/libs/opencv/lib/vs/x64/Debug/opencv_features2d310d.lib
37725554 addons/ofxOpenCv/libs/opencv/lib/vs/Win32/Debug/opencv_core310d.lib
37318682 libs/openssl/lib/vs/Win32/libeay32md.lib
36407582 libs/poco/lib/vs2010/PocoUtilmdd.lib
35693876 addons/ofxOpenCv/libs/opencv/lib/vs/x64/Debug/opencv_stitching310d.lib
35439892 libs/poco/lib/ios/PocoNet.a
34527594 libs/poco/lib/vs2010/PocoXMLmdd.lib
33943392 libs/openFrameworksCompiled/lib/linuxarmv7l/libopenFrameworksDebug.a
32873948 libs/poco/lib/vs/Win32/PocoNetSSLmd.lib
32690848 addons/ofxAssimpModelLoader/libs/assimp/lib/android/x86/libassimp.a
32645956 libs/poco/lib/vs/PocoNetmd.lib
32102648 libs/FreeImage/lib/android/x86/libfreeimage.a
31881836 libs/poco/lib/ios/PocoData.a
31689102 libs/poco/lib/android/armeabi-v7a/libPocoFoundation.a
31389852 libs/poco/lib/vs/Win32/PocoCryptomd.lib
31293960 libs/poco/lib/vs2010/PocoNetmtd.lib
31276192 addons/ofxOpenCv/libs/opencv/lib/vs2010/opencv_imgproc220d.lib
30660202 libs/cairo/lib/vs/Win32/cairo-static.lib
28975452 addons/ofxOpenCv/libs/opencv/lib/vs2010/opencv_features2d220d.lib
28913660 libs/poco/lib/vs2010/PocoFoundationmtd.lib
28654928 addons/ofxOpenCv/libs/opencv/lib/vs/x64/Debug/opencv_ml310d.lib
28609368 libs/poco/lib/vs2012/PocoXMLmdd.lib
28315478 addons/ofxOpenCv/libs/opencv/lib/vs/x64/opencv_imgproc249d.lib
28315478 addons/ofxOpenCv/libs/opencv/lib/vs/x64/Release/opencv_imgproc249d.lib
28315478 addons/ofxOpenCv/libs/opencv/lib/vs/x64/Debug/opencv_imgproc249d.lib
28181368 addons/ofxOpenCv/libs/opencv/lib/vs/x64/opencv_core249d.lib
28181368 addons/ofxOpenCv/libs/opencv/lib/vs/x64/Release/opencv_core249d.lib
28181368 addons/ofxOpenCv/libs/opencv/lib/vs/x64/Debug/opencv_core249d.lib
28064256 libs/FreeImage/lib/iphone/freeimage-iphone.a
27883010 libs/poco/lib/vs/Win32/PocoNetmdd.lib
27673506 addons/ofxOpenCv/libs/opencv/lib/vs/x64/Debug/opencv_flann310d.lib
26931248 libs/poco/lib/vs2012/PocoNetSSLmd.lib
26931248 libs/poco/lib/vs2010/PocoNetSSLmd.lib
26275490 addons/ofxOpenCv/libs/opencv/lib/vs/Win32/Debug/opencv_calib3d310d.lib
26162498 addons/ofxOpenCv/libs/opencv/lib/vs/x64/opencv_contrib249d.lib
26162498 addons/ofxOpenCv/libs/opencv/lib/vs/x64/Release/opencv_contrib249d.lib
26162498 addons/ofxOpenCv/libs/opencv/lib/vs/x64/Debug/opencv_contrib249d.lib
25898084 libs/poco/lib/vs2012/PocoCryptomd.lib
25898084 libs/poco/lib/vs2010/PocoCryptomd.lib
25893242 libs/openssl/lib/android/x86/libcrypto.a
25808204 addons/ofxOpenCv/opencv2/lib/linux64/libcv.a
25682042 libs/poco/lib/vs2012/PocoUtilmdd.lib
25415396 addons/ofxOpenCv/libs/opencv/lib/vs/Win32/Debug/opencv_features2d310d.lib
25414064 libs/poco/lib/osx/PocoFoundation.a
25365696 libs/poco/lib/linux64/libPocoFoundation.a
25049670 libs/openssl/lib/vs/x64/libeay32md.lib
24888120 libs/cairo/lib/vs/x64/cairo-static.lib
24311312 libs/poco/lib/vs2010/PocoFoundationmd.lib
23953480 addons/ofxOpenCv/libs/opencv/lib/vs/opencv_imgproc231d.lib
23770660 addons/ofxOpenCv/libs/opencv/lib/android/armeabi-v7a/libopencv_core.a
23555988 addons/ofxOpenCv/libs/opencv/lib/android/armeabi-v7a/libopencv_imgproc.a
23399692 addons/ofxOpenCv/libs/opencv/lib/vs/Win32/Debug/opencv_stitching310d.lib
23306848 addons/ofxOpenCv/libs/opencv/lib/osx/openCV.a
23306848 addons/ofxOpenCv/libs/opencv/lib/osx/libOpenCV.a
23240316 libs/poco/lib/vs/x64/PocoUtilmdd.lib
23045456 addons/ofxOpenCv/opencv2/lib/linux64/libcxcore.a

dimitre · 2022-05-29T16:24:09Z

From actual OF git (without installing libs) we don't have any useful file with more than 10Mb. the biggest file is Batang.ttf
we can check with this command

find . -type f -size +10M -exec ls -lh {} \;

there is this nice tool, git-filter-repo which can remove blobs from history, like

git filter-repo --strip-blobs-bigger-than 13M

after the command and reflog the repo goes down from 2.2Gb to 1.13Gb

roymacdonald · 2023-05-02T17:57:31Z

sounds like a great idea.
Have you tried out BFG ? Might work for the task and claims to be a lot faster than the regular git commands.

danoli3 · 2023-05-03T03:19:46Z

Rewriting history should happen, it will just kill all concurrent PR’s as every hash is changed all the way back to the first rewrite. I would recommend we merge all the major PR’s first. We do need to remove the binaries

…

On Wed, 3 May 2023 at 3:57 am, Roy Macdonald ***@***.***> wrote: sounds like a great idea. Have you tried out BFG <https://rtyley.github.io/bfg-repo-cleaner/> ? Might work for the task and claims to be a lot faster than the regular git commands. — Reply to this email directly, view it on GitHub <#6835 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGK2HFBJ3WWHS34AJOS6XTXEFDJ3ANCNFSM5MI3DRHQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

ofTheo · 2023-05-03T04:10:36Z

yeah - I am not in a rush to do a rewrite.
maybe when we are doing a bunch of other big cleanup work and things are moving so much.

danoli3 · 2023-05-03T05:02:39Z

See our 2017 discussion about this here: #5562

Also all the commands and even a chime in from the BFG creator :D!

danoli3 · 2023-05-03T05:04:07Z

Sorry this discussion: #1805 (comment)

danoli3 · 2023-05-03T05:06:11Z

The analysis then was:

I did a quick test with Big Repo Cleaner after seeing a checkout of the current openFrameworks branch is

1.43 GB
Test results after running Big Repo Cleaner:
https://rtyley.github.io/bfg-repo-cleaner/

After a full binary removal (.a, .exe, .lib, .so, .dll, .app) from all history the repo size is reduced to:

111 MB
So the current concerns I heard a while ago was that a repo clean like this destroys the history and breaks forks.

Well I tested both and my results were surprisingly positive with:

1950+ Binaries Removed

Down to ~111MB - This would retain GIT stability for the future.
Anytime I have tried to checkout the repo on a shaky internet connection without a solid internet connection it has always timed out without shallow clone

danoli3 · 2023-05-03T05:08:52Z

I've done this process since this initial analysis in 2017 for numerous clients and game projects where the binaries were attached. Or in cases where a GIT LFS upgrade was performed.

It caused chaos if any FORK or un pushed code was then compared or attempted to be merged as every history was then compared. I do think there will a moment we should do this. It's just a messy one. I can experiment again on another repo with an up to date analysis and code to do so

rajeevraizada · 2023-08-05T13:13:53Z

I did a git clone just now (Aug.5, 2023), and was surprised by how long the download was taking.

It turned out that it included almost 2Gb of .git/objects/pack files. Googling told me that "The packfile is a single file containing the contents of all the objects that were removed from your filesystem." So, it seems like it would be completely harmless, and also very easy, to remove those files. That single step would reduce the download size by almost 90%.

dimitre changed the title ~~Suggestion: Branches cleanup~~ Suggestion: Github Cleanup (Branches, tags, some history leftovers) Apr 5, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Suggestion: Github Cleanup (Branches, tags, some history leftovers) #6835

Suggestion: Github Cleanup (Branches, tags, some history leftovers) #6835

dimitre commented Jan 19, 2022 •

edited

Loading

dimitre commented Apr 5, 2022 •

edited

Loading

dimitre commented May 28, 2022 •

edited

Loading

dimitre commented May 29, 2022 •

edited

Loading

roymacdonald commented May 2, 2023

danoli3 commented May 3, 2023 via email

ofTheo commented May 3, 2023

danoli3 commented May 3, 2023

danoli3 commented May 3, 2023

danoli3 commented May 3, 2023

danoli3 commented May 3, 2023

rajeevraizada commented Aug 5, 2023

Suggestion: Github Cleanup (Branches, tags, some history leftovers) #6835

Suggestion: Github Cleanup (Branches, tags, some history leftovers) #6835

Comments

dimitre commented Jan 19, 2022 • edited Loading

dimitre commented Apr 5, 2022 • edited Loading

dimitre commented May 28, 2022 • edited Loading

dimitre commented May 29, 2022 • edited Loading

roymacdonald commented May 2, 2023

danoli3 commented May 3, 2023 via email

ofTheo commented May 3, 2023

danoli3 commented May 3, 2023

danoli3 commented May 3, 2023

danoli3 commented May 3, 2023

danoli3 commented May 3, 2023

rajeevraizada commented Aug 5, 2023

dimitre commented Jan 19, 2022 •

edited

Loading

dimitre commented Apr 5, 2022 •

edited

Loading

dimitre commented May 28, 2022 •

edited

Loading

dimitre commented May 29, 2022 •

edited

Loading