Migrate all branches to LFS #690
Comments
|
bash Or something like that. Oh and have another clone for backup. |
|
Should make CI builds a lot faster |
|
I suspect something may need to be done for tags too, but can't be certain without trying it. |
|
Ive just discovered that LFS on github has a quite small quota of 1gb, with additional purchasable data packs for both bandwidth (50gb for $5). As we are working with a large amount of data (git lfs pull says 2.3GB) the quota and datapacks arent going to last long, especially once automated builds are added into the mix. One key thing as well is that any forks use the quota from the repo, so any automated builds i setup for my branches will be using your quota. Meaning that any push I make to my fork will use ~4gb quota (assuming i build 2 versions). Seeing as yesterday I pushed 4 commits, that if my builds were running would have cost SVT $1.6, for changes which I don't have to submit back in a PR Depending on the ci it might be possible to cache the lfs data, but that depends on whether the cache is restored before or after the git clone travis-ci/travis-ci#8787 and relies on whoever set up the builds to consider doing that Some other projects have faced similar issues and have reconsidered their usage of lfs because of it. |
|
@dotarmin ping |
|
We hsouldm aybe just delete files in history using bfg |
He is away currently, but will be back tomorrow.
Would that be instead of using lfs? |
Absolutely, I can do that.
I think if we start with branches first and then move on to tags but I have to try it out first before I say more about specifically tags. |
That's right, when automated builds are added into the mix again the quota won't last for that long just as you say, but this issue only applies if we decide to have a build triggered for each push to GitHub. If we instead go with the approach to build once or two times per day then we could get rid of this problem and won't "waste" bandwidth. The bandwidth is used when a fork is created, when changes are pulled from upstream and when a commit and push is done to a file tracked with Git LFS source. If needed it would still be possible to start a manual build on-demand. Automated builds once or two times per day:
This isn't an issue for the moment but might be if we go with triggering a build for each push/pr merge to upstream and or if the bandwidth hits the ceil all the time.
I will check if their is a way to use cached Git LFS files. If their is a way to use the cached files then those would be pulled instead and you would receive only the files that has changed. But all this depends on how each community member sets up their own build-servers of course, if a fresh clone is done for each triggered build then this won't help. As the last way out to this is maybe to go back to not use Git LFS (puuh). What do you think? |
I don't have experience with bfg. |
I don't know if that is easily possible using a service like Travis, but is via Jenkins. I have no problem with branch builds happening up to a couple of times a day, but I would like to see PR builds. There have been a few PRs that don't build (mine included) on at least one platform, and it hasn't been noticed until after they are merged which then leaves them to someone else to fix, or for them to be reverted (both of which have happened recently)
I did have a quick google the other day, but didn't find anything. They are cached under .git/lfs so you will only need to pull them for each tree once, but each fresh tree will need to pull them again. The impact of this for ci will vary, depending on the service, as those which don't do a fresh clone each time (Jenkins typically does this) will have less impact.
I am leaning this way currently, but am undecided. Especially with the LFS issues I was having LFS not detecting files that should have been included. |
|
After I found a clone of 2.2.0 took a long time was curious about clone speed of 2.1.0 vs 2.2.0. You can see my results below. One big thing to note is that cloning the dependencies is quicker from Github than from LFS, and results in considerably less data transfer. I checked, and the unzipped dependencies in 2.1.0 totals up to 2.2G, so while slightly less than what 2.2.0 was pulling, not enough to invalidate any conclusions drawn. After this, I think that going back to non-lfs is the right way to go. Perhaps it would be better to compress each large file separately though, rather than grouped in archives, to help reduce churn when dependencies are updated. I have had a play with the deps in the repo, and by compressing a couple of the larger ones, the clone times are much better than 2.1.0. This is without adding anything back in the history, so this clone time will increase over time, but that will only happen when deps are updated which shouldn't happen often. 2.2.0: 2.2.0 with deps moved in tree (with some in archives to reduce size) 2.1.0: 2.1.0 shallow (useful for ci): 2.2.0 with deps in tree - shallow: |
|
@Julusian I've nugetified the windows build so the repo should be significantly smaller now. |
|
@Julusian You might want to look into nugetifying CEF (there are nuget packages for it). |
|
Yep, that clone time is signifantly quicker now: shallow: |
|
@ronag I can take a look at doing that at some point. |
|
I'll deal with the linux deps....just recovering from a really nasty flu bug! |
|
@Julusian once we get out all the large deps we can remove LFS. Until then it will have to stay. |
To properly clean up the root git. Applies to:
remotes/origin/2.1.0
remotes/origin/avoid_readback
remotes/origin/flash_monitor
remotes/origin/master
remotes/origin/matrix-based-transforms
@dotarmin maybe something you could do?
See, https://github.com/CasparCG/Server/blob/2.2.0/.gitattributes and git lfs migrate with rewrite history.
The text was updated successfully, but these errors were encountered: