-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: prevent excessive repo-server disk usage for large repos (#8845) #8897
fix: prevent excessive repo-server disk usage for large repos (#8845) #8897
Conversation
Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>
Codecov Report
@@ Coverage Diff @@
## master #8897 +/- ##
==========================================
+ Coverage 43.10% 43.13% +0.03%
==========================================
Files 186 186
Lines 23359 23359
==========================================
+ Hits 10069 10077 +8
+ Misses 11849 11841 -8
Partials 1441 1441
Continue to review full report at Codecov.
|
Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>
Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>
Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>
9515384
to
03d93c0
Compare
Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>
0d0208b
to
8ea0cba
Compare
Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>
Signed-off-by: Michael Crenshaw <michael@crenshaw.dev>
Note: planning to run this internally for a week to see if it solves an internal problem we faced. This is sensitive code with a lot of edge cases, so we should be careful. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks a lot @crenshaw-dev ! Internal testing was successful. LGTM
…oj#8845) (argoproj#8897) fix: prevent excessive repo-server disk usage for large repos (argoproj#8845) (argoproj#8897) Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> Signed-off-by: wojtekidd <wojtek.cichon@protonmail.com>
* fix(ui): Applications page incorrectly resets to tiles view. Fixes argoproj#8702 (argoproj#8718) Signed-off-by: Yuan Tang <terrytangyuan@gmail.com> * fix: correct jsonnet paths resolution (argoproj#8721) Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com> * chore: Bump stable version of application set addon (argoproj#8744) Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com> * fix: Retry checkbox unchecked unexpectedly; Sync up with YAML (argoproj#8682) (argoproj#8720) Signed-off-by: Keith Chong <kykchong@redhat.com> * Bump version to 2.3.1 * Bump version to 2.3.1 * Merge pull request from GHSA-2f5v-8r3f-8pww * fix: application resource APIs must enforce project restrictions Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com> * Fix unit tests Signed-off-by: jannfis <jann@mistrust.net> Co-authored-by: jannfis <jann@mistrust.net> * chore: remove lint-docs CI task (argoproj#8722) (argoproj#8858) * chore: remove lint-docs CI task Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com> * chore: remove not longer necessary url-allow-list Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com> Co-authored-by: Alexander Matyushentsev <AMatyushentsev@gmail.com> * chore: fix imports (argoproj#8859) Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> * Bump version to 2.3.2 * Bump version to 2.3.2 * fix: Set QPS and burst rate for resource ops client (argoproj#8915) * fix: Set QPS and burst rate for resource ops client Signed-off-by: jannfis <jann@mistrust.net> * fix: prevent excessive repo-server disk usage for large repos (argoproj#8845) (argoproj#8897) fix: prevent excessive repo-server disk usage for large repos (argoproj#8845) (argoproj#8897) Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> * fix: bump gitops engine version to v0.6.2 Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com> * docs: update v2.4+ roadmap items (argoproj#8593) Signed-off-by: ishitasequeira <isequeir@redhat.com> * docs: reflect v2.3 release changes in roadmap.md (argoproj#8747) docs: reflect v2.3 release changes in roadmap.md (argoproj#8747) Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com> * Bump version to 2.3.3 * Bump version to 2.3.3 * Add manifest for OnePipeline (cherry picked from commit 15aa211080ef020e6a2ceaee9b845eb3259db237) * Load additional resource overrides from dedicated ConfigMap * Run unit tests (cherry picked from commit 7605d5b0e2e816bb1cf9a29c5910c0fd511900c2) * Install and config Git for unit tests (cherry picked from commit 05dda11f6adf3191712b4598c8d55fe8ca1647a6) * Add doc for changes * feat: Argo CD CI pipeline changes (argoproj#4) * updated cicd image * upadted registry region * updated one.pipeline.yaml to use the latest scripts * updated makefile to add required targets * feat: Argo CD v2.3.2 (argoproj#5) * fix(ui): Applications page incorrectly resets to tiles view. Fixes argoproj#8702 (argoproj#8718) Signed-off-by: Yuan Tang <terrytangyuan@gmail.com> * fix: correct jsonnet paths resolution (argoproj#8721) Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com> * fix: Retry checkbox unchecked unexpectedly; Sync up with YAML (argoproj#8682) (argoproj#8720) Signed-off-by: Keith Chong <kykchong@redhat.com> * chore: Bump stable version of application set addon (argoproj#8744) Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com> * Bump version to 2.3.1 * Bump version to 2.3.1 * Merge pull request from GHSA-2f5v-8r3f-8pww * fix: application resource APIs must enforce project restrictions Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com> * Fix unit tests Signed-off-by: jannfis <jann@mistrust.net> Co-authored-by: jannfis <jann@mistrust.net> * chore: remove lint-docs CI task (argoproj#8722) (argoproj#8858) * chore: remove lint-docs CI task Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com> * chore: remove not longer necessary url-allow-list Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com> Co-authored-by: Alexander Matyushentsev <AMatyushentsev@gmail.com> * chore: fix imports (argoproj#8859) Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> * Bump version to 2.3.2 * Bump version to 2.3.2 * feat: Updated CHANGES.md Co-authored-by: Yuan Tang <terrytangyuan@gmail.com> Co-authored-by: Alexander Matyushentsev <AMatyushentsev@gmail.com> Co-authored-by: Keith Chong <kykchong@redhat.com> Co-authored-by: argo-bot <argoproj@gmail.com> Co-authored-by: jannfis <jann@mistrust.net> Co-authored-by: Michael Crenshaw <michael@crenshaw.dev> * feat: Add .whitesource configuration file (argoproj#6) Co-authored-by: whitesource-ets[bot] <328400+whitesource-ets[bot]@users.noreply.github.ibm.com> * docs: CHANGES.md Updated the CHANGES.md file to include updated information about changes made. Contributes to: automation-saas/native-AWS#1413 Signed-off-by: Sujeily Fonseca <sujeily.fonseca@ibm.com> Co-authored-by: Yuan Tang <terrytangyuan@gmail.com> Co-authored-by: Alexander Matyushentsev <AMatyushentsev@gmail.com> Co-authored-by: Keith Chong <kykchong@redhat.com> Co-authored-by: argo-bot <argoproj@gmail.com> Co-authored-by: jannfis <jann@mistrust.net> Co-authored-by: Michael Crenshaw <michael@crenshaw.dev> Co-authored-by: Ishita Sequeira <46771830+ishitasequeira@users.noreply.github.com> Co-authored-by: Nikolas McGovern <Nikolas.McGovern@ibm.com> Co-authored-by: Rahul Mourya <Rahul.Mourya2@ibm.com> Co-authored-by: whitesource-ets[bot] <328400+whitesource-ets[bot]@users.noreply.github.ibm.com>
* fix(ui): Applications page incorrectly resets to tiles view. Fixes argoproj#8702 (argoproj#8718) Signed-off-by: Yuan Tang <terrytangyuan@gmail.com> * fix: correct jsonnet paths resolution (argoproj#8721) Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com> * chore: Bump stable version of application set addon (argoproj#8744) Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com> * fix: Retry checkbox unchecked unexpectedly; Sync up with YAML (argoproj#8682) (argoproj#8720) Signed-off-by: Keith Chong <kykchong@redhat.com> * Bump version to 2.3.1 * Bump version to 2.3.1 * Merge pull request from GHSA-2f5v-8r3f-8pww * fix: application resource APIs must enforce project restrictions Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com> * Fix unit tests Signed-off-by: jannfis <jann@mistrust.net> Co-authored-by: jannfis <jann@mistrust.net> * chore: remove lint-docs CI task (argoproj#8722) (argoproj#8858) * chore: remove lint-docs CI task Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com> * chore: remove not longer necessary url-allow-list Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com> Co-authored-by: Alexander Matyushentsev <AMatyushentsev@gmail.com> * chore: fix imports (argoproj#8859) Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> * Bump version to 2.3.2 * Bump version to 2.3.2 * fix: Set QPS and burst rate for resource ops client (argoproj#8915) * fix: Set QPS and burst rate for resource ops client Signed-off-by: jannfis <jann@mistrust.net> * fix: prevent excessive repo-server disk usage for large repos (argoproj#8845) (argoproj#8897) fix: prevent excessive repo-server disk usage for large repos (argoproj#8845) (argoproj#8897) Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> * fix: bump gitops engine version to v0.6.2 Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com> * docs: update v2.4+ roadmap items (argoproj#8593) Signed-off-by: ishitasequeira <isequeir@redhat.com> * docs: reflect v2.3 release changes in roadmap.md (argoproj#8747) docs: reflect v2.3 release changes in roadmap.md (argoproj#8747) Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com> * Bump version to 2.3.3 * Bump version to 2.3.3 * fix: Fix docs build error (argoproj#8895) * work with specific jinja version Signed-off-by: pashavictorovich <pavel@codefresh.io> * fix: fix broken monaco editor collapse icons (argoproj#8709) Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com> * chore: upgrade to go 1.17.8 (argoproj#8866) (argoproj#9004) * chore: upgrade to go 1.17.8 Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> * chore: use 1.17 so it's always latest in the series Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> * fix: allow cli/ui to follow logs (argoproj#8987) (argoproj#9065) Signed-off-by: Daniel Helfand <helfand.4@gmail.com> * Merge pull request from GHSA-xmg8-99r8-jc2j Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> Co-authored-by: Michael Crenshaw <michael@crenshaw.dev> * Merge pull request from GHSA-6gcg-hp2x-q54h * fix: do not allow symlinks from directory-type applications Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> * chore: add new util file Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> * chore: lint Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> * chore: use t.TempDir for simpler tests Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> * address comments Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> * Merge pull request from GHSA-r642-gv9p-2wjj Signed-off-by: jannfis <jann@mistrust.net> Co-authored-by: Michael Crenshaw <michael@crenshaw.dev> Co-authored-by: Michael Crenshaw <michael@crenshaw.dev> * Bump version to 2.3.4 * Bump version to 2.3.4 * test: fix ErrorContains (argoproj#9445) Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> * fix: missing Helm params (argoproj#9565) (argoproj#9566) * fix: missing Helm params Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> * use absolute paths, fix tests Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> * fix race in test Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> * chore: upgrade golangci-lint to v1.46.2 (argoproj#9448) * chore: upgrade golangci-lint to v1.46.2 Because: * Installation of golangci-lint v1.45.2 is currently broken and fails silently due to a redacted dependency (blizzy78/varnamelen#13) This commit: * Upgrades golangci-lint to v1.46.2 Signed-off-by: Tommaso Sardelli <lacapannadelloziotom@gmail.com> * fix: lint Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> * fix: lint Signed-off-by: Tommaso Sardelli <lacapannadelloziotom@gmail.com> Co-authored-by: Michael Crenshaw <michael@crenshaw.dev> Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> * fix: test race (argoproj#9469) Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> * chore: lint issues Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> * chore: update golangci-lint (argoproj#8988) * chore: update golangci-lint Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> * chore: remove obsolete repo-server unit test (argoproj#9559) Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com> * chore: Make unit tests run on platforms other than amd64 (argoproj#8995) Signed-off-by: jannfis <jann@mistrust.net> Co-authored-by: Michael Crenshaw <michael@crenshaw.dev> Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> * chore: eliminate go-mpatch dependency (argoproj#9045) * chore: eliminate go-mpatch dependency Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> * chore: abstract out resource list function Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> * chore: don't exit the program in anything but the main function Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> * chore: better error messages Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> * chore: better error messages Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> * test: directory app manifest generation (argoproj#9503) * test: directory app manifest generation Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> * git doesn't support empty dirs Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> * Merge pull request from GHSA-h4w9-6x78-8vrj Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> * Merge pull request from GHSA-2m7h-86qq-fp4v Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> fix references Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> use long enough state param for oauth2 Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> typo Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> more entropy Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> fix test Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> * Merge pull request from GHSA-q4w5-4gq2-98vm Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> * Merge pull request from GHSA-jhqp-vf4w-rpwq Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> defer instead of multiple close calls Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> oops Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> don't count jsonnet against max Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> fix codegen Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> add caveat about 300x ratio Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> fix versions Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> fix tests/lint Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> * chore: fix docs gen Signed-off-by: Michael Crenshaw <michael@crenshaw.dev> * Bump version to 2.3.5 * Bump version to 2.3.5 * docs: Changes for v2.3.5 Documented key decision factors to use Argo CD v2.3.5. Contributes to: automation-saas/automation-saas/native-AWS#1972 Signed-off-by: Sujeily Fonseca <sujeily.fonseca@ibm.com> Co-authored-by: Yuan Tang <terrytangyuan@gmail.com> Co-authored-by: Alexander Matyushentsev <AMatyushentsev@gmail.com> Co-authored-by: Keith Chong <kykchong@redhat.com> Co-authored-by: argo-bot <argoproj@gmail.com> Co-authored-by: jannfis <jann@mistrust.net> Co-authored-by: Michael Crenshaw <michael@crenshaw.dev> Co-authored-by: Ishita Sequeira <46771830+ishitasequeira@users.noreply.github.com> Co-authored-by: pasha-codefresh <pavel@codefresh.io> Co-authored-by: Daniel Helfand <helfand.4@gmail.com> Co-authored-by: Tommaso Sardelli <lacapannadelloziotom@gmail.com>
Fixes #8845
Fetching specific commits allowed us to get commits pointing to non-standard refspecs (like refs/pulls/*/head).
But it costs a lot more disk space (demonstrated in the issue) and slows down app refreshes after commits.
This change defaults to the old behavior (fetch standard refspecs, checkout commit) and then falls back to fetching a specific commit if the checkout fails.
So if someone has an Application pointing to
HEAD
of a repo with 100k commits, this will efficiently fetch each commit without filling up the disk.And if someone has an application pointing to
refs/pulls/123/head
, this will fail on the initial fetch and then fall back to fetching the specific commit. It's higher latency, but I think it's justified to avoid filling up the disk.I've also confirmed there is no regression by testing this app: