Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Backport 2.23-maintenance] Increase download buffer size and improve tarball import logging #11214

Merged
merged 4 commits into from
Jul 29, 2024
Merged

[Backport 2.23-maintenance] Increase download buffer size and improve tarball import logging #11214

merged 4 commits into from
Jul 29, 2024

Commits on Jul 29, 2024

  1. Log download durations 

    (cherry picked from commit caf4e98)
    @edolstra @github-actions
    edolstra authored and github-actions[bot] committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    7698e53 View commit details
    Browse the repository at this point in the history

  2. Add 'download-buffer-size' setting 

    We are piping curl downloads into `unpackTarfileToSink()`, but the
latter is typically slower than the former if you're on a fast
connection. So the download could appear unnecessarily slow. (There is
even a risk that if the Git import is *really* slow for whatever
reason, the TCP connection could time out.)

So let's make the download buffer bigger by default - 64 MiB is big
enough for the Nixpkgs tarball. Perhaps in the future, we could have
an unlimited buffer that spills data to disk beyond a certain
threshold, but that's probably overkill.

(cherry picked from commit 8ffea0a)
    @edolstra @github-actions
    edolstra authored and github-actions[bot] committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    7804645 View commit details
    Browse the repository at this point in the history

  3. Warn if the download buffer is full 

    (cherry picked from commit f6a9a71)
    @edolstra @github-actions
    edolstra authored and github-actions[bot] committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    c3e907a View commit details
    Browse the repository at this point in the history

  4. Show when we're unpacking an archive into the Git cache 

    This happens in parallel with the download (which starts later), so
you only see this message when the download has finished but the
import hasn't.

(cherry picked from commit 01839b5)
    @edolstra @github-actions
    edolstra authored and github-actions[bot] committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    dd0412d View commit details
    Browse the repository at this point in the history