Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Paramfetch with aria2 support #32

Merged
merged 5 commits into from
Jun 6, 2024
Merged

Paramfetch with aria2 support #32

merged 5 commits into from
Jun 6, 2024

Conversation

magik6k
Copy link
Collaborator

@magik6k magik6k commented Jun 4, 2024

This is a copy of https://github.com/filecoin-project/go-paramfetch/blob/master/paramfetch.go with:

  • Heavily opinionated support for aria2
  • Parallel fetch support

In the future we can think of adding support for p2p fetches from the cluster, but for now this makes my paramfetch operations 7-10x faster

Copy link
Contributor

@LexLuthr LexLuthr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should update our dependencies to include aria2c here on Linux and MacOS systems.

@magik6k
Copy link
Collaborator Author

magik6k commented Jun 4, 2024

Yes, but not that it is technically optional in this PR, so nothing breaks if you don't have it, just the initial paramfetch is not as fast

@snadrus
Copy link
Contributor

snadrus commented Jun 4, 2024

Please add:
, aria2
to
File: apt/DEBIAN/control
Line: starts with "Depends:"

I've verified that name in the past 4 Ubuntu versions in the Universe repo.

@magik6k magik6k requested review from LexLuthr and snadrus June 6, 2024 11:28
Copy link
Contributor

@LexLuthr LexLuthr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested. Looks good.

Copy link
Contributor

@LexLuthr LexLuthr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just 1 one small change to speed up download

lib/fastparamfetch/paramfetch.go Outdated Show resolved Hide resolved
Co-authored-by: LexLuthr <88259624+LexLuthr@users.noreply.github.com>
@magik6k magik6k merged commit a9eac97 into main Jun 6, 2024
5 of 6 checks passed
@magik6k magik6k deleted the feat/aria-paramfetch branch June 6, 2024 12:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants