Parallel layer upload for s3 cache #5270
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bpaquet
force-pushed
the
s3_parallel_upload
branch
2 times, most recently
from
7756b08
to
a7a00b8
Compare
tonistiigi
reviewed
Aug 27, 2024
README.md
Outdated
| @@ -578,6 +578,7 @@ Other options are: | |||
| * Multiple manifest names can be specified at the same time, separated by `;`. The standard use case is to use the git sha1 as name, and the branch name as duplicate, and load both with 2 `import-cache` commands. | |||
| * `ignore-error=<false|true>`: specify if error is ignored in case cache export fails (default: `false`) | |||
| * `touch_refresh=24h`: Instead of being uploaded again when not changed, blobs files will be "touched" on s3 every `touch_refresh`, default is 24h. Due to this, an expiration policy can be set on the S3 bucket to cleanup useless files automatically. Manifests files are systematically rewritten, there is no need to touch them. | |||
| * `upload_parallelism=10`: This parameter changes the number of layers uploaded to s3 in parallel. Each individual layer is uploaded with 5 threads, using the Upload manager provided by the AWS SDK. | |||
There was a problem hiding this comment.
Isn't 10 maybe bit too big for default? In registry we use 4 (+1 for meta-requests) as a default https://github.com/moby/buildkit/blob/master/util/resolver/limited/group.go#L23
Signed-off-by: Bertrand Paquet <bertrand.paquet@gmail.com>
bpaquet
force-pushed
the
s3_parallel_upload
branch
from
a7a00b8
to
22f6b3e
Compare
tonistiigi
approved these changes
Aug 28, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR allows to use multiple go routines to upload layers to S3 in parallel.
Parallelism is controlled by
upload_parallelismIndividually, each layer is already send in parallel, using the standard Upload Manager provided by the S3 SDK.
Inspired by 6c439bd.