-
Notifications
You must be signed in to change notification settings - Fork 527
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support WithProgress for remote Writes #967
Conversation
2809cb0
to
bb071df
Compare
Codecov Report
@@ Coverage Diff @@
## main #967 +/- ##
==========================================
- Coverage 75.19% 74.68% -0.51%
==========================================
Files 107 107
Lines 4837 4994 +157
==========================================
+ Hits 3637 3730 +93
- Misses 669 710 +41
- Partials 531 554 +23
Continue to review full report at Codecov.
|
- supported in WriteLayer, Write, WriteIndex, MultiWrite - chan is closed when write completes - an error is sent along the chan if there are any non-temporary errors uploading - if a layer already exists or is mounted, progress updates immediately to account for that - if layer upload fails and is retried, progress goes backward and goes back up
I was just hunting for something like this (especially after having put together the tarball one at #675 :-) . I was stumped, as I did not see it in godoc or pkg.go.dev, and was going to start trying to think about doing it, when I found this PR. Apparently godoc only takes the latest semver release (if one exists). Does this send v1.Update on the channel: type Update struct {
Total int64
Complete int64
Error error
} so it has just those fields? Does it calculate all of the bytes before sending, across all of the manifests and layers (e.g. I also saw discussion about how docker does its I came across this with a |
I'm actually planning on doing a ggcr release today! In the meantime, https://pkg.go.dev/github.com/google/go-containerregistry@main/pkg/v1/remote should work.
That's correct. It sends
I think that's something we could investigate and implement, yes. We could accomplish this by adding fields to the |
OK, I will start with that. I have been very bogged down (in a positive sense) with various deliverables the last two weeks, otherwise I would take a stab at it. If I can at some point, I will. |
This question is slightly off topic, but triggered here. How do I get a f, err := os.Open(somefile)
defer f.Close()
layer = stream.NewLayer(f) But that then gives you a |
https://pkg.go.dev/github.com/google/go-containerregistry@main/pkg/v1/tarball#LayerFromFile is the best we have right now. Ideally updates would be able to handle streaming layers by increasing the total as bytes are streamed, but this hasn't been done yet. |
True, but in this case, I actually know the total size (it is just a file), so no reason not to have proper updates. Would that actually work? Or only with a tarball? |
Would it make sense to have some |
That certainly would make it easier to get At some point, if/when we decide to have per-layer progress reports, we will need the size for each layer as we push it up, won't we? This problem becomes the same there. Although, as I think about it, when we push an |
A Care is taken so that mutations are lazily applied, so you can The use case for |
So thinking about the bigger picture, we either need a concept of presetting size and digest on What do |
Really the only time you'd have to deal with them is when you
|
This is a slight oversimplification. If you've accessed a layer via If you access a layer via @deitch do you have this layer inside a layout? Or is it just a file? Where do you have this metadata stored? We'd probably want to implement something that knows how to read/parse this metadata and can return a layer with all of this pre-populated, rather than you having to supply each piece individually. |
Thanks for this factcheck, you're right. In either case, |
Some maybe useful example of similar situations... cosign uploading signatures:It's a very basic static It does this itself because in this case we don't want to compress the layer, which rules_docker translating from bazel's representation to v1.Layer:
crane uploading a layer from a file:go-containerregistry/pkg/crane/append.go Lines 42 to 52 in 37467b5
If it's a streaming thing, we use |
Ah all of that makes sense and clarifies a lot. I didn't think about the implications when I had a file, so I just wrapped it all with a
Both use cases, actually. I am guessing that you are hinting that in a layout, I can just use the What about for just a file? |
As for the progress, I really would like to figure out a way to get |
Fixes #952
Shoutout to @DennisDenuto who paired with me on this 🎉
remote.WriteLayer
,remote.Write
,remote.WriteIndex
,remote.MultiWrite
d7fe938 demonstrates how easy it is to consume this channel, by hacking in an upload progress bar for
crane cp
(not part of this PR)Fixes #852 as well as a bonus ⏩