container: Work on new skopeo backend #18
Comments
|
I think as a PoC, we can make a new sub-project like The ideal then would be to fold that code back into |
cgwalters
added a commit
to cgwalters/ostree-rs-ext
that referenced
this issue
Sep 20, 2021
The use of `docker-archive:` originally came from ostreedev#18 but it turns out I was wrong, skopeo ends up staging everything anyways. In practice, what we really want to do here is land custom code in skopeo to do what we need, instead of treating this as a `copy` operation, because it isn't. Relatedly, I also noticed in our `docker-archive:` parsing that we weren't verifying the relationship of the layer symlink to the sha256 digest; we were only verifying the *presence* of the symlink. In practice, skopeo will already be doing the layer verification for us, but this mechanism is clearer. Prep for derivation.
|
Some extremely hacky WIP on this in https://github.com/cgwalters/ostree-rs-ext/commits/import-via-proxy |
|
OK I thought this didn't work, but it turns out it's racy. The race appears to be somewhere in the Rust async code around buffering, but I haven't figured out where yet. Debugging this has been extremely painful. I'm getting burned by lack of unit tests at different places in the pipeline, so working on that now. |
cgwalters
added a commit
to cgwalters/skopeo
that referenced
this issue
Oct 6, 2021
This imports the code from https://github.com/cgwalters/container-image-proxy First, assume one is operating on a codebase that isn't Go, but wants to interact with container images - we can't just include the Go containers/image library. The primary intended use case of this is for things like [ostree-containers](ostreedev/ostree-rs-ext#18) where we're using container images to encapsulate host operating system updates, but we don't want to involve the [containers/image](github.com/containers/image/) storage layer. Vendoring the containers/image stack in another project is a large lift; the stripped binary for this proxy standalone weighs in at 16M (I'm sure the lack of LTO and the overall simplicity of the Go compiler is a large factor). Anyways, I'd like to avoid shipping another copy. This command is marked as experimental, and hidden. The goal is just to use it from the ostree stack for now, ideally shipping at least in CentOS 9 Stream relatively soon. We can (and IMO should) change and improve it later. A lot more discussion in cgwalters/container-image-proxy#1
cgwalters
added a commit
to cgwalters/skopeo
that referenced
this issue
Oct 6, 2021
This imports the code from https://github.com/cgwalters/container-image-proxy First, assume one is operating on a codebase that isn't Go, but wants to interact with container images - we can't just include the Go containers/image library. The primary intended use case of this is for things like [ostree-containers](ostreedev/ostree-rs-ext#18) where we're using container images to encapsulate host operating system updates, but we don't want to involve the [containers/image](github.com/containers/image/) storage layer. Vendoring the containers/image stack in another project is a large lift; the stripped binary for this proxy standalone weighs in at 16M (I'm sure the lack of LTO and the overall simplicity of the Go compiler is a large factor). Anyways, I'd like to avoid shipping another copy. This command is marked as experimental, and hidden. The goal is just to use it from the ostree stack for now, ideally shipping at least in CentOS 9 Stream relatively soon. We can (and IMO should) change and improve it later. A lot more discussion in cgwalters/container-image-proxy#1
cgwalters
added a commit
to cgwalters/skopeo
that referenced
this issue
Oct 9, 2021
This imports the code from https://github.com/cgwalters/container-image-proxy First, assume one is operating on a codebase that isn't Go, but wants to interact with container images - we can't just include the Go containers/image library. The primary intended use case of this is for things like [ostree-containers](ostreedev/ostree-rs-ext#18) where we're using container images to encapsulate host operating system updates, but we don't want to involve the [containers/image](github.com/containers/image/) storage layer. Vendoring the containers/image stack in another project is a large lift; the stripped binary for this proxy standalone weighs in at 16M (I'm sure the lack of LTO and the overall simplicity of the Go compiler is a large factor). Anyways, I'd like to avoid shipping another copy. This command is marked as experimental, and hidden. The goal is just to use it from the ostree stack for now, ideally shipping at least in CentOS 9 Stream relatively soon. We can (and IMO should) change and improve it later. A lot more discussion in cgwalters/container-image-proxy#1
cgwalters
added a commit
to cgwalters/skopeo
that referenced
this issue
Oct 12, 2021
This imports the code from https://github.com/cgwalters/container-image-proxy First, assume one is operating on a codebase that isn't Go, but wants to interact with container images - we can't just include the Go containers/image library. The primary intended use case of this is for things like [ostree-containers](ostreedev/ostree-rs-ext#18) where we're using container images to encapsulate host operating system updates, but we don't want to involve the [containers/image](github.com/containers/image/) storage layer. Vendoring the containers/image stack in another project is a large lift; the stripped binary for this proxy standalone weighs in at 16M (I'm sure the lack of LTO and the overall simplicity of the Go compiler is a large factor). Anyways, I'd like to avoid shipping another copy. This command is marked as experimental, and hidden. The goal is just to use it from the ostree stack for now, ideally shipping at least in CentOS 9 Stream relatively soon. We can (and IMO should) change and improve it later. A lot more discussion in cgwalters/container-image-proxy#1
cgwalters
added a commit
to cgwalters/skopeo
that referenced
this issue
Oct 12, 2021
This imports the code from https://github.com/cgwalters/container-image-proxy First, assume one is operating on a codebase that isn't Go, but wants to interact with container images - we can't just include the Go containers/image library. The primary intended use case of this is for things like [ostree-containers](ostreedev/ostree-rs-ext#18) where we're using container images to encapsulate host operating system updates, but we don't want to involve the [containers/image](github.com/containers/image/) storage layer. Vendoring the containers/image stack in another project is a large lift; the stripped binary for this proxy standalone weighs in at 16M (I'm sure the lack of LTO and the overall simplicity of the Go compiler is a large factor). Anyways, I'd like to avoid shipping another copy. This command is marked as experimental, and hidden. The goal is just to use it from the ostree stack for now, ideally shipping at least in CentOS 9 Stream relatively soon. We can (and IMO should) change and improve it later. A lot more discussion in cgwalters/container-image-proxy#1
cgwalters
added a commit
to cgwalters/skopeo
that referenced
this issue
Oct 12, 2021
This imports the code from https://github.com/cgwalters/container-image-proxy First, assume one is operating on a codebase that isn't Go, but wants to interact with container images - we can't just include the Go containers/image library. The primary intended use case of this is for things like [ostree-containers](ostreedev/ostree-rs-ext#18) where we're using container images to encapsulate host operating system updates, but we don't want to involve the [containers/image](github.com/containers/image/) storage layer. Vendoring the containers/image stack in another project is a large lift; the stripped binary for this proxy standalone weighs in at 16M (I'm sure the lack of LTO and the overall simplicity of the Go compiler is a large factor). Anyways, I'd like to avoid shipping another copy. This command is marked as experimental, and hidden. The goal is just to use it from the ostree stack for now, ideally shipping at least in CentOS 9 Stream relatively soon. We can (and IMO should) change and improve it later. A lot more discussion in cgwalters/container-image-proxy#1
cgwalters
added a commit
to cgwalters/skopeo
that referenced
this issue
Oct 12, 2021
This imports the code from https://github.com/cgwalters/container-image-proxy First, assume one is operating on a codebase that isn't Go, but wants to interact with container images - we can't just include the Go containers/image library. The primary intended use case of this is for things like [ostree-containers](ostreedev/ostree-rs-ext#18) where we're using container images to encapsulate host operating system updates, but we don't want to involve the [containers/image](github.com/containers/image/) storage layer. Vendoring the containers/image stack in another project is a large lift; the stripped binary for this proxy standalone weighs in at 16M (I'm sure the lack of LTO and the overall simplicity of the Go compiler is a large factor). Anyways, I'd like to avoid shipping another copy. This command is marked as experimental, and hidden. The goal is just to use it from the ostree stack for now, ideally shipping at least in CentOS 9 Stream relatively soon. We can (and IMO should) change and improve it later. A lot more discussion in cgwalters/container-image-proxy#1
cgwalters
added a commit
to cgwalters/skopeo
that referenced
this issue
Oct 12, 2021
This imports the code from https://github.com/cgwalters/container-image-proxy First, assume one is operating on a codebase that isn't Go, but wants to interact with container images - we can't just include the Go containers/image library. The primary intended use case of this is for things like [ostree-containers](ostreedev/ostree-rs-ext#18) where we're using container images to encapsulate host operating system updates, but we don't want to involve the [containers/image](github.com/containers/image/) storage layer. Vendoring the containers/image stack in another project is a large lift; the stripped binary for this proxy standalone weighs in at 16M (I'm sure the lack of LTO and the overall simplicity of the Go compiler is a large factor). Anyways, I'd like to avoid shipping another copy. This command is marked as experimental, and hidden. The goal is just to use it from the ostree stack for now, ideally shipping at least in CentOS 9 Stream relatively soon. We can (and IMO should) change and improve it later. A lot more discussion in cgwalters/container-image-proxy#1
cgwalters
added a commit
to cgwalters/skopeo
that referenced
this issue
Oct 13, 2021
This imports the code from https://github.com/cgwalters/container-image-proxy First, assume one is operating on a codebase that isn't Go, but wants to interact with container images - we can't just include the Go containers/image library. The primary intended use case of this is for things like [ostree-containers](ostreedev/ostree-rs-ext#18) where we're using container images to encapsulate host operating system updates, but we don't want to involve the [containers/image](github.com/containers/image/) storage layer. Vendoring the containers/image stack in another project is a large lift; the stripped binary for this proxy standalone weighs in at 16M (I'm sure the lack of LTO and the overall simplicity of the Go compiler is a large factor). Anyways, I'd like to avoid shipping another copy. This command is marked as experimental, and hidden. The goal is just to use it from the ostree stack for now, ideally shipping at least in CentOS 9 Stream relatively soon. We can (and IMO should) change and improve it later. A lot more discussion in cgwalters/container-image-proxy#1
cgwalters
added a commit
to cgwalters/skopeo
that referenced
this issue
Oct 13, 2021
This imports the code from https://github.com/cgwalters/container-image-proxy First, assume one is operating on a codebase that isn't Go, but wants to interact with container images - we can't just include the Go containers/image library. The primary intended use case of this is for things like [ostree-containers](ostreedev/ostree-rs-ext#18) where we're using container images to encapsulate host operating system updates, but we don't want to involve the [containers/image](github.com/containers/image/) storage layer. Vendoring the containers/image stack in another project is a large lift; the stripped binary for this proxy standalone weighs in at 16M (I'm sure the lack of LTO and the overall simplicity of the Go compiler is a large factor). Anyways, I'd like to avoid shipping another copy. This command is marked as experimental, and hidden. The goal is just to use it from the ostree stack for now, ideally shipping at least in CentOS 9 Stream relatively soon. We can (and IMO should) change and improve it later. A lot more discussion in cgwalters/container-image-proxy#1
cgwalters
added a commit
to cgwalters/skopeo
that referenced
this issue
Oct 13, 2021
This imports the code from https://github.com/cgwalters/container-image-proxy First, assume one is operating on a codebase that isn't Go, but wants to interact with container images - we can't just include the Go containers/image library. The primary intended use case of this is for things like [ostree-containers](ostreedev/ostree-rs-ext#18) where we're using container images to encapsulate host operating system updates, but we don't want to involve the [containers/image](github.com/containers/image/) storage layer. Vendoring the containers/image stack in another project is a large lift; the stripped binary for this proxy standalone weighs in at 16M (I'm sure the lack of LTO and the overall simplicity of the Go compiler is a large factor). Anyways, I'd like to avoid shipping another copy. This command is marked as experimental, and hidden. The goal is just to use it from the ostree stack for now, ideally shipping at least in CentOS 9 Stream relatively soon. We can (and IMO should) change and improve it later. A lot more discussion in cgwalters/container-image-proxy#1
cgwalters
added a commit
to cgwalters/skopeo
that referenced
this issue
Oct 13, 2021
This imports the code from https://github.com/cgwalters/container-image-proxy First, assume one is operating on a codebase that isn't Go, but wants to interact with container images - we can't just include the Go containers/image library. The primary intended use case of this is for things like [ostree-containers](ostreedev/ostree-rs-ext#18) where we're using container images to encapsulate host operating system updates, but we don't want to involve the [containers/image](github.com/containers/image/) storage layer. Vendoring the containers/image stack in another project is a large lift; the stripped binary for this proxy standalone weighs in at 16M (I'm sure the lack of LTO and the overall simplicity of the Go compiler is a large factor). Anyways, I'd like to avoid shipping another copy. This command is marked as experimental, and hidden. The goal is just to use it from the ostree stack for now, ideally shipping at least in CentOS 9 Stream relatively soon. We can (and IMO should) change and improve it later. A lot more discussion in cgwalters/container-image-proxy#1
cgwalters
added a commit
to cgwalters/skopeo
that referenced
this issue
Oct 13, 2021
This imports the code from https://github.com/cgwalters/container-image-proxy First, assume one is operating on a codebase that isn't Go, but wants to interact with container images - we can't just include the Go containers/image library. The primary intended use case of this is for things like [ostree-containers](ostreedev/ostree-rs-ext#18) where we're using container images to encapsulate host operating system updates, but we don't want to involve the [containers/image](github.com/containers/image/) storage layer. Vendoring the containers/image stack in another project is a large lift; the stripped binary for this proxy standalone weighs in at 16M (I'm sure the lack of LTO and the overall simplicity of the Go compiler is a large factor). Anyways, I'd like to avoid shipping another copy. This command is marked as experimental, and hidden. The goal is just to use it from the ostree stack for now, ideally shipping at least in CentOS 9 Stream relatively soon. We can (and IMO should) change and improve it later. A lot more discussion in cgwalters/container-image-proxy#1
cgwalters
added a commit
to cgwalters/skopeo
that referenced
this issue
Oct 14, 2021
This imports the code from https://github.com/cgwalters/container-image-proxy First, assume one is operating on a codebase that isn't Go, but wants to interact with container images - we can't just include the Go containers/image library. The primary intended use case of this is for things like [ostree-containers](ostreedev/ostree-rs-ext#18) where we're using container images to encapsulate host operating system updates, but we don't want to involve the [containers/image](github.com/containers/image/) storage layer. Vendoring the containers/image stack in another project is a large lift; the stripped binary for this proxy standalone weighs in at 16M (I'm sure the lack of LTO and the overall simplicity of the Go compiler is a large factor). Anyways, I'd like to avoid shipping another copy. This command is marked as experimental, and hidden. The goal is just to use it from the ostree stack for now, ideally shipping at least in CentOS 9 Stream relatively soon. We can (and IMO should) change and improve it later. A lot more discussion in cgwalters/container-image-proxy#1
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The way we fork skopeo as a subprocess has a lot of flaws.
See: containers/image#1209
The text was updated successfully, but these errors were encountered: