commit: Try reflinks for local commits by default #3106
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cgwalters
added a commit
to cgwalters/bootc
that referenced
this pull request
Dec 3, 2023
See containers#147 (comment) With this bootc starts to really gain support for a different backend than ostree. Here we basically just fork off `podman pull` to fetch container images into an *alternative root* in `/ostree/container-storage`, (Because otherwise basic things like `podman image prune` would delete the OS image) This is quite distinct from our use of `skopeo` in the ostree-ext project because suddenly now we gain support for things implemented in the containers/storage library like `zstd:chunked` and OCI crypt. *However*...today we still need to generate a final flattened filesystem tree (and an ostree commit) in order to maintain compatibilty with stuff in rpm-ostree. (A corrollary to this is we're not booting into a `podman mount` overlayfs stack) Related to this, we also need to handle SELinux labeling. Hence, we implement "layer squashing", and then do some final "postprocessing" on the resulting image matching the same logic that's done in ostree-ext such as `etc -> usr/etc` and handling `/var`. Note this also really wants ostreedev/ostree#3106 to avoid duplicating disk space.
cgwalters
added a commit
to cgwalters/bootc
that referenced
this pull request
Dec 4, 2023
See containers#147 (comment) With this bootc starts to really gain support for a different backend than ostree. Here we basically just fork off `podman pull` to fetch container images into an *alternative root* in `/ostree/container-storage`, (Because otherwise basic things like `podman image prune` would delete the OS image) This is quite distinct from our use of `skopeo` in the ostree-ext project because suddenly now we gain support for things implemented in the containers/storage library like `zstd:chunked` and OCI crypt. *However*...today we still need to generate a final flattened filesystem tree (and an ostree commit) in order to maintain compatibilty with stuff in rpm-ostree. (A corrollary to this is we're not booting into a `podman mount` overlayfs stack) Related to this, we also need to handle SELinux labeling. Hence, we implement "layer squashing", and then do some final "postprocessing" on the resulting image matching the same logic that's done in ostree-ext such as `etc -> usr/etc` and handling `/var`. Note this also really wants ostreedev/ostree#3106 to avoid duplicating disk space. Signed-off-by: Colin Walters <walters@verbum.org>
cgwalters
added a commit
to cgwalters/bootc
that referenced
this pull request
Dec 4, 2023
See containers#147 (comment) With this bootc starts to really gain support for a different backend than ostree. Here we basically just fork off `podman pull` to fetch container images into an *alternative root* in `/ostree/container-storage`, (Because otherwise basic things like `podman image prune` would delete the OS image) This is quite distinct from our use of `skopeo` in the ostree-ext project because suddenly now we gain support for things implemented in the containers/storage library like `zstd:chunked` and OCI crypt. *However*...today we still need to generate a final flattened filesystem tree (and an ostree commit) in order to maintain compatibilty with stuff in rpm-ostree. (A corrollary to this is we're not booting into a `podman mount` overlayfs stack) Related to this, we also need to handle SELinux labeling. Hence, we implement "layer squashing", and then do some final "postprocessing" on the resulting image matching the same logic that's done in ostree-ext such as `etc -> usr/etc` and handling `/var`. Note this also really wants ostreedev/ostree#3106 to avoid duplicating disk space. Signed-off-by: Colin Walters <walters@verbum.org>
cgwalters
added a commit
to cgwalters/bootc
that referenced
this pull request
Dec 4, 2023
See containers#147 (comment) With this bootc starts to really gain support for a different backend than ostree. Here we basically just fork off `podman pull` to fetch container images into an *alternative root* in `/ostree/container-storage`, (Because otherwise basic things like `podman image prune` would delete the OS image) This is quite distinct from our use of `skopeo` in the ostree-ext project because suddenly now we gain support for things implemented in the containers/storage library like `zstd:chunked` and OCI crypt. *However*...today we still need to generate a final flattened filesystem tree (and an ostree commit) in order to maintain compatibilty with stuff in rpm-ostree. (A corrollary to this is we're not booting into a `podman mount` overlayfs stack) Related to this, we also need to handle SELinux labeling. Hence, we implement "layer squashing", and then do some final "postprocessing" on the resulting image matching the same logic that's done in ostree-ext such as `etc -> usr/etc` and handling `/var`. Note this also really wants ostreedev/ostree#3106 to avoid duplicating disk space. Signed-off-by: Colin Walters <walters@verbum.org>
cgwalters
added a commit
to cgwalters/bootc
that referenced
this pull request
Dec 4, 2023
See containers#147 (comment) With this bootc starts to really gain support for a different backend than ostree. Here we basically just fork off `podman pull` to fetch container images into an *alternative root* in `/ostree/container-storage`, (Because otherwise basic things like `podman image prune` would delete the OS image) This is quite distinct from our use of `skopeo` in the ostree-ext project because suddenly now we gain support for things implemented in the containers/storage library like `zstd:chunked` and OCI crypt. *However*...today we still need to generate a final flattened filesystem tree (and an ostree commit) in order to maintain compatibilty with stuff in rpm-ostree. (A corrollary to this is we're not booting into a `podman mount` overlayfs stack) Related to this, we also need to handle SELinux labeling. Hence, we implement "layer squashing", and then do some final "postprocessing" on the resulting image matching the same logic that's done in ostree-ext such as `etc -> usr/etc` and handling `/var`. Note this also really wants ostreedev/ostree#3106 to avoid duplicating disk space. Signed-off-by: Colin Walters <walters@verbum.org>
cgwalters
added a commit
to cgwalters/bootc
that referenced
this pull request
Dec 4, 2023
See containers#147 (comment) With this bootc starts to really gain support for a different backend than ostree. Here we basically just fork off `podman pull` to fetch container images into an *alternative root* in `/ostree/container-storage`, (Because otherwise basic things like `podman image prune` would delete the OS image) This is quite distinct from our use of `skopeo` in the ostree-ext project because suddenly now we gain support for things implemented in the containers/storage library like `zstd:chunked` and OCI crypt. *However*...today we still need to generate a final flattened filesystem tree (and an ostree commit) in order to maintain compatibilty with stuff in rpm-ostree. (A corrollary to this is we're not booting into a `podman mount` overlayfs stack) Related to this, we also need to handle SELinux labeling. Hence, we implement "layer squashing", and then do some final "postprocessing" on the resulting image matching the same logic that's done in ostree-ext such as `etc -> usr/etc` and handling `/var`. Note this also really wants ostreedev/ostree#3106 to avoid duplicating disk space. Signed-off-by: Colin Walters <walters@verbum.org>
cgwalters
force-pushed
the
commit-reflink
branch
from
d47380a
to
212c4d9
Compare
cgwalters
commented
Dec 4, 2023
Comment on lines
-610
to
+609
| if (glnx_regfile_copy_bytes (infd, tmpf.fd, (off_t)length) < 0) | ||
| return glnx_throw_errno_prefix (error, "regfile copy"); | ||
| int infd = g_file_descriptor_based_get_fd ((GFileDescriptorBased *)original_input); | ||
| if (ioctl (tmpf.fd, FICLONE, infd) == 0) |
There was a problem hiding this comment.
While glnx_regfile_copy_bytes will try reflinks, that will never work because we've wrapped the input with a checksum stream. And actually since we have to checksum, we can't do copy_file_range either, etc.
Ultimately this issue will be moot with a switch to composefs where we'll use fsverity in kernel to do checksums.
I think we originally used to do this, but at some point in a code refactoring, this optimization got lost. It's a quite important optimization for the case of writing content generated by an external system into an ostree repository.
cgwalters
force-pushed
the
commit-reflink
branch
from
212c4d9
to
c3dff62
Compare
jmarrero
approved these changes
Dec 5, 2023
jeckersb
pushed a commit
to jeckersb/bootc
that referenced
this pull request
Jan 23, 2024
See containers#147 (comment) With this bootc starts to really gain support for a different backend than ostree. Here we basically just fork off `podman pull` to fetch container images into an *alternative root* in `/ostree/container-storage`, (Because otherwise basic things like `podman image prune` would delete the OS image) This is quite distinct from our use of `skopeo` in the ostree-ext project because suddenly now we gain support for things implemented in the containers/storage library like `zstd:chunked` and OCI crypt. *However*...today we still need to generate a final flattened filesystem tree (and an ostree commit) in order to maintain compatibilty with stuff in rpm-ostree. (A corrollary to this is we're not booting into a `podman mount` overlayfs stack) Related to this, we also need to handle SELinux labeling. Hence, we implement "layer squashing", and then do some final "postprocessing" on the resulting image matching the same logic that's done in ostree-ext such as `etc -> usr/etc` and handling `/var`. Note this also really wants ostreedev/ostree#3106 to avoid duplicating disk space. Signed-off-by: Colin Walters <walters@verbum.org>
jeckersb
pushed a commit
to jeckersb/bootc
that referenced
this pull request
Mar 6, 2024
See containers#147 (comment) With this bootc starts to really gain support for a different backend than ostree. Here we basically just fork off `podman pull` to fetch container images into an *alternative root* in `/ostree/container-storage`, (Because otherwise basic things like `podman image prune` would delete the OS image) This is quite distinct from our use of `skopeo` in the ostree-ext project because suddenly now we gain support for things implemented in the containers/storage library like `zstd:chunked` and OCI crypt. *However*...today we still need to generate a final flattened filesystem tree (and an ostree commit) in order to maintain compatibilty with stuff in rpm-ostree. (A corrollary to this is we're not booting into a `podman mount` overlayfs stack) Related to this, we also need to handle SELinux labeling. Hence, we implement "layer squashing", and then do some final "postprocessing" on the resulting image matching the same logic that's done in ostree-ext such as `etc -> usr/etc` and handling `/var`. Note this also really wants ostreedev/ostree#3106 to avoid duplicating disk space. Signed-off-by: Colin Walters <walters@verbum.org>
jeckersb
pushed a commit
to jeckersb/bootc
that referenced
this pull request
Mar 20, 2024
See containers#147 (comment) With this bootc starts to really gain support for a different backend than ostree. Here we basically just fork off `podman pull` to fetch container images into an *alternative root* in `/ostree/container-storage`, (Because otherwise basic things like `podman image prune` would delete the OS image) This is quite distinct from our use of `skopeo` in the ostree-ext project because suddenly now we gain support for things implemented in the containers/storage library like `zstd:chunked` and OCI crypt. *However*...today we still need to generate a final flattened filesystem tree (and an ostree commit) in order to maintain compatibilty with stuff in rpm-ostree. (A corrollary to this is we're not booting into a `podman mount` overlayfs stack) Related to this, we also need to handle SELinux labeling. Hence, we implement "layer squashing", and then do some final "postprocessing" on the resulting image matching the same logic that's done in ostree-ext such as `etc -> usr/etc` and handling `/var`. Note this also really wants ostreedev/ostree#3106 to avoid duplicating disk space. Signed-off-by: Colin Walters <walters@verbum.org>
jeckersb
pushed a commit
to jeckersb/bootc
that referenced
this pull request
Mar 28, 2024
See containers#147 (comment) With this bootc starts to really gain support for a different backend than ostree. Here we basically just fork off `podman pull` to fetch container images into an *alternative root* in `/ostree/container-storage`, (Because otherwise basic things like `podman image prune` would delete the OS image) This is quite distinct from our use of `skopeo` in the ostree-ext project because suddenly now we gain support for things implemented in the containers/storage library like `zstd:chunked` and OCI crypt. *However*...today we still need to generate a final flattened filesystem tree (and an ostree commit) in order to maintain compatibilty with stuff in rpm-ostree. (A corrollary to this is we're not booting into a `podman mount` overlayfs stack) Related to this, we also need to handle SELinux labeling. Hence, we implement "layer squashing", and then do some final "postprocessing" on the resulting image matching the same logic that's done in ostree-ext such as `etc -> usr/etc` and handling `/var`. Note this also really wants ostreedev/ostree#3106 to avoid duplicating disk space. Signed-off-by: Colin Walters <walters@verbum.org>
jeckersb
pushed a commit
to jeckersb/bootc
that referenced
this pull request
May 31, 2024
See containers#147 (comment) With this bootc starts to really gain support for a different backend than ostree. Here we basically just fork off `podman pull` to fetch container images into an *alternative root* in `/ostree/container-storage`, (Because otherwise basic things like `podman image prune` would delete the OS image) This is quite distinct from our use of `skopeo` in the ostree-ext project because suddenly now we gain support for things implemented in the containers/storage library like `zstd:chunked` and OCI crypt. *However*...today we still need to generate a final flattened filesystem tree (and an ostree commit) in order to maintain compatibilty with stuff in rpm-ostree. (A corrollary to this is we're not booting into a `podman mount` overlayfs stack) Related to this, we also need to handle SELinux labeling. Hence, we implement "layer squashing", and then do some final "postprocessing" on the resulting image matching the same logic that's done in ostree-ext such as `etc -> usr/etc` and handling `/var`. Note this also really wants ostreedev/ostree#3106 to avoid duplicating disk space. Signed-off-by: Colin Walters <walters@verbum.org>
jeckersb
pushed a commit
to jeckersb/bootc
that referenced
this pull request
Jun 4, 2024
See containers#147 (comment) With this bootc starts to really gain support for a different backend than ostree. Here we basically just fork off `podman pull` to fetch container images into an *alternative root* in `/ostree/container-storage`, (Because otherwise basic things like `podman image prune` would delete the OS image) This is quite distinct from our use of `skopeo` in the ostree-ext project because suddenly now we gain support for things implemented in the containers/storage library like `zstd:chunked` and OCI crypt. *However*...today we still need to generate a final flattened filesystem tree (and an ostree commit) in order to maintain compatibilty with stuff in rpm-ostree. (A corrollary to this is we're not booting into a `podman mount` overlayfs stack) Related to this, we also need to handle SELinux labeling. Hence, we implement "layer squashing", and then do some final "postprocessing" on the resulting image matching the same logic that's done in ostree-ext such as `etc -> usr/etc` and handling `/var`. Note this also really wants ostreedev/ostree#3106 to avoid duplicating disk space. Signed-off-by: Colin Walters <walters@verbum.org>
jeckersb
pushed a commit
to jeckersb/bootc
that referenced
this pull request
Jun 5, 2024
See containers#147 (comment) With this bootc starts to really gain support for a different backend than ostree. Here we basically just fork off `podman pull` to fetch container images into an *alternative root* in `/ostree/container-storage`, (Because otherwise basic things like `podman image prune` would delete the OS image) This is quite distinct from our use of `skopeo` in the ostree-ext project because suddenly now we gain support for things implemented in the containers/storage library like `zstd:chunked` and OCI crypt. *However*...today we still need to generate a final flattened filesystem tree (and an ostree commit) in order to maintain compatibilty with stuff in rpm-ostree. (A corrollary to this is we're not booting into a `podman mount` overlayfs stack) Related to this, we also need to handle SELinux labeling. Hence, we implement "layer squashing", and then do some final "postprocessing" on the resulting image matching the same logic that's done in ostree-ext such as `etc -> usr/etc` and handling `/var`. Note this also really wants ostreedev/ostree#3106 to avoid duplicating disk space. Signed-off-by: Colin Walters <walters@verbum.org>
jeckersb
pushed a commit
to jeckersb/bootc
that referenced
this pull request
Jun 14, 2024
See containers#147 (comment) With this bootc starts to really gain support for a different backend than ostree. Here we basically just fork off `podman pull` to fetch container images into an *alternative root* in `/ostree/container-storage`, (Because otherwise basic things like `podman image prune` would delete the OS image) This is quite distinct from our use of `skopeo` in the ostree-ext project because suddenly now we gain support for things implemented in the containers/storage library like `zstd:chunked` and OCI crypt. *However*...today we still need to generate a final flattened filesystem tree (and an ostree commit) in order to maintain compatibilty with stuff in rpm-ostree. (A corrollary to this is we're not booting into a `podman mount` overlayfs stack) Related to this, we also need to handle SELinux labeling. Hence, we implement "layer squashing", and then do some final "postprocessing" on the resulting image matching the same logic that's done in ostree-ext such as `etc -> usr/etc` and handling `/var`. Note this also really wants ostreedev/ostree#3106 to avoid duplicating disk space. Signed-off-by: Colin Walters <walters@verbum.org>
jeckersb
pushed a commit
to cgwalters/bootc
that referenced
this pull request
Jul 2, 2024
See containers#147 (comment) With this bootc starts to really gain support for a different backend than ostree. Here we basically just fork off `podman pull` to fetch container images into an *alternative root* in `/ostree/container-storage`, (Because otherwise basic things like `podman image prune` would delete the OS image) This is quite distinct from our use of `skopeo` in the ostree-ext project because suddenly now we gain support for things implemented in the containers/storage library like `zstd:chunked` and OCI crypt. *However*...today we still need to generate a final flattened filesystem tree (and an ostree commit) in order to maintain compatibilty with stuff in rpm-ostree. (A corrollary to this is we're not booting into a `podman mount` overlayfs stack) Related to this, we also need to handle SELinux labeling. Hence, we implement "layer squashing", and then do some final "postprocessing" on the resulting image matching the same logic that's done in ostree-ext such as `etc -> usr/etc` and handling `/var`. Note this also really wants ostreedev/ostree#3106 to avoid duplicating disk space. Signed-off-by: Colin Walters <walters@verbum.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I think we originally used to do this, but at some point in a code refactoring, this optimization got lost.
It's a quite important optimization for the case of writing content generated by an external system into an ostree repository.