-
Notifications
You must be signed in to change notification settings - Fork 290
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Rework devino caching to support invalidation if metadata changes #1170
Conversation
@@ -2858,16 +2858,6 @@ write_dfd_iter_to_mtree_internal (OstreeRepo *self, | |||
if (!glnx_fstatat (src_dfd_iter->fd, dent->d_name, &stbuf, AT_SYMLINK_NOFOLLOW, error)) | |||
return FALSE; | |||
|
|||
const char *loose_checksum = devino_cache_lookup (self, modifier, stbuf.st_dev, stbuf.st_ino); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this patch should be included with the last commit since they're not
really independent in the sense that I think we were doing this early lookup
intentionally before. It was intending to be a fast path.
src/libostree/ostree-repo-commit.c
Outdated
loose_checksum = devino_cache_lookup (self, modifier, | ||
g_file_info_get_attribute_uint32 (child_info, "unix::device"), | ||
g_file_info_get_attribute_uint64 (child_info, "unix::inode")); | ||
/* only check the devino cache if the file info was not modified */ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What about xattrs? For the rpm-ostree use case it'd make commit notably more expensive as suddenly we'd have to load and compare the xattrs.
If we're explicitly not comparing xattrs then let's at least add a comment.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ahh of course, we need to check xattrs too. Will add that.
Hmm, do you mean in the rpm-ostree local assembly path? Though we don't provide an xattr modifier when checking out pkgs there, so we can probably do it in a smart way so we don't even bother comparing in such cases.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm...but actually in the layering case I think we should be setting the sepolicy to ensure we relabel any files created by scripts. (In practice this probably doesn't matter much and really everything should mostly be usr_t
but still).
21776b6
to
7d9059c
Compare
OK, this now also checks for modifications to extended attributes. One thing I'm not sure of is:
Without that change, really there's not much point in using a devino cache if also using an xattr cb. |
Pull out the first commit to a separate PR? |
I'd like to defer this until we've had a chance to discuss in #1165 |
☔ The latest upstream changes (presumably e44631e) made this pull request unmergeable. Please resolve the merge conflicts. |
src/libostree/ostree-repo-commit.c
Outdated
@@ -2557,6 +2571,7 @@ write_directory_content_to_mtree_internal (OstreeRepo *self, | |||
g_autoptr(GFileInfo) modified_info = NULL; | |||
OstreeRepoCommitFilterResult filter_result = | |||
_ostree_repo_commit_modifier_apply (self, modifier, child_relpath, child_info, &modified_info); | |||
const gboolean child_info_was_modified = (child_info != modified_info); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think my main problem is this - if we have say a commit filter active and that filter just wants to change some paths like gnome-continuous does with suid binaries, this will still forcibly re-checksum everything since the info will be different.
I think we need to compare it structurally the same way we do xattrs.
This allows users to easily re-initialize the test repo.
7d9059c
to
211e748
Compare
Rebased and updated to also inspect the |
@@ -141,6 +141,7 @@ setup_test_repository () { | |||
fi | |||
|
|||
cd ${test_tmpdir} | |||
rm -rf repo |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This makes sense, though FWIW I tend to do:
$ make check TESTS=tests/test-commit-sign.sh TEST_SKIP_CLEANUP=err
$ make && sudo make install
$ cd /var/tmp/tap-test.XXX...
$ ostree ...
|
||
/* those aren't stored by ostree, but used by the devino cache */ | ||
g_file_info_set_attribute_uint32 (ret, "unix::device", stbuf->st_dev); | ||
g_file_info_set_attribute_uint64 (ret, "unix::inode", stbuf->st_ino); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ooh, tricky.
src/libostree/ostree-repo-commit.c
Outdated
/* fetch on-disk xattrs if needed & not disabled */ | ||
g_autoptr(GVariant) original_xattrs = NULL; | ||
if (!(modifier && (modifier->flags & (OSTREE_REPO_COMMIT_MODIFIER_FLAGS_SKIP_XATTRS | | ||
OSTREE_REPO_COMMIT_MODIFIER_FLAGS_CANONICAL_PERMISSIONS)) > 0) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not new, but maybe while we're here, host out the flags checking like:
const gboolean flags_skip_xattrs = (modifier->flags & (OSTREE_REPO_COMMIT_MODIFIER_FLAGS_SKIP_XATTRS |
OREE_REPO_COMMIT_MODIFIER_FLAGS_CANONICAL_PERMISSIONS)) > 0;
if (!(modifier && flags_skip_xattrs && !self->disable_xattrs))
{
...
assert_not_file_has_content diff-test2 'M */baz/cowro$' | ||
assert_not_file_has_content diff-test2 'baz/saucer' | ||
# only /baz/cow is a cache miss | ||
assert_file_has_content stats.txt '^Content Written: 1$' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, right I forgot we already had a way to unit test this. Cool.
tests/test-basic-c.c
Outdated
|
||
if (on_overlay) | ||
{ | ||
g_test_skip ("overlayfs detected"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is fine for now, though a more accurate fix would be porting the "can relabel" logic. Though OTOH I agree it is perhaps easier for dev testing to set the directory. Maybe we want TMPDIR_PREFIX though?
Eh, might as well zap 🗲 this one; we can do my one minor style tweak later. |
This allows users to easily re-initialize the test repo. Closes: #1170 Approved by: cgwalters
This was originally part of #1199, but never got in. The work in #1218 is awesome for CI, though for the developer workflow, I find it much easier to just set $TEST_TMPDIR to a non-overlayfs mount point. Teach the test libraries to just look at $PWD rather than /. Also export the function that does the overlayfs checking. Prep for a future patch. Closes: #1170 Approved by: cgwalters
We can't use the cache if the file we want to commit has been modified by the client through the file info or xattr modifiers. We would prematurely look into the cache in `write_dfd_iter_to_mtree_internal`, regardless of whether any filtering applied. We remove that path there, and make sure that we only use the cache if there were no modifications. We rename the `get_modified_xattrs` to `get_final_xattrs` to reflect the fact that the xattrs may not be modified. One tricky bit that took me some time was that we now need to store the st_dev & st_ino values in the GFileInfo because the cache lookup relies on it. I'm guessing we regressed on this at some point. This patch does slightly change the semantics of the xattr callback. Previously, returning NULL from the cb meant no xattrs at all. Now, it means to default to the on-disk state. We might want to consider putting that behind a flag instead. Though it seems like a more useful behaviour so that callers can only override the files they want to without losing original on-disk state (and if they don't want that, just return an empty GVariant). Closes: #1165 Closes: #1170 Approved by: cgwalters
💔 Test failed - status-atomicjenkins |
tests/libostreetest.c
Outdated
{ | ||
/* Keep this in sync with the overlayfs bits in libtest.sh */ | ||
struct statfs stbuf; | ||
const char *pwd = g_getenv ("PWD"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Surely g_get_current_dir()
(needs to be assigned to a g_autofree gchar *
), or simply replace pwd
with "."
below?
I'm not completely sure of the startup sequence of this stuff, but if the test isn't guaranteed to have been started by a shell designed for interactive use, then I don't think PWD
is guaranteed to be up-to-date.
According to its documentation, g_get_current_dir()
returns PWD
, but only after verifying that PWD
is in fact the same directory as "."
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure. I had started with getenv ("TEST_TMPDIR")
at first, though changed it afterwards since it doesn't actually appear when running with gdtr.
tests/libtest.sh
Outdated
@@ -73,7 +73,7 @@ export OSTREE_GPG_HOME=${test_tmpdir}/gpghome/trusted | |||
# See comment in ot-builtin-commit.c and https://github.com/ostreedev/ostree/issues/758 | |||
# Also keep this in sync with the bits in libostreetest.c | |||
echo evaluating for overlayfs... | |||
case $(stat -f --printf '%T' /) in | |||
case $(stat -f --printf '%T' $PWD) in |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same here: this could be .
(although at least this is a bash script so PWD
is guaranteed(?) to be set and up-to-date)
tests/test-basic-c.c
Outdated
g_assert (ret); | ||
|
||
gboolean on_overlay; | ||
ret = ot_check_for_overlay (&on_overlay, &error); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If this is a proxy for "check for a filesystem where we can set xattrs", please probe for that instead - tmpfs can't have xattrs either, and for optimal speed (and lack of fsync slowdown), some Debian buildds use a chroot where the entire visible filesystem is on tmpfs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, @cgwalters echoed the same sentiments higher up. I'll rework it.
We can't use the cache if the file we want to commit has been modified by the client through the file info or xattr modifiers. We would prematurely look into the cache in `write_dfd_iter_to_mtree_internal`, regardless of whether any filtering applied. We remove that path there, and make sure that we only use the cache if there were no modifications. We rename the `get_modified_xattrs` to `get_final_xattrs` to reflect the fact that the xattrs may not be modified. One tricky bit that took me some time was that we now need to store the st_dev & st_ino values in the GFileInfo because the cache lookup relies on it. I'm guessing we regressed on this at some point. This patch does slightly change the semantics of the xattr callback. Previously, returning NULL from the cb meant no xattrs at all. Now, it means to default to the on-disk state. We might want to consider putting that behind a flag instead. Though it seems like a more useful behaviour so that callers can only override the files they want to without losing original on-disk state (and if they don't want that, just return an empty GVariant). Closes: ostreedev#1165
Instead of checking for overlayfs, let's explicitly check for our ability to relabel files since we now have a `libtest` function to do this. Also port that logic to `libostreetest`. Note that overlayfs *does* allow manipulating user xattrs. So ideally, we should break down `OSTREE_NO_XATTRS` further to distinguish between tests that use bare repos from other modes. We check the current directory instead of `/` so that developers can just point `TEST_TMPDIR` to a non-overlayfs mount point when hacking from a container.
211e748
to
37a2fad
Compare
OK, I reworked the tests so that they explicitly check for relabeling support rather than overlayfs. It's a bit confusing because |
Since we now have a subtest there that needs full xattr support.
And now we also run as an installed test to make sure the new xattr test actually runs! ⬆️ |
g_autoptr(GBytes) bytes = glnx_fgetxattr_bytes (tmpf.fd, "security.selinux", &local_error); | ||
if (!bytes) | ||
{ | ||
/* libglnx preserves errno */ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm...not explicitly. I think it likely mostly works today, but I'm certain that e.g. glnx_null_throw_errno_prefix()
might invoke malloc()
which can definitely make system calls... if we want to make that guarantee I think we'll need something more like systemd's _PROTECT_ERRNO
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nevermind, I forgot we do save it.
tests/libtest.sh
Outdated
|
||
_have_user_xattrs='' | ||
have_user_xattrs() { | ||
assert_has_setfattr |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Minor, but maybe move the assertion below the test so we don't call it every time?
OK, this is looking pretty good to me! Let's get it in, we can do any followups after. Maybe at some point we can stub out a context or case that uses tmpfs so @smvc doesn't have to come in and fix things up after every release. |
⚡ Test exempted: merge already tested. |
We can't use the cache if the file we want to commit has been modified by the client through the file info or xattr modifiers. We would prematurely look into the cache in `write_dfd_iter_to_mtree_internal`, regardless of whether any filtering applied. We remove that path there, and make sure that we only use the cache if there were no modifications. We rename the `get_modified_xattrs` to `get_final_xattrs` to reflect the fact that the xattrs may not be modified. One tricky bit that took me some time was that we now need to store the st_dev & st_ino values in the GFileInfo because the cache lookup relies on it. I'm guessing we regressed on this at some point. This patch does slightly change the semantics of the xattr callback. Previously, returning NULL from the cb meant no xattrs at all. Now, it means to default to the on-disk state. We might want to consider putting that behind a flag instead. Though it seems like a more useful behaviour so that callers can only override the files they want to without losing original on-disk state (and if they don't want that, just return an empty GVariant). Closes: #1165 Closes: #1170 Approved by: cgwalters
Instead of checking for overlayfs, let's explicitly check for our ability to relabel files since we now have a `libtest` function to do this. Also port that logic to `libostreetest`. Note that overlayfs *does* allow manipulating user xattrs. So ideally, we should break down `OSTREE_NO_XATTRS` further to distinguish between tests that use bare repos from other modes. We check the current directory instead of `/` so that developers can just point `TEST_TMPDIR` to a non-overlayfs mount point when hacking from a container. Closes: #1170 Approved by: cgwalters
Since we now have a subtest there that needs full xattr support. Closes: #1170 Approved by: cgwalters
…xattrs This is more subtle fallout from: ostreedev#1170 AKA commit: 8fe4536 Before, if we found a devino cache hit, we'd use it unconditionally. Recall that `bare-user` repositories are very special in that they're the only mode where the on disk state ("physical state") is not the "real" state. The latter is stored in the `user.ostreemeta` xattr. (`bare-user` repos are also highly special in that symlinks are regular files physically, but that's not immediately relevant here). Since we now have `bare-user-only` for the "pure unprivileged container" case, `bare-user` should just be used for "OS builds" which have nonzero uids (and possibly SELinux labels etc.) In an experimental tool I'm writing "skopeo2ostree" which imports OCI images into refs, then squashes them together into a single final commit, we lost the the `81` group ID for `/usr/libexec/dbus-1/dbus-daemon-launch-helper`. This happened because the commit code was loading the "physical" disk state, where the uid/gid are zero because that's the uid I happened to be using. We didn't just directly do the link speedup because I was using `--selinux-policy` which caused the xattrs to change, which caused us to re-commit objects from the physical state. The unit test I added actually doesn't quite trigger this, but I left it because "why not". Really testing this requires the installed test which uses SELinux policy from `/`. The behavior without this fix looks like: ``` -00755 0 0 12 { [(b'user.ostreemeta', [byte 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x51, 0x00, 0x00, 0x81, 0xed]), (b'security.selinux', b'system_u:object_r:lib_t:s0')] } /usr/lib/dbus-daemon-helper ``` which was obviously totally broken - we shouldn't be picking up the `user.ostreemeta` xattr and actually committing it of course.
This is more subtle fallout from: ostreedev#1170 AKA commit: 8fe4536 Before, if we found a devino cache hit, we'd use it unconditionally. Recall that `bare-user` repositories are very special in that they're the only mode where the on disk state ("physical state") is not the "real" state. The latter is stored in the `user.ostreemeta` xattr. (`bare-user` repos are also highly special in that symlinks are regular files physically, but that's not immediately relevant here). Since we now have `bare-user-only` for the "pure unprivileged container" case, `bare-user` should just be used for "OS builds" which have nonzero uids (and possibly SELinux labels etc.) In an experimental tool I'm writing "skopeo2ostree" which imports OCI images into refs, then squashes them together into a single final commit, we lost the the `81` group ID for `/usr/libexec/dbus-1/dbus-daemon-launch-helper`. This happened because the commit code was loading the "physical" disk state, where the uid/gid are zero because that's the uid I happened to be using. We didn't just directly do the link speedup because I was using `--selinux-policy` which caused the xattrs to change, which caused us to re-commit objects from the physical state. The unit test I added actually doesn't quite trigger this, but I left it because "why not". Really testing this requires the installed test which uses SELinux policy from `/`. The behavior without this fix looks like: ``` -00755 0 0 12 { [(b'user.ostreemeta', [byte 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x51, 0x00, 0x00, 0x81, 0xed]), (b'security.selinux', b'system_u:object_r:lib_t:s0')] } /usr/lib/dbus-daemon-helper ``` which was obviously totally broken - we shouldn't be picking up the `user.ostreemeta` xattr and actually committing it of course.
This is more subtle fallout from: #1170 AKA commit: 8fe4536 Before, if we found a devino cache hit, we'd use it unconditionally. Recall that `bare-user` repositories are very special in that they're the only mode where the on disk state ("physical state") is not the "real" state. The latter is stored in the `user.ostreemeta` xattr. (`bare-user` repos are also highly special in that symlinks are regular files physically, but that's not immediately relevant here). Since we now have `bare-user-only` for the "pure unprivileged container" case, `bare-user` should just be used for "OS builds" which have nonzero uids (and possibly SELinux labels etc.) In an experimental tool I'm writing "skopeo2ostree" which imports OCI images into refs, then squashes them together into a single final commit, we lost the the `81` group ID for `/usr/libexec/dbus-1/dbus-daemon-launch-helper`. This happened because the commit code was loading the "physical" disk state, where the uid/gid are zero because that's the uid I happened to be using. We didn't just directly do the link speedup because I was using `--selinux-policy` which caused the xattrs to change, which caused us to re-commit objects from the physical state. The unit test I added actually doesn't quite trigger this, but I left it because "why not". Really testing this requires the installed test which uses SELinux policy from `/`. The behavior without this fix looks like: ``` -00755 0 0 12 { [(b'user.ostreemeta', [byte 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x51, 0x00, 0x00, 0x81, 0xed]), (b'security.selinux', b'system_u:object_r:lib_t:s0')] } /usr/lib/dbus-daemon-helper ``` which was obviously totally broken - we shouldn't be picking up the `user.ostreemeta` xattr and actually committing it of course. Closes: #1297 Approved by: jlebon
I was seeing the `Writing OSTree commit...` phase of rpm-ostree being very slow lately. This turns out to be more fallout from ostreedev#1170 AKA commit: 8fe4536 Loading the xattrs is slow on my system (F27AW, XFS+LVM, NVMe). I haven't fully traced through why, but AIUI at least on XFS the xattrs are often stored outside of the inode so it's a little bit like doing an `open()+read()`. Plus there's the LSM overhead, etc. The thing is that for rpm-ostree's package layering use case, we basically always want to treat the on-disk state as canonical. (There's a subtle case here if one does overrides for something that contains policy but we'll fix that). Anyways, so we're in a state now where we do the slow but correct thing by default, which seems sane. But let's allow the app to opt-in to telling us "really trust devino". The difference between a `stat()` + hash table lookup versus the full xattr load on my test case of `rpm-ostree install ./tree-1.7.0-10.fc27.x86_64.rpm` is absolutely dramatic; consistently on the order of 10s without this support, and <1s with (800ms). No tests yet.
I was seeing the `Writing OSTree commit...` phase of rpm-ostree being very slow lately. This turns out to be more fallout from ostreedev#1170 AKA commit: 8fe4536 Loading the xattrs is slow on my system (F27AW, XFS+LVM, NVMe). I haven't fully traced through why, but AIUI at least on XFS the xattrs are often stored outside of the inode so it's a little bit like doing an `open()+read()`. Plus there's the LSM overhead, etc. The thing is that for rpm-ostree's package layering use case, we basically always want to treat the on-disk state as canonical. (There's a subtle case here if one does overrides for something that contains policy but we'll fix that). Anyways, so we're in a state now where we do the slow but correct thing by default, which seems sane. But let's allow the app to opt-in to telling us "really trust devino". The difference between a `stat()` + hash table lookup versus the full xattr load on my test case of `rpm-ostree install ./tree-1.7.0-10.fc27.x86_64.rpm` is absolutely dramatic; consistently on the order of 10s without this support, and <1s with (800ms).
I was seeing the `Writing OSTree commit...` phase of rpm-ostree being very slow lately. This turns out to be more fallout from #1170 AKA commit: 8fe4536 Loading the xattrs is slow on my system (F27AW, XFS+LVM, NVMe). I haven't fully traced through why, but AIUI at least on XFS the xattrs are often stored outside of the inode so it's a little bit like doing an `open()+read()`. Plus there's the LSM overhead, etc. The thing is that for rpm-ostree's package layering use case, we basically always want to treat the on-disk state as canonical. (There's a subtle case here if one does overrides for something that contains policy but we'll fix that). Anyways, so we're in a state now where we do the slow but correct thing by default, which seems sane. But let's allow the app to opt-in to telling us "really trust devino". The difference between a `stat()` + hash table lookup versus the full xattr load on my test case of `rpm-ostree install ./tree-1.7.0-10.fc27.x86_64.rpm` is absolutely dramatic; consistently on the order of 10s without this support, and <1s with (800ms). Closes: #1357 Approved by: jlebon
For our build-system we have a [patched version of `fakeroot`][1] which understands the `user.ostreemeta` xattr. It's intended for use with `ostree checkout --user-mode --require-hardlinks`. When programs are run with `LD_PRELOAD=libfakeroot...` they see the ownership and permissions that ostree understands, rather than the real ones. Our build system mostly consists of calls to: ostree checkout --user-mode --require-hardlinks $COMMIT root LD_PRELOAD=fakeroot <make some changes> ostree commit --tree=dir=root --link-checkout-speedup --devino-canonical in the past this seemed to work fine but at some point I started losing the setuid bit on `sudo`. I suspect the previous behaviour was a bug, possibly fixed in ostreedev#1170 (8fe4536). This commit makes the `ostreemeta` preserving behaviour explicit. If the user.ostreemeta xattr doesn't exist the file is given the sensible default uid/gid of 0:0. `--use-bare-user-xattrs` will currently be ignored when loading a commit from a tarball. This can be fixed in the future if there's any demand for it. [1]: https://github.com/stb-tester/fakeroot/tree/ostree
Fixes #1165.