Add bare-user-only mode that works on non-xattrs filesystems #750

alexlarsson · 2017-03-22T11:37:20Z

This mode is similar to bare-user, but does not store the permission,
ownership (uid/gid) and xattrs in an xattr on the file objects in the
repo. Additionally it stores symlinks as symlinks rather than as
regular files+xattrs, like the bare mode. The later is needed because
we can't store the is-symlink in the xattr.

This means that some metadata is lost, such as the uid. When reading a
repo like this we always report uid, gid as 0, and no xattrs, so
unless this is true in the commit the resulting repository will
not fsck correctly.

However, it the main usecase of the repository is to check out with
--user-mode, then no information is lost, and the repository can
work on filesystems without xattrs (such as tmpfs).

alexlarsson · 2017-03-22T11:37:46Z

I guess it needs some tests too

cgwalters · 2017-03-22T15:14:49Z

src/ostree/ot-builtin-commit.c

@@ -84,6 +85,7 @@ static GOptionEntry options[] = {
  { "add-detached-metadata-string", 0, 0, G_OPTION_ARG_STRING_ARRAY, &opt_detached_metadata_strings, "Add a key/value pair to detached metadata", "KEY=VALUE" },
  { "owner-uid", 0, 0, G_OPTION_ARG_INT, &opt_owner_uid, "Set file ownership user id", "UID" },
  { "owner-gid", 0, 0, G_OPTION_ARG_INT, &opt_owner_gid, "Set file ownership group id", "GID" },
+  { "canonical-permissions", 0, 0, G_OPTION_ARG_NONE, &opt_canonical_permissions, "Canonicallize permissions in the same way bare-user does for hardlinked files", NULL },


s/Canonicallize/Canonicalize

cgwalters · 2017-03-22T15:39:08Z

src/ostree/ot-builtin-commit.c

@@ -196,6 +198,13 @@ commit_filter (OstreeRepo         *self,
      g_hash_table_remove (mode_adds, path);
    }

+  if (opt_canonical_permissions &&
+      g_file_info_get_file_type (file_info) == G_FILE_TYPE_REGULAR)


Why isn't this just the default if writing to bare-user-only though?

It could be, but we might want to commit this to an archive-z2 repo and then pull it into a bare-user-only one.

Yep, I understand the rationale for the option, but I'd still say it should be on by default for bare-user. Among other things, every API caller will need to duplicate this logic if it's not.

cgwalters · 2017-03-22T15:41:09Z

This all looks pretty good to me! One comment about the perms on commit; feels like if we do that it'd reduce the diff in the tests a lot too.

I also wonder if we should basically force on -U for checking out in this mode too?

alexlarsson · 2017-03-22T15:44:51Z

Force -U, or fail without it?

alexlarsson · 2017-03-22T15:47:08Z

Ah, i see what you mean, if its automatic we get less testcase noise.

cgwalters · 2017-03-22T15:51:47Z

I'd vote force -U.

And with both of these things, what I'm actually arguing is we do the defaulting not at the command line level, but at the libostree level. For checkout this should be pretty easy, check the mode in ostree_repo_checkout_at() and override.

For commit it's likely to be trickier in the general case because we have ostree_repo_write_content() - we'd have to crack it open and mutate it. We might be able to get away without doing that for now if we just change the bit in write_directory_content_to_mtree_internal() where we pull from the filesystem.

cgwalters · 2017-03-22T15:52:45Z

To put this another way - I'd say we've succeeded if flatpak doesn't need any code changes to work with this new repo mode other than the parts that e.g. initialize the repo with the new mode. Right?

alexlarsson · 2017-03-22T15:54:20Z

Well, some of the ostree test cases commit to archive-z2 repos, which are then pulled to bare-user-only, so forcing it on commit to bare-user-only is not enough.

cgwalters · 2017-03-22T15:55:48Z

Right, I agree we still need the canonical permissions argument. Along the API lines, what about making it OSTREE_REPO_COMMIT_MODIFIER_FLAGS_CANONICAL_PERMISSIONS ?

alexlarsson · 2017-03-22T15:57:33Z

flatpak doesn't technically need any changes (other than making repos of this type). However, it would be nice if flatpak build-export creates commits that are canonical so that fsck works on the flatpak local repo. However, build-export generally exports to archive-z2 repos, so any change we make to that will not affect much. The amount of changes needed are very small though.

cgwalters · 2017-03-22T17:20:36Z

src/libostree/ostree-repo-commit.c

+      g_file_info_get_file_type (file_info) == G_FILE_TYPE_REGULAR)
+    {
+      guint current_mode = g_file_info_get_attribute_uint32 (modified_info, "unix::mode");
+      g_file_info_set_attribute_uint32 (modified_info, "unix::mode", current_mode | 0744);


Don't we also need to set uid/gid to zero?

Its already possible to do this with the separate uid/gid options, so i made it only do what we couldn't already do.

We could make this automatically mean uid/gid 0 and no-xatts too, but that means you can't do the indiviual operations separately

If you're committing to a bare-user-only repo, those are going to be discarded.

I can't think of a scenario where one is committing to an archive repo and wants to use this flag, but doesn't also want the uid/gid/no-xattrs.

I get what you're saying about everything else being achievable, but the flip side is - if we're not aware of a use case, why add it?

(A small reason to go the flag = full canon route is that at some point I'd like to optimize commit so we don't malloc() like there's no tomorrow, and part of that is going to be to stop allocating GFileInfo in favor of a struct stat on the stack. Which we can only do right now if we don't have a commit modifier)

That said this is a "lean against" feeling - if you really prefer it the way it is now, given flatpak would be the primary user that's OK by me.

alexlarsson · 2017-03-24T10:38:17Z

I made --canonical-permissions imply everything.

alexlarsson · 2017-03-24T10:39:53Z

I'm gonna skip modifying the commit deep internals to do this automatically for bare-user-only repos though. There is to much code that has to be modified for that, and its not the common usecase i imagine anyway.

cgwalters · 2017-03-24T13:02:27Z

Re changing the commit core, agree that's probably not worth it right now. I have a little concern we're going to hit corner cases without doing it, but OTOH if the tests pass, I feel fairly confident.

Maybe though when this shows up in the next release we mark it as experimental, until flatpak has been using it for a bit in its devel stream?

@rh-atomic-bot r+ a84ec48

rh-atomic-bot · 2017-03-24T13:02:33Z

⌛ Testing commit a84ec48 with merge 6a5d253...

This cleans up some existing code, but it also allows us to later add new bare modes. Closes: #750 Approved by: cgwalters

This mode is similar to bare-user, but does not store the permission, ownership (uid/gid) and xattrs in an xattr on the file objects in the repo. Additionally it stores symlinks as symlinks rather than as regular files+xattrs, like the bare mode. The later is needed because we can't store the is-symlink in the xattr. This means that some metadata is lost, such as the uid. When reading a repo like this we always report uid, gid as 0, and no xattrs, so unless this is true in the commit the resulting repository will not fsck correctly. However, it the main usecase of the repository is to check out with --user-mode, then no information is lost, and the repository can work on filesystems without xattrs (such as tmpfs). Closes: #750 Approved by: cgwalters

This adds to file permission masks the same bitmask that will be applied to file objects in bare-user* repos. This will be needed in the testsuite to ensure that the things we commit will be expressable in bare-user-only repos. Closes: #750 Approved by: cgwalters

This is somewhat complicated by such repos only properly supporting some subset of file metadata (uid/gid 0, etc). We fix this by always commiting with filters that make it work. Closes: #750 Approved by: cgwalters

cgwalters · 2017-03-24T13:03:14Z

Hm, actually it looks like some of these test failures are not flakes, e.g.:

error: lsetxattrOperation not supported

alexlarsson · 2017-03-24T13:04:58Z

I though that was some generic ci issue. Does other PRs not get it?

cgwalters · 2017-03-24T13:05:52Z

With our Homu instance, every test there except Travis has to pass to be pushed.

alexlarsson · 2017-03-24T13:06:56Z

But how my changes cause enosupp?

rh-atomic-bot · 2017-03-24T13:11:09Z

💔 Test failed - status-atomicjenkins

cgwalters · 2017-03-24T13:16:19Z

Not sure yet; built your branch and make check TESTS=tests/test-basic.sh passes here.

alexlarsson · 2017-03-24T13:19:09Z

All the tests work here, but its not only test-basic thats failing with ENOSUPP, even tests/test-sysroot-c

rh-atomic-bot · 2017-03-24T22:28:05Z

☔ The latest upstream changes (presumably 455cc5e) made this pull request unmergeable. Please resolve the merge conflicts.

This cleans up some existing code, but it also allows us to later add new bare modes.

This mode is similar to bare-user, but does not store the permission, ownership (uid/gid) and xattrs in an xattr on the file objects in the repo. Additionally it stores symlinks as symlinks rather than as regular files+xattrs, like the bare mode. The later is needed because we can't store the is-symlink in the xattr. This means that some metadata is lost, such as the uid. When reading a repo like this we always report uid, gid as 0, and no xattrs, so unless this is true in the commit the resulting repository will not fsck correctly. However, it the main usecase of the repository is to check out with --user-mode, then no information is lost, and the repository can work on filesystems without xattrs (such as tmpfs).

This adds to file permission masks the same bitmask that will be applied to file objects in bare-user* repos. This will be needed in the testsuite to ensure that the things we commit will be expressable in bare-user-only repos.

This is somewhat complicated by such repos only properly supporting some subset of file metadata (uid/gid 0, etc). We fix this by always commiting with filters that make it work.

alexlarsson · 2017-03-26T09:11:03Z

I applied the fixups and rebased to master, fixing the (trivial) conflicts.

alexlarsson · 2017-03-27T07:28:23Z

So, i really want to start using this for the flatpak runtime builds, so i'm ack:ing this again (since colin already ack:ed it above). Hope that is ok.
@rh-atomic-bot r+ ae279d0

rh-atomic-bot · 2017-03-27T07:28:24Z

@alexlarsson: 🔑 Insufficient privileges: Collaborator required

cgwalters · 2017-03-27T13:48:39Z

@rh-atomic-bot r+ ae279d0

rh-atomic-bot · 2017-03-27T13:48:44Z

⌛ Testing commit ae279d0 with merge 7c8f95c...

This mode is similar to bare-user, but does not store the permission, ownership (uid/gid) and xattrs in an xattr on the file objects in the repo. Additionally it stores symlinks as symlinks rather than as regular files+xattrs, like the bare mode. The later is needed because we can't store the is-symlink in the xattr. This means that some metadata is lost, such as the uid. When reading a repo like this we always report uid, gid as 0, and no xattrs, so unless this is true in the commit the resulting repository will not fsck correctly. However, it the main usecase of the repository is to check out with --user-mode, then no information is lost, and the repository can work on filesystems without xattrs (such as tmpfs). Closes: #750 Approved by: cgwalters

This adds to file permission masks the same bitmask that will be applied to file objects in bare-user* repos. This will be needed in the testsuite to ensure that the things we commit will be expressable in bare-user-only repos. Closes: #750 Approved by: cgwalters

This is somewhat complicated by such repos only properly supporting some subset of file metadata (uid/gid 0, etc). We fix this by always commiting with filters that make it work. Closes: #750 Approved by: cgwalters

rh-atomic-bot · 2017-03-27T13:59:34Z

☀️ Test successful - status-atomicjenkins
Approved by: cgwalters
Pushing 7c8f95c to master...

cgwalters reviewed Mar 22, 2017

View reviewed changes

rh-atomic-bot pushed a commit that referenced this pull request Mar 24, 2017

Add _ostree_repo_mode_is_bare helper

5d9a8e3

This cleans up some existing code, but it also allows us to later add new bare modes. Closes: #750 Approved by: cgwalters

alexlarsson added 3 commits March 26, 2017 11:00

Add _ostree_repo_mode_is_bare helper

e1592a5

This cleans up some existing code, but it also allows us to later add new bare modes.

commit: Add --canonical-permissions argument

2cbc605

This adds to file permission masks the same bitmask that will be applied to file objects in bare-user* repos. This will be needed in the testsuite to ensure that the things we commit will be expressable in bare-user-only repos.

Add basic tests for bare-user-only repo modes

ae279d0

This is somewhat complicated by such repos only properly supporting some subset of file metadata (uid/gid 0, etc). We fix this by always commiting with filters that make it work.

alexlarsson force-pushed the bare-user-only branch from 6008b97 to ae279d0 Compare March 26, 2017 09:10

rh-atomic-bot closed this in 612150f Mar 27, 2017

Add bare-user-only mode that works on non-xattrs filesystems #750

Add bare-user-only mode that works on non-xattrs filesystems #750

Conversation

alexlarsson commented Mar 22, 2017

alexlarsson commented Mar 22, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cgwalters commented Mar 22, 2017

alexlarsson commented Mar 22, 2017

alexlarsson commented Mar 22, 2017

cgwalters commented Mar 22, 2017

cgwalters commented Mar 22, 2017 • edited Loading

alexlarsson commented Mar 22, 2017

cgwalters commented Mar 22, 2017

alexlarsson commented Mar 22, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexlarsson commented Mar 24, 2017

alexlarsson commented Mar 24, 2017

cgwalters commented Mar 24, 2017

rh-atomic-bot commented Mar 24, 2017

cgwalters commented Mar 24, 2017

alexlarsson commented Mar 24, 2017

cgwalters commented Mar 24, 2017

alexlarsson commented Mar 24, 2017

rh-atomic-bot commented Mar 24, 2017

cgwalters commented Mar 24, 2017

alexlarsson commented Mar 24, 2017

rh-atomic-bot commented Mar 24, 2017

alexlarsson commented Mar 26, 2017

alexlarsson commented Mar 27, 2017

rh-atomic-bot commented Mar 27, 2017

cgwalters commented Mar 27, 2017

rh-atomic-bot commented Mar 27, 2017

rh-atomic-bot commented Mar 27, 2017

cgwalters commented Mar 22, 2017 •

edited

Loading