ostree/summary: Generate an ostree-metadata ref when updating summary #1158

pwithnall · 2017-09-11T14:41:06Z

This is the new way of publishing repository metadata, rather than as
additional-metadata in the summary file. The use of an ostree-metadata
ref means that the metadata from multiple upstream collections is not
conflated when doing P2P mirroring of many repositories.

The new ref is only generated if the repository has a collection ID set.
The old summary file continues to be generated for backwards
compatibility (and because it continues to be the canonical ref →
checksum map for the repository).

The new code is only used if configured with --enable-experimental-api.

Includes unit tests.

Signed-off-by: Philip Withnall withnall@endlessm.com

pwithnall · 2017-09-11T14:41:28Z

The original introduction of ostree-metadata was discussed in PR #946.

pwithnall · 2017-09-12T15:10:58Z

@rh-atomic-bot, retest this please

cgwalters · 2017-09-12T20:17:43Z

src/ostree/ot-builtin-summary.c

+          g_autoptr(OstreeMutableTree) mtree = NULL;
+          g_autoptr(OstreeRepoFile) repo_file = NULL;
+          g_autoptr(GVariantDict) new_summary_commit_dict = NULL;
+          g_autoptr(GVariant) new_summary_commit = NULL;


We prefer declare-and-initialize for new code; doesn't have to be done in this patch but someone will probably do it later if not.

This code was copied from the almost-identical implementation I did in flatpak; so I’d prefer not to rewrite it on import.

cgwalters · 2017-09-12T20:18:51Z

src/ostree/ot-builtin-summary.c

+            return FALSE;
+
+          mtree = ostree_mutable_tree_new ();
+          if (!ostree_repo_write_dfd_to_mtree (repo, AT_FDCWD, ".", mtree, NULL, NULL, error))


Hmm? I don't think we should be relying on the process' cwd here, right? It looks like that cwd is the entire unit test directory at least from the tests.

Are you intending to include any content in the metadata ref? If not, the libarchive code has some "create empty directory" code.

Oops. The intention is to not include any content in the commit, correct. I can’t see anything immediately obvious for this in the libarchive code. Which code do you mean, exactly?

I need to fix flatpak to not use AT_FDCWD, "." too, once this is done.

It's this bit:

/* If we didn't import anything at all, and autocreation of parents * is enabled, automatically create a root directory. This is * useful primarily when importing Docker image layers, which can * just be metadata. */ if (opts->autocreate_parents && ostree_mutable_tree_get_metadata_checksum (mtree) == NULL) { glnx_unref_object GFileInfo *fi = g_file_info_new (); g_file_info_set_attribute_uint32 (fi, "unix::uid", 0); g_file_info_set_attribute_uint32 (fi, "unix::gid", 0); g_file_info_set_attribute_uint32 (fi, "unix::mode", DEFAULT_DIRMODE); g_autoptr(GFileInfo) mfi = NULL; (void)_ostree_repo_commit_modifier_apply (self, modifier, "/", fi, &mfi); if (!aic_ensure_parent_dir_with_file_info (&aictx, mtree, "/", mfi, NULL, cancellable, error)) goto out; }

Great, thanks. I’ve updated the patch and added a test.

I’ll fix flatpak tomorrow; need to head off now.

cgwalters · 2017-09-13T19:06:14Z

src/ostree/ot-builtin-summary.c

+          if (!ostree_repo_write_mtree (repo, mtree, (GFile **) &repo_file, NULL, error))
+            return FALSE;
+
+          if (!ostree_repo_write_commit (repo, old_ostree_metadata_checksum,


What's the rationale behind the commit having a parent? In what cases would one want old metadata?

I guess with this administrators could do ostree --repo=repo prune --only-branch=ostree-metadata --depth=0 when they wanted, but still.

@pwithnall Did you have an opinion on this issue? (Ah, I see the unit tests rely on it)

The unit tests don’t rely on it inasmuch as they test that the code implements it.

The rationale for metadata commits having a parent is that it allows the metadata history of the repository to be represented, at fairly low cost. As you say, admins can always prune it later if they want.

cgwalters · 2017-09-13T19:09:24Z

src/ostree/ot-builtin-summary.c

+      const char *collection_id = NULL;
+#endif  /* OSTREE_ENABLE_EXPERIMENTAL_API */
+
+      /* Write out a new metadata commit for the repository. */


Can you explain a bit what metadata is being included here? It looks it's basically a stub, but callers could use --add-metadata?

additional_metadata is passed into the vardict for new_summary_commit, so the stored metadata is identical to that in the summary file (plus the collection/ref binding keys). By default, that is indeed nothing, unless the user provides an --add-metadata option.

Hmm, but if the intention is to get rid of the summary file down the line, don't we want the list of refs/deltas?

The intention is not to get rid of the summary file down the line, since it remains a useful cache for the list of refs/deltas. The ostree-metadata ref is only replacing the additional metadata from the summary.

cgwalters · 2017-09-13T19:09:59Z

Is it OK if this waits till the next release cycle?

pwithnall · 2017-09-13T23:19:11Z

Is it OK if this waits till the next release cycle?

Sure, although we might carry it as a patch in our OSTree package until then, so if you have any architectural issues to raise with it, please do so now. I don’t want our OSTree package to end up diverging from upstream.

cgwalters · 2017-09-19T20:30:40Z

I'm going to mark this as WIP since it feels like it needs a bit of design. Are we agreed that this replaces the whole summary file down the line? Maybe we'll need infrastructure to re-synthesize something that looks summary-file like for the pull code?

One thing I'd hope we can achieve here is split up the metadata into separate chunks to scale better.

pwithnall · 2017-09-19T22:44:19Z

Are we agreed that this replaces the whole summary file down the line?

Nope, the intention is that this only replaces the additional metadata in the summary file. The summary file remains present, unsigned, as a list of the refs.

The deltas are currently stored in the additional metadata map (right? I haven’t checked), so I guess they end up moving to ostree-metadata. I’m more on the fence about that one, because the list of deltas is content which could vary between a repository and a mirror of that repository; just like the list of refs. Maybe I need to look into that aspect of it further.

pwithnall · 2017-09-20T12:53:00Z

(Would it help to have a call about this?)

cgwalters · 2017-09-22T14:07:24Z

One thing that occurred to me from our discussion is that if we want to deprecate the summary file, perhaps we need an ostree-summary-metadata ref which is distinct from the ostree-metadata which sounds more static?

pwithnall · 2017-09-22T16:20:20Z

One thing that occurred to me from our discussion is that if we want to deprecate the summary file, perhaps we need an ostree-summary-metadata ref which is distinct from the ostree-metadata which sounds more static?

What are you suggesting this ostree-summary-metadata ref stores? The list of other refs in the repository? Currently, I think I’d rather keep the summary file for that. However, I don’t want to get too sucked in to discussing the long-term future of the summary file in this PR, given that it’s still a bit nebulous. Open a new issue and we can plan there (and pull in Alex and anyone else interested)?

Signed-off-by: Philip Withnall <withnall@endlessm.com>

There is no error handling to do, so just return everywhere instead. Signed-off-by: Philip Withnall <withnall@endlessm.com>

This is the new way of publishing repository metadata, rather than as additional-metadata in the summary file. The use of an ostree-metadata ref means that the metadata from multiple upstream collections is not conflated when doing P2P mirroring of many repositories. The new ref is only generated if the repository has a collection ID set. The old summary file continues to be generated for backwards compatibility (and because it continues to be the canonical ref → checksum map for the repository). The new code is only used if configured with --enable-experimental-api. Includes unit tests. Signed-off-by: Philip Withnall <withnall@endlessm.com>

pwithnall · 2017-09-28T13:53:37Z

Rebased against master and squashed in the fixups. Ready for a second round of review.

cgwalters · 2017-10-02T13:39:13Z

src/ostree/ot-builtin-summary.c

+                {
+                  const char *key_id = *iter;
+
+                  if (!ostree_repo_sign_commit (repo,


Generating, signing, and committing in one go makes doing signing asynchronously harder. At least in Fedora releng does signing async. But this is something we can enhance later.

If opt_key_ids == NULL then signing won’t happen as part of this utility call.

cgwalters · 2017-10-02T13:39:39Z

@rh-atomic-bot r+ f126610

rh-atomic-bot · 2017-10-02T13:39:43Z

⌛ Testing commit f126610 with merge feeb354...

There is no error handling to do, so just return everywhere instead. Signed-off-by: Philip Withnall <withnall@endlessm.com> Closes: #1158 Approved by: cgwalters

This is the new way of publishing repository metadata, rather than as additional-metadata in the summary file. The use of an ostree-metadata ref means that the metadata from multiple upstream collections is not conflated when doing P2P mirroring of many repositories. The new ref is only generated if the repository has a collection ID set. The old summary file continues to be generated for backwards compatibility (and because it continues to be the canonical ref → checksum map for the repository). The new code is only used if configured with --enable-experimental-api. Includes unit tests. Signed-off-by: Philip Withnall <withnall@endlessm.com> Closes: #1158 Approved by: cgwalters

rh-atomic-bot · 2017-10-02T13:52:25Z

☀️ Test successful - status-atomicjenkins
Approved by: cgwalters
Pushing feeb354 to master...

When building the ostree-metadata branch (which only happens when configured with --enable-p2p), we are supposed to create empty commits which contain only metadata. However, the code to do this was wrong, and was instead pulling in all the files from the current working directory and committing them. Fix that code to actually create an empty commit. This could have been a fairly serious bug were it not for the fact that nobody’s using this code because it’s all experimental. Spotted as part of ostreedev/ostree#1158. Signed-off-by: Philip Withnall <withnall@endlessm.com>

When building the ostree-metadata branch (which only happens when configured with --enable-p2p), we are supposed to create empty commits which contain only metadata. However, the code to do this was wrong, and was instead pulling in all the files from the current working directory and committing them. Fix that code to actually create an empty commit. This could have been a fairly serious bug were it not for the fact that nobody’s using this code because it’s all experimental. Spotted as part of ostreedev/ostree#1158. Signed-off-by: Philip Withnall <withnall@endlessm.com> Closes: #1066 Approved by: alexlarsson

pwithnall self-assigned this Sep 11, 2017

pwithnall requested review from mbarnes and cgwalters September 11, 2017 14:41

cgwalters reviewed Sep 12, 2017

View reviewed changes

cgwalters reviewed Sep 13, 2017

View reviewed changes

cgwalters added for-after-release WIP and removed for-after-release labels Sep 14, 2017

pwithnall added 3 commits September 28, 2017 14:52

man: Fix a copypasta error in ostree-summary.xml

bfbe06d

Signed-off-by: Philip Withnall <withnall@endlessm.com>

ostree/summary: Eliminate redundant gotos from error handling path

b4cc86b

There is no error handling to do, so just return everywhere instead. Signed-off-by: Philip Withnall <withnall@endlessm.com>

pwithnall force-pushed the summary-update-ostree-metadata branch from 0a13fd6 to f126610 Compare September 28, 2017 13:53

cgwalters removed the WIP label Sep 28, 2017

cgwalters reviewed Oct 2, 2017

View reviewed changes

cgwalters added the homu/approved label Oct 2, 2017

rh-atomic-bot closed this in d1a5652 Oct 2, 2017

pwithnall mentioned this pull request Oct 2, 2017

common: Don’t commit arbitrary files into the ostree-metadata branch flatpak/flatpak#1066

Closed

pwithnall deleted the summary-update-ostree-metadata branch October 3, 2017 12:07

pwithnall mentioned this pull request Oct 4, 2017

common: Don’t commit arbitrary files into the ostree-metadata branch endlessm/flatpak#56

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ostree/summary: Generate an ostree-metadata ref when updating summary #1158

ostree/summary: Generate an ostree-metadata ref when updating summary #1158

pwithnall commented Sep 11, 2017

pwithnall commented Sep 11, 2017

pwithnall commented Sep 12, 2017

cgwalters Sep 12, 2017

pwithnall Sep 13, 2017

cgwalters Sep 12, 2017

pwithnall Sep 13, 2017

cgwalters Sep 13, 2017

pwithnall Sep 13, 2017

pwithnall Sep 13, 2017

cgwalters Sep 13, 2017

cgwalters Oct 2, 2017

pwithnall Oct 2, 2017

cgwalters Sep 13, 2017

pwithnall Sep 13, 2017

cgwalters Sep 19, 2017

pwithnall Sep 19, 2017

cgwalters commented Sep 13, 2017

pwithnall commented Sep 13, 2017

cgwalters commented Sep 19, 2017

pwithnall commented Sep 19, 2017

pwithnall commented Sep 20, 2017

cgwalters commented Sep 22, 2017

pwithnall commented Sep 22, 2017

pwithnall commented Sep 28, 2017

cgwalters Oct 2, 2017

pwithnall Oct 2, 2017

cgwalters commented Oct 2, 2017

rh-atomic-bot commented Oct 2, 2017

rh-atomic-bot commented Oct 2, 2017

ostree/summary: Generate an ostree-metadata ref when updating summary #1158

ostree/summary: Generate an ostree-metadata ref when updating summary #1158

Conversation

pwithnall commented Sep 11, 2017

pwithnall commented Sep 11, 2017

pwithnall commented Sep 12, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cgwalters commented Sep 13, 2017

pwithnall commented Sep 13, 2017

cgwalters commented Sep 19, 2017

pwithnall commented Sep 19, 2017

pwithnall commented Sep 20, 2017

cgwalters commented Sep 22, 2017

pwithnall commented Sep 22, 2017

pwithnall commented Sep 28, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cgwalters commented Oct 2, 2017

rh-atomic-bot commented Oct 2, 2017

rh-atomic-bot commented Oct 2, 2017