Add concept of "staged" deployment #1503

cgwalters · 2018-03-18T21:06:10Z

Add API to write a deployment state to /run/ostree/staged-deployment,
along with a systemd service which runs at shutdown time.

Just compile tested, still WIP.

For: coreos/rpm-ostree#40

jlebon

Only did a high-level overview so far, but looks nice! 👍

jlebon · 2018-03-19T13:11:38Z

src/libostree/ostree-sysroot-deploy.c

+ * ostree_sysroot_stage_tree() API.
+ */
+gboolean
+ostree_sysroot_deploy_tree (OstreeSysroot     *self,


Maybe let's start by splitting out this initialize + finalize separation in a prep PR?

rh-atomic-bot · 2018-03-19T17:57:59Z

☔ The latest upstream changes (presumably 01717d7) made this pull request unmergeable. Please resolve the merge conflicts.

rh-atomic-bot · 2018-03-20T20:09:25Z

💥 Invalid .papr.yml: file could not be parsed as valid YAML.

cgwalters · 2018-03-20T20:41:15Z

Depends: https://src.fedoraproject.org/rpms/ostree/pull-request/8

rh-atomic-bot · 2018-03-21T16:03:01Z

☔ The latest upstream changes (presumably 22cd178) made this pull request unmergeable. Please resolve the merge conflicts.

cgwalters · 2018-03-22T17:18:08Z

TODO:

Clean up the story around kargs
Determine whether staged deployment is part of _get_deployments(). Tradeoffs both ways. If not (as currently) then all tools will need to learn to query for it.
tests

rh-atomic-bot · 2018-03-22T21:34:48Z

☔ The latest upstream changes (presumably 1f3f657) made this pull request unmergeable. Please resolve the merge conflicts.

rh-atomic-bot · 2018-03-26T14:25:53Z

☔ The latest upstream changes (presumably 7ec3d06) made this pull request unmergeable. Please resolve the merge conflicts.

cgwalters · 2018-04-05T18:52:52Z

Sigh, needed to move it into the main /usr/bin/ostree as that's install_exec_t which is what we need to do deployments.

peterbaouoft

Looks really nice, learned a lot from just reading it :D. Some minor comments and questions :p.

peterbaouoft · 2018-04-06T03:39:05Z

src/libostree/ostree-sysroot-deploy.c

@@ -2388,6 +2398,7 @@ ostree_sysroot_deploy_tree (OstreeSysroot     *self,
    return FALSE;

  _ostree_deployment_set_bootcsum (new_deployment, kernel_layout->bootcsum);
+  _ostree_deployment_set_bootconfig_from_kargs (new_deployment, override_kernel_argv);


Hmm, this seems to be also done line #2404 to #2417 (the one right below)?

Yep, that was all duplicate code. Fixed, thanks!

peterbaouoft · 2018-04-06T03:48:06Z

src/libostree/ostree-sysroot-deploy.c

                                                  cancellable, error))
        return FALSE;
    }

-  /* Don't fsync here, as we assume that's all done in
-   * ostree_sysroot_write_deployments().


Minor: do we still want to keep this comment for non-staged case? /me assuming we are still supporting "non-staged" deployments (the old 3 way merge way)?

peterbaouoft · 2018-04-06T03:55:18Z

src/libostree/ostree-sysroot-deploy.c

+    g_variant_builder_add (builder, "{sv}", "kargs",
+                           g_variant_new_strv ((const char *const*)override_kernel_argv, -1));
+
+  char *dnbuf = strdupa (_OSTREE_SYSROOT_RUNSTATE_STAGED);


Question: I am not too familiar with "/run" directories, wondering any reasons / benefits to store data there? :-) /me assuming we are writing deployment info into that location.

The basic idea of /run is it's guaranteed to go away if the system reboots (it's a tmpfs, stored only in memory). Which I think helps simplify the logic around failure cases. I feel like we only want to try to process staging data once rather than having it be persistent.

peterbaouoft · 2018-04-06T03:57:37Z

src/libostree/ostree-sysroot-deploy.c

+                               GCancellable  *cancellable,
+                               GError       **error)
+{
+  /* The service has a ConditionPathExists=/run/ostree/staged-deployment */


Minor: seems like /run/ostree/staged-deployment is not there in the service file. Only ConditionPathExists=/run/ostree-booted seems to be there?

That's not minor, that's a bug! The service would fail if there was no staged deployments. Fixed.

peterbaouoft · 2018-04-06T04:10:46Z

src/libostree/ostree-sysroot-deploy.c

+
+  /* TODO: Proxy across flags too? */
+  OstreeSysrootSimpleWriteDeploymentFlags flags = 0;
+  if (!ostree_sysroot_simple_write_deployment (self, ostree_deployment_get_osname (self->staged_deployment),


Question/minor: noticed in rpm-ostree, seems like we had rpmostree_syscore_write_deployment to add some flags for livefs and rollback, do we need to consider a bit here to make it easier to integrate this into rpm-ostree in the future?(if needed :p)

I think you're right that integrating this nicely will require pulling down some of that logic.

peterbaouoft · 2018-04-06T04:38:38Z

src/libostree/ostree-sysroot-cleanup.c

+  if (self->staged_deployment)
+    {
+      char *deployment_path = ostree_sysroot_get_deployment_dirpath (self, self->staged_deployment);
+      g_hash_table_replace (active_deployment_dirs, deployment_path, deployment_path);


Question: wanted to confirm, we don't include bootcsum here because the "staged deployment" is not ready for boot yet? ( i.e the merge and boot entries weren't ready)

Nope, that was an oversight. One not revealed because we weren't installing a staged deployment with a new boot checksum.

I have kind of been going back and forth on whether /boot should be modified during staging or finalization. Currently it's during staging.

jlebon · 2018-04-09T16:07:09Z

src/boot/ostree-deploy-staged.service

+# https://lists.freedesktop.org/archives/systemd-devel/2018-March/040557.html
+[Unit]
+Description=OSTree Deploy Staged
+ConditionPathExists=/run/ostree-booted


I see we do tolerate no staged deployments in _ostree_sysroot_deploy_staged, though we might as well short-circuit this from here and add a ConditionPathExists=/run/ostree/staged-deployment here too, no? Making it explicit also makes it easier to discover how these pieces fit together.

/after reading more code

Oh right, this won't work with the multi-user.target approach since it's evaluated on startup. Though it should work if we use shutdown.target, no? (Related to next comment).

Yeah, can't do that, see https://lists.freedesktop.org/archives/systemd-devel/2018-March/040557.html

jlebon · 2018-04-09T16:20:00Z

Makefile-boot.am

@@ -39,7 +39,7 @@ endif

 if BUILDOPT_SYSTEMD
 systemdsystemunit_DATA = src/boot/ostree-prepare-root.service \
-	src/boot/ostree-remount.service
+	src/boot/ostree-remount.service src/boot/ostree-deploy-staged.service


This is slightly bikeshed, though it feels like it would be more appropriate to name the service something like ostree-finalize-staged.service. (And the private function _ostree_sysroot_finalize_staged).

My reasoning is that a staged deployment is in some ways already deployed, right? And I see the rest of the new APIs/CLIs reflect that as well (e.g. ostree_sysroot_get_staged_deployment, and ostree admin deploy --stage).

jlebon · 2018-04-09T16:33:15Z

src/boot/ostree-deploy-staged.service

+[Service]
+Type=oneshot
+RemainAfterExit=yes
+ExecStop=/usr/bin/ostree admin deploy-staged


Hmm, this seems odd to me. I'm guessing the idea here is that the Before=multi-user.target will ensure that most services will have been shut down, assuming services are stopped in reverse order. Is this the canonical way to run on shutdown? Why not WantedBy=shutdown.target?

jlebon · 2018-04-09T17:15:17Z

src/libostree/ostree-sysroot-deploy.c

@@ -808,6 +809,7 @@ static gboolean
 write_origin_file_internal (OstreeSysroot         *sysroot,
                            OstreeSePolicy        *sepolicy,
                            OstreeDeployment      *deployment,
+                            gboolean               if_not_exists,


I'm confused why we need this. Why not just unconditionally overwrite it in finalize? Then even in the stage path, we're sure of using the right label for it post-merge.

jlebon · 2018-04-09T17:17:41Z

src/libostree/ostree-sysroot-deploy.c

+static GVariant *
+serialize_deployment_to_variant (OstreeDeployment *deployment)
+{
+  g_autoptr(GVariantBuilder) builder = g_variant_builder_new ((GVariantType*)"a{sv}");


Should we stick with g_auto where we can for good form?

jlebon · 2018-04-09T17:21:24Z

src/libostree/ostree-sysroot-deploy.c

+  g_autoptr(GVariantBuilder) builder = g_variant_builder_new ((GVariantType*)"a{sv}");
+  g_autofree char *name =
+    g_strdup_printf ("%s.%d", ostree_deployment_get_csum (deployment),
+                     ostree_deployment_get_deployserial (deployment));


How about keeping those separate to make deconstruction lower down easier?

Eh, we already need to parse them from the filesystem and had a function for it.

jlebon · 2018-04-09T17:30:06Z

src/libostree/ostree-sysroot-deploy.c

+                           g_variant_new_strv ((const char *const*)override_kernel_argv, -1));
+
+  char *dnbuf = strdupa (_OSTREE_SYSROOT_RUNSTATE_STAGED);
+  const char *parent = dirname (dnbuf);


Minor: any reason not to collapse these two lines? (i.e. dirname (strdupa (...)))

jlebon · 2018-04-09T17:34:55Z

src/libostree/ostree-sysroot-deploy.c

+  g_autoptr(OstreeDeployment) deployment = NULL;
+  if (!sysroot_initialize_deployment (self, osname, revision, origin, override_kernel_argv,
+                                      &deployment, cancellable, error))
+    return FALSE;


Hmm, I wonder here if we should do something if there's a deployment already staged. E.g. in rpm-ostree, if we have a staged auto-updates policy, we could just continuously be staging newer updates as we see them right? I guess we can put the cleanup there, though it does feel like someone calling ostree_sysroot_stage_tree expects it to implicitly delete a previously staged deployment, rather than having them pile up.

jlebon · 2018-04-09T17:43:46Z

src/libostree/ostree-sysroot.c

-  if (self->ostree_booted && self->root_is_sysroot
-      && !self->booted_deployment)
+  const gboolean root_is_ostree_booted =
+    self->ostree_booted && self->root_is_sysroot;


At this point, we might as well make replace instances of those with a new field.

jlebon · 2018-04-09T17:49:38Z

src/ostree/ot-admin-builtin-deploy.c

-  if (opt_retain)
-    flags |= OSTREE_SYSROOT_SIMPLE_WRITE_DEPLOYMENT_FLAGS_RETAIN;
-  else
+  if (opt_stage)
    {


Let's just error out here if any of the --retain opts are given?

cgwalters · 2018-04-11T17:37:08Z

Pushed a fixup ⬆️ which addresses a lot of comments (but not all). In the meantime split out prep patches into #1535

rh-atomic-bot · 2018-04-11T19:12:49Z

☔ The latest upstream changes (presumably b9fc3ea) made this pull request unmergeable. Please resolve the merge conflicts.

Add API to write a deployment state to `/run/ostree/staged-deployment`, along with a systemd service which runs at shutdown time. This is a big change to the ostree model for hosts, but it closes a longstanding set of bugs; many, many people have hit the "losing changes in /etc" problem. It also avoids the other problem of racing with programs that modify `/etc` such as LVM backups: https://bugzilla.redhat.com/show_bug.cgi?id=1365297 We need this in particular to go to a full-on model for automatically updated host systems where (like a dual-partition model) everything is fully prepared and the reboot can be taken asynchronously. Closes: ostreedev#545

jlebon · 2018-04-12T14:30:16Z

Let's do this!
@rh-atomic-bot r+ 7adfe24

rh-atomic-bot · 2018-04-12T14:55:15Z

⌛ Testing commit 7adfe24 with merge eb506c7...

rh-atomic-bot · 2018-04-12T15:29:41Z

☀️ Test successful - status-atomicjenkins
Approved by: jlebon
Pushing eb506c7 to master...

Followup to: ostreedev#1503 After starting some more work on on this in rpm-ostree, it is actually simpler if the staged deployment just shows up in the list. It's effectively opt-in today; down the line we may make it the default, but I worry about breaking things that e.g. assume they can mutate the deployment before rebooting and have `/etc` already merged. There's not that many things in libostree that iterate over the deployment list. The biggest change here is around the `ostree_sysroot_write_deployments_with_options` API. I initially tried hard to support a use case like "push a rollback" while retaining the staged deployment, but everything gets very messy because that function truly is operating on the bootloader list. For now what I settled on is to just discard the staged deployment; down the line we can enhance things. Where we then have some new gymnastics is around implementing the finalization; we need to go to some effort to pull the staged deployment out of the list and mark it as unstaged, and then pass it down to `write_deployments()`.

Followup to: #1503 After starting some more work on on this in rpm-ostree, it is actually simpler if the staged deployment just shows up in the list. It's effectively opt-in today; down the line we may make it the default, but I worry about breaking things that e.g. assume they can mutate the deployment before rebooting and have `/etc` already merged. There's not that many things in libostree that iterate over the deployment list. The biggest change here is around the `ostree_sysroot_write_deployments_with_options` API. I initially tried hard to support a use case like "push a rollback" while retaining the staged deployment, but everything gets very messy because that function truly is operating on the bootloader list. For now what I settled on is to just discard the staged deployment; down the line we can enhance things. Where we then have some new gymnastics is around implementing the finalization; we need to go to some effort to pull the staged deployment out of the list and mark it as unstaged, and then pass it down to `write_deployments()`. Closes: #1539 Approved by: jlebon

Prep for ostreedev/ostree#1503 which will add `/usr/lib/ostree/ostree-deploy-staged`.

cgwalters mentioned this pull request Mar 18, 2018

Have upgrade only do /etc merge just before reboot coreos/rpm-ostree#40

Closed

jlebon reviewed Mar 19, 2018

View reviewed changes

jlebon added the WIP label Mar 19, 2018

cgwalters mentioned this pull request Mar 19, 2018

Support commit, syncfs, /etc merge just before reboot #545

Closed

cgwalters force-pushed the deployment-pending branch from 49edd1e to c68f28f Compare March 20, 2018 12:34

cgwalters force-pushed the deployment-pending branch from 3109757 to feca528 Compare March 21, 2018 20:43

cgwalters mentioned this pull request Mar 23, 2018

Two deployment stage prep patches #1514

Closed

cgwalters force-pushed the deployment-pending branch from feca528 to 009dec7 Compare March 25, 2018 20:11

cgwalters force-pushed the deployment-pending branch 2 times, most recently from 08cb290 to e2838e7 Compare March 27, 2018 14:32

jlebon mentioned this pull request Mar 29, 2018

Add "ex-stage" automatic update policy coreos/rpm-ostree#1321

Closed

cgwalters force-pushed the deployment-pending branch from e2838e7 to cda2b06 Compare April 5, 2018 18:49

peterbaouoft reviewed Apr 6, 2018

View reviewed changes

cgwalters force-pushed the deployment-pending branch from cda2b06 to 164481d Compare April 6, 2018 22:02

cgwalters removed the WIP label Apr 6, 2018

cgwalters changed the title ~~WIP: Add concept of "staged" deployment~~ Add concept of "staged" deployment Apr 6, 2018

cgwalters force-pushed the deployment-pending branch 2 times, most recently from a81672c to f56d41d Compare April 7, 2018 15:52

jlebon reviewed Apr 9, 2018

View reviewed changes

cgwalters force-pushed the deployment-pending branch from 0ba40db to 7adfe24 Compare April 12, 2018 13:21

jlebon added the homu/approved label Apr 12, 2018

rh-atomic-bot closed this in eb506c7 Apr 12, 2018

cgwalters mentioned this pull request Apr 13, 2018

lib/sysroot: Move staged into deployment list, rework handling #1539

Closed

cgwalters mentioned this pull request Apr 27, 2018

rework grub2 backend for new fedora grub model #717

Closed

cgwalters mentioned this pull request Jun 16, 2018

Add ostree checkout overwrite mode "--update" #1408

Open

2 tasks

LorbusChris pushed a commit to LorbusChris/ostree-spec that referenced this pull request Oct 23, 2018

Extend glob for new helper binaries

c412307

Prep for ostreedev/ostree#1503 which will add `/usr/lib/ostree/ostree-deploy-staged`.

jlebon mentioned this pull request Jan 29, 2019

Throttled update rollouts coreos/fedora-coreos-tracker#83

Closed

ricardosalveti mentioned this pull request Feb 27, 2019

OTA-2135: Aktualizr's config and recipe to auto reboot just after update advancedtelematic/meta-updater#486

Merged

Add concept of "staged" deployment #1503

Add concept of "staged" deployment #1503

Conversation

cgwalters commented Mar 18, 2018

jlebon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rh-atomic-bot commented Mar 19, 2018

rh-atomic-bot commented Mar 20, 2018

cgwalters commented Mar 20, 2018

rh-atomic-bot commented Mar 21, 2018

cgwalters commented Mar 22, 2018

rh-atomic-bot commented Mar 22, 2018

rh-atomic-bot commented Mar 26, 2018

cgwalters commented Apr 5, 2018

peterbaouoft left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cgwalters commented Apr 11, 2018

rh-atomic-bot commented Apr 11, 2018

jlebon commented Apr 12, 2018

rh-atomic-bot commented Apr 12, 2018

rh-atomic-bot commented Apr 12, 2018