Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs/atomic-rollbacks: Add a section on rollbacks #3171

Merged
merged 1 commit into from
Feb 13, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
nav_order: 19
nav_order: 190
---

# Contributing
Expand Down
2 changes: 1 addition & 1 deletion docs/README-historical.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
nav_order: 99
nav_order: 990
title: Historical OSTree README
---

Expand Down
2 changes: 1 addition & 1 deletion docs/adapting-existing.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
nav_order: 6
ericcurtin marked this conversation as resolved.
Show resolved Hide resolved
nav_order: 70
---

# Adapting existing mainstream distributions
Expand Down
176 changes: 176 additions & 0 deletions docs/atomic-rollbacks.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,176 @@
---
nav_order: 60
---

# Atomic Rollbacks
{: .no_toc }

1. TOC
{:toc}

## Automatic rollbacks

See [greenboot](https://github.com/fedora-iot/greenboot/blob/main/README.md) for information on automatic rollbacks and how to integrate
without your bootloader.

## Manual rollbacks

Ostree writes bootloader entries that are interpreted by the bootloader. To
manually rollback, for bootloaders such as GRUB and syslinux that have an
interactive UI, it is possible to select a previous boot entry. In the case of
an Android bootloader, a slot switch may be triggererd using an AB switching
tool. This may be useful for testing purposes.

## Rollbacks

```
+------------------------------------------------+
+------------------+ | |
| | | |
| | | |
| | | (ostree:0) latest (multi-user.target) |
| | | |
| Bootloader |--->+ root |
| | | |
| | | (ostree:1) latest - 1 (multi-user.target) |
| | | |
| | | |
+------------------+ | |
+------------------------------------------------+
```

Bootloaders have multiple boot entries to choose from after upgrade. On
rollback, the bootloader will boot the "latest - 1" version, rather than the
latest version of the OS.

## Alternate rollback techniques

Below is an alternate technique to traditional AB switching that can be used.
On rollback, an alternative boot target is used, rather than booting as
default.target.

```
+------------------------------------------------+
+------------------+ | |
| | | |
| | | |
| | | (ostree:0) latest (multi-user.target) |
| | | |
| Bootloader |--->+ root |
| | | |
| | | (ostree:1) latest - 1 (rescue.target) |
| | | |
| | | |
+------------------+ | |
+------------------------------------------------+
```

In this case, instead of rolling back to an older version, we also boot
into an alternate systemd boot target. Here we will describe how you can put
togther an alternate systemd boot target, using the built-in rescue.target as
an example.

Below is a rescue.service file, it essentially executes systemd-sulogin-shell
rescue when this service is activated.

rescue.service:

```
# SPDX-License-Identifier: LGPL-2.1-or-later
#
# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.

[Unit]
Description=Rescue Shell
Documentation=man:sulogin(8)
DefaultDependencies=no
Conflicts=shutdown.target
After=sysinit.target plymouth-start.service
Before=shutdown.target

[Service]
Environment=HOME=/root
WorkingDirectory=-/root
ExecStartPre=-/usr/bin/plymouth --wait quit
ExecStart=-/usr/lib/systemd/systemd-sulogin-shell rescue
Type=idle
StandardInput=tty-force
StandardOutput=inherit
StandardError=inherit
KillMode=process
IgnoreSIGPIPE=no
SendSIGHUP=yes
```

Below is a rescue.target file, it is reached once rescue.service is complete.

rescue.target:

```
# SPDX-License-Identifier: LGPL-2.1-or-later
#
# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.

[Unit]
Description=Rescue Mode
Documentation=man:systemd.special(7)
Requires=sysinit.target rescue.service
After=sysinit.target rescue.service
AllowIsolate=yes
```

This is a simple bash script, it checks whether `ostree admin status -D` is
`not-default` and if it is, it notifies systemd to alternatively boot into
rescue.target.

In the happy path, when we have booted the latest version
`ostree admin status -D` would output `default`.

ostree-rollback-to-rescue:

```
#!/usr/bin/bash

set -euo pipefail

if [ "$(ostree admin status -D)" = "not-default" ]; then
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks fine, that said...now that we've done all this I am still coming around to the idea that ostree should automatically inject something like ostree.fallback=/path/to/root for all non-default entries, that way we can reliably detect this from the initramfs and properly set up the default target from the initramfs.

Anyways, something to just consider for later.

Copy link
Collaborator Author

@ericcurtin ericcurtin Feb 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We normally fallback at the bootloader level right? So if we have loaded a kernel+initrd+cmdline from a fallback boot entry the ostree= karg would have the fallback root and ostree.fallback= would have the next fallback if that makes sense.

Also... cmdlines are generated server-side, this wouldn't be able to account for clients that missed an upgrade.

You can skip an upgrade in ostree right, if you haven't upgraded in a while?

That's why it's kinda nice to have the /boot/loader data client-side, fallbacks seem like something that should be client-side determined. Because each client may have a different upgrade path (and different fallback versions, or the fallback could be on A on some clients and B on others).

exec systemctl --no-block isolate rescue.target
fi
```

This is a systemd service file that runs ostree-rollback-to-rescue early in the
boot sequence, it is essential that this service is run early to ensure we
don't execute a full boot sequence, hence options `DefaultDependencies=no` and
`Before=` are used.

ostree-rollback-to-rescue.service

```
[Unit]
Description=OSTree rollback to rescue
DefaultDependencies=no
OnFailure=emergency.target
OnFailureJobMode=replace-irreversibly
After=initrd-root-fs.target initrd-fs.target initrd.target boot.mount
Before=cryptsetup.target integritysetup.target remote-fs.target slices.target swap.target veritysetup.target

[Service]
Type=oneshot
ExecStart=/usr/sbin/ostree-rollback-to-rescue

[Install]
WantedBy=sysinit.target
```

###### Licensing for this document:
`SPDX-License-Identifier: (CC-BY-SA-3.0 OR GFDL-1.3-or-later)`
2 changes: 1 addition & 1 deletion docs/atomic-upgrades.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
nav_order: 5
nav_order: 50
---

# Atomic Upgrades
Expand Down
2 changes: 1 addition & 1 deletion docs/authenticated-repos.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
nav_order: 9
nav_order: 100
---

# Handling access to authenticated remote repositories
Expand Down
2 changes: 1 addition & 1 deletion docs/bootloaders.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
nav_order: 11
nav_order: 120
---

# Bootloaders
Expand Down
2 changes: 1 addition & 1 deletion docs/buildsystem-and-repos.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
nav_order: 8
nav_order: 90
---

# Writing a buildsystem and managing repositories
Expand Down
2 changes: 1 addition & 1 deletion docs/composefs.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
nav_order: 10
nav_order: 110
---

# Using composefs with OSTree
Expand Down
2 changes: 1 addition & 1 deletion docs/contributing-tutorial.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
nav_order: 20
nav_order: 200
---

# OSTree Contributing Tutorial
Expand Down
2 changes: 1 addition & 1 deletion docs/deployment.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
nav_order: 4
nav_order: 40
---

# Deployments
Expand Down
2 changes: 1 addition & 1 deletion docs/formats.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
nav_order: 7
nav_order: 80
---

# OSTree data formats
Expand Down
2 changes: 1 addition & 1 deletion docs/ima.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
nav_order: 10
nav_order: 110
---

# Using Linux IMA with OSTree
Expand Down
2 changes: 1 addition & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
nav_order: 1
nav_order: 10
---

# libostree
Expand Down
2 changes: 1 addition & 1 deletion docs/introduction.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
nav_order: 2
nav_order: 20
---

# OSTree Overview
Expand Down
2 changes: 1 addition & 1 deletion docs/related-projects.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
nav_order: 10
nav_order: 110
---

# Related Projects
Expand Down
2 changes: 1 addition & 1 deletion docs/repo.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
nav_order: 3
nav_order: 30
---

# Anatomy of an OSTree repository
Expand Down
2 changes: 1 addition & 1 deletion docs/repository-management.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
nav_order: 9
nav_order: 100
---

# Managing content in OSTree repositories
Expand Down
2 changes: 1 addition & 1 deletion docs/var.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
nav_order: 6
nav_order: 70
---

# OSTree and /var handling
Expand Down
Loading