Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support commit, syncfs, /etc merge just before reboot #545

Closed
cgwalters opened this issue Oct 24, 2016 · 6 comments
Closed

Support commit, syncfs, /etc merge just before reboot #545

cgwalters opened this issue Oct 24, 2016 · 6 comments

Comments

@cgwalters
Copy link
Member

cgwalters commented Oct 24, 2016

PR: #1503


Migrating this from coreos/rpm-ostree#40 and https://bugzilla.gnome.org/show_bug.cgi?id=758088

To fix the "changes in /etc are lost" problem, we add the concept of a "staged deployment". Currently there are conceptually 3 parts to ostree_sysroot_deploy_tree() which is used for upgrades, etc.

  • Check out commit tree
  • Merge configuration
  • Adjust bootloader config

This changes to just be the first phase. We record the staged state in /run/ostree/staged-deployment. A new systemd service ostree-finalize-staged.service is activated when the new API ostree_sysroot_stage_tree() is used, and runs at shutdown time to do the configuration merge and bootloader.

@cgwalters cgwalters changed the title Handle files in /etc being deleted/changed during upgrades Support /etc merge just before reboot May 10, 2017
@dustymabe
Copy link
Contributor

We don't change the bootloader or do the /etc merge. Both of those would happen on shutdown, just before reboot.

does that mean we would properly handle the "power loss" case where I've "upgraded" but not yet reboot yet and i suffer a power loss?

@cgwalters
Copy link
Member Author

The short answer is "yes". I'm not sure exactly what your concern is, but the only power loss thing to be concerned about here is that if you get a system crash during the "final processing" phase where we do the /etc merge and bootloader swap, you'd be left in the current system, not the new one.

@dustymabe
Copy link
Contributor

ok cool. so if we lost power before reboot we'd simply stay in the state where we are booting into the old deployment on every boot. Would the user just rpm-ostree upgrade again to get to the newer deployment (again).

@cgwalters
Copy link
Member Author

The way I'd been thinking of implementing things, the update would still be "pending" when the system came back up, so doing a reboot would retry the /etc merge and bootloader swap.

@cgwalters cgwalters changed the title Support /etc merge just before reboot Support commit, syncfs, /etc merge just before reboot Sep 12, 2017
@cgwalters
Copy link
Member Author

I think we should also try to only do syncfs() at this same time too. This would be even nicer for rpm-ostree livefs style workflows.

The user-visible effect here though would be that it'd be possible for an operation that was applied live could be "undone" after a system crash.

Things get tricky though as we need to think about the flatpak (and other container) use cases. What degree of expected durability is there?

Perhaps scoping in the change to syncfs() would be too hard here and we should do the /etc merge first - particularly given that I think in the modern day we basically expect people to be using SSD for the base OS data, so write times and latency imposed by syncing are a lot less of a concern than the days of rotational storage.

cgwalters added a commit to cgwalters/rpm-ostree that referenced this issue Nov 29, 2017
I was playing with `--download-only` a bit with an eye to
having something like this be used by Cockpit/gnome-software instead
of what it's doing now, but a problem is that at the moment we
don't have a way to reflect the "changed" state back to clients.

This is a first step towards that by simply printing a different
message.

I think really to make all of this work more nicely though, including
supporting e.g. rpm database diffs, we are going to have to instead
work on the [pending deployment(ostreedev/ostree#545)
path.
cgwalters added a commit to cgwalters/rpm-ostree that referenced this issue Nov 29, 2017
I was playing with `--download-only` a bit with an eye to
having something like this be used by Cockpit/gnome-software instead
of what it's doing now, but a problem is that at the moment we
don't have a way to reflect the "changed" state back to clients.

This is a first step towards that by simply printing a different
message.

I think really to make all of this work more nicely though, including
supporting e.g. rpm database diffs, we are going to have to instead
work on the [pending deployment](ostreedev/ostree#545)
path.  That way we'll have done the depsolve, stored repo timestamps
etc.; we'll be able to accurately show what *did* change rather than
try to recreate what will happen on the next `rpm-ostree upgrade --cache-only`.
rh-atomic-bot pushed a commit to coreos/rpm-ostree that referenced this issue Nov 29, 2017
I was playing with `--download-only` a bit with an eye to
having something like this be used by Cockpit/gnome-software instead
of what it's doing now, but a problem is that at the moment we
don't have a way to reflect the "changed" state back to clients.

This is a first step towards that by simply printing a different
message.

I think really to make all of this work more nicely though, including
supporting e.g. rpm database diffs, we are going to have to instead
work on the [pending deployment](ostreedev/ostree#545)
path.  That way we'll have done the depsolve, stored repo timestamps
etc.; we'll be able to accurately show what *did* change rather than
try to recreate what will happen on the next `rpm-ostree upgrade --cache-only`.

Closes: #1118
Approved by: jlebon
@cgwalters
Copy link
Member Author

PR: #1503

cgwalters added a commit to cgwalters/ostree that referenced this issue Apr 6, 2018
Add API to write a deployment state to `/run/ostree/staged-deployment`,
along with a systemd service which runs at shutdown time.

This is a big change to the ostree model for hosts,
but it closes a longstanding set of bugs; many, many people have
hit the "losing changes in /etc" problem.  It also avoids
the other problem of racing with programs that modify `/etc`
such as LVM backups:
https://bugzilla.redhat.com/show_bug.cgi?id=1365297

We need this in particular to go to a full-on model for
automatically updated host systems where (like a dual-partition model)
everything is fully prepared and the reboot can be taken
asynchronously.

Closes: ostreedev#545
cgwalters added a commit to cgwalters/ostree that referenced this issue Apr 7, 2018
Add API to write a deployment state to `/run/ostree/staged-deployment`,
along with a systemd service which runs at shutdown time.

This is a big change to the ostree model for hosts,
but it closes a longstanding set of bugs; many, many people have
hit the "losing changes in /etc" problem.  It also avoids
the other problem of racing with programs that modify `/etc`
such as LVM backups:
https://bugzilla.redhat.com/show_bug.cgi?id=1365297

We need this in particular to go to a full-on model for
automatically updated host systems where (like a dual-partition model)
everything is fully prepared and the reboot can be taken
asynchronously.

Closes: ostreedev#545
cgwalters added a commit to cgwalters/ostree that referenced this issue Apr 12, 2018
Add API to write a deployment state to `/run/ostree/staged-deployment`,
along with a systemd service which runs at shutdown time.

This is a big change to the ostree model for hosts,
but it closes a longstanding set of bugs; many, many people have
hit the "losing changes in /etc" problem.  It also avoids
the other problem of racing with programs that modify `/etc`
such as LVM backups:
https://bugzilla.redhat.com/show_bug.cgi?id=1365297

We need this in particular to go to a full-on model for
automatically updated host systems where (like a dual-partition model)
everything is fully prepared and the reboot can be taken
asynchronously.

Closes: ostreedev#545
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants