Introduce `ex livefs` for layering newly added packages 🗱 #652

cgwalters · 2017-03-02T03:48:35Z

WIP due to missing tests and docs.

But I've been playing with this interactively, and it's at a useful place
already. The primary target here is supporting live addition of new packages,
while also allowing people to do completely arbitrary replacement if that's
what they want.

Depends:

Closes: #639

cgwalters · 2017-03-02T03:49:00Z

Demo:

[root@localhost ~]# rpm-ostree install cowsay
...
Transaction complete; bootconfig swap: no deployment count change: 0
Added:
  cowsay-3.04-4.fc25.noarch
...
  perl-Carp-1.40-365.fc25.noarch
  perl-Data-Dumper-2.161-1.fc25.x86_64
  (and like 5 million other perl deps)
...
Run "systemctl reboot" to start a reboot
[root@localhost ~]# rpm-ostree status
State: idle
Deployments:
  hacking
           Timestamp: 2017-03-02 03:39:47
          BaseCommit: 2ccf10b97719203d8b87c062e8dffb17b24dd13415e56c7e17200433f0d3749b
              Commit: bb78eb713479299500edde8c5a35c9161ea6474da4c198bc9ad2bcaf46ad080d
              OSName: fedora-atomic
     LayeredPackages: cowsay

● hacking
           Timestamp: 2017-03-02 03:37:08
              Commit: 2ccf10b97719203d8b87c062e8dffb17b24dd13415e56c7e17200433f0d3749b
              OSName: fedora-atomic
[root@localhost ~]# rpm-ostree livefs
notice: livefs is an experimental command and subject to change.
warning: livefs would change configuration in /etc
error: livefs update would perform changes beyond new files in /usr, and partial updates not enabled
[root@localhost ~]# rpm-ostree livefs --help
notice: livefs is an experimental command and subject to change.
Usage:
  rpm-ostree livefs [OPTION...] - Apply pending deployment changes to booted deployment

Help Options:
  -h, --help            Show help options

Application Options:
  -n, --dry-run         Only perform analysis, do not make changes
  --allow-replace       Allow replacing existing files
  --accept-partial      Continue even on partial updates
  --sysroot=SYSROOT     Use system root SYSROOT (default: /)
  --peer                Force a peer-to-peer connection instead of using the system message bus
  --version             Print version information and exit

[root@localhost ~]# rpm-ostree livefs --accept-partial
notice: livefs is an experimental command and subject to change.
warning: livefs would change configuration in /etc
Overlaying /usr... done
[root@localhost ~]# cowsay hi
 ____
< hi >
 ----
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||
[root@localhost ~]# rpm -q cowsay
cowsay-3.04-4.fc25.noarch
[root@localhost ~]#

giuseppe · 2017-03-02T23:13:07Z

This looks great!

It will be very useful for projectatomic/atomic#767

giuseppe · 2017-03-03T10:59:26Z

will it be possible to support removal of an rpm as well?

I am asking as that would be helpful for supporting containers update here.

rh-atomic-bot · 2017-03-16T20:23:38Z

☔ The latest upstream changes (presumably 710ea30) made this pull request unmergeable. Please resolve the merge conflicts.

cgwalters · 2017-03-17T12:47:30Z

src/daemon/rpmostreed-utils.c:673:70: warning: comparison of unsigned expression < 0 is always false [-Wtautological-compare]

Nice, thanks clang!

cgwalters · 2017-03-20T19:17:39Z

🆕 commits.

Dependency on sysroot: Add ostree_sysroot_write_deployments_with_options() ostreedev/ostree#745 to make "push a rollback" work better
Some local test hacks
More internal work; needed to bump mtime to get status correct

Also, a fun twist: ostree admin unlock + livefs is pretty broken because lowerdir mutation is undefined. This kind of breaks my dev workflow for this 😦. I'm thinking about a variant of ostree admin unlock which is more like unshare -m && mount overlayfs / usr && exec rpm-ostreed or so. Will play.

rh-atomic-bot · 2017-03-20T21:05:04Z

☔ The latest upstream changes (presumably 19b093e) made this pull request unmergeable. Please resolve the merge conflicts.

rh-atomic-bot · 2017-03-22T16:02:39Z

☔ The latest upstream changes (presumably 1c42102) made this pull request unmergeable. Please resolve the merge conflicts.

rh-atomic-bot · 2017-03-27T19:22:53Z

☔ The latest upstream changes (presumably 75d91a1) made this pull request unmergeable. Please resolve the merge conflicts.

rh-atomic-bot · 2017-05-11T18:24:08Z

⌛ Testing commit 4f3a70d with merge f3b71ca...

There are a few different use cases here. First, for layering new packages, there's no good reason for us to force a reboot. Second, we want some support for cherry-picking security updates and allowing admins to restart services. Finally, at some point we should offer support for entirely replacing the running tree if that's what the user wants. Until now we've been very conservative, but there's a spectrum here. In particular, this patch changes things so we push a rollback before we start doing anything live. I think in practice, many use cases would be totally fine with doing most changes live, and falling back to the rollback if something went wrong. This initial code drop *only* supports live layering of new packages. However, a lot of the base infrastructure is laid for future work. For now, this will be classified as an experimental feature, hence `ex livefs`. Part of: #639 Closes: #652 Approved by: jlebon

rh-atomic-bot · 2017-05-11T18:53:31Z

💔 Test failed - status-atomicjenkins

jlebon · 2017-05-11T19:41:48Z

Hmm, looks like the /etc checkout isn't quite working:

+ vm_cmd cat /etc/test-livefs-with-etc.conf
+ ssh -o User=root -o ControlMaster=auto -o ControlPath=/var/tmp/checkout/ssh-vmcheck1.sock -o ControlPersist=yes vmcheck1 cat /etc/test-livefs-with-etc.conf
cat: /etc/test-livefs-with-etc.conf: No such file or directory
+ _cleanup_tmpdir
+ test -z ''
+ test -f /var/tmp/checkout/tests/vmcheck/test.zWXsLD/.test
+ rm /var/tmp/checkout/tests/vmcheck/test.zWXsLD -rf
FAIL

jlebon · 2017-05-11T19:42:54Z

tests/vmcheck/test-livefs.sh

+assert_livefs_ok
+vm_rpmostree ex livefs
+vm_cmd rpm -q foo test-livefs-with-etc > rpmq.txt
+assert_file_has_content rpmq.txt foo-1.0-1 test-livefs-with-etc-1.0-1


Actually, looking at the test output now, I just realized assert_file_has_content doesn't actually handle arguments past $2.

For a future patch.

There are a few different use cases here. First, for layering new packages, there's no good reason for us to force a reboot. Second, we want some support for cherry-picking security updates and allowing admins to restart services. Finally, at some point we should offer support for entirely replacing the running tree if that's what the user wants. Until now we've been very conservative, but there's a spectrum here. In particular, this patch changes things so we push a rollback before we start doing anything live. I think in practice, many use cases would be totally fine with doing most changes live, and falling back to the rollback if something went wrong. This initial code drop *only* supports live layering of new packages. However, a lot of the base infrastructure is laid for future work. For now, this will be classified as an experimental feature, hence `ex livefs`. Part of: coreos#639

This is what caused the merge of coreos/rpm-ostree#652 to blow up, since ostreedev#848 landed right before we tried to merge it. When I was writing that PR I remember having an uncertain feeling since we were doing a `mkdirat` above, but at the time I thought we'd have test suite coverage...turns out we didn't. For backwards compatibility, we need to continue to do a `mkdirat` here of the parent. However...I can't think of a reason anyone would *want* that behavior. Hence, let's add a special trick - if the destination name is `.`, we skip `mkdirat()`. That way rpm-ostree for example can open a dfd for `/etc` and avoid the `mkdir`. Fold the subpath tests into `test-basic.sh` since it's not worth a separate file. Add a test case for checking out a file.

cgwalters · 2017-05-12T13:39:22Z

ostreedev/ostree#854

This is what caused the merge of coreos/rpm-ostree#652 to blow up, since #848 landed right before we tried to merge it. When I was writing that PR I remember having an uncertain feeling since we were doing a `mkdirat` above, but at the time I thought we'd have test suite coverage...turns out we didn't. For backwards compatibility, we need to continue to do a `mkdirat` here of the parent. However...I can't think of a reason anyone would *want* that behavior. Hence, let's add a special trick - if the destination name is `.`, we skip `mkdirat()`. That way rpm-ostree for example can open a dfd for `/etc` and avoid the `mkdir`. Fold the subpath tests into `test-basic.sh` since it's not worth a separate file. Add a test case for checking out a file. Closes: #854 Approved by: jlebon

cgwalters · 2017-05-12T14:16:42Z

Ok, fixup ⬆️

jlebon · 2017-05-12T14:43:27Z

@rh-atomic-bot r+ 0060833

rh-atomic-bot · 2017-05-12T14:43:33Z

⌛ Testing commit 0060833 with merge 9379618...

For a future patch. Closes: #652 Approved by: jlebon

There are a few different use cases here. First, for layering new packages, there's no good reason for us to force a reboot. Second, we want some support for cherry-picking security updates and allowing admins to restart services. Finally, at some point we should offer support for entirely replacing the running tree if that's what the user wants. Until now we've been very conservative, but there's a spectrum here. In particular, this patch changes things so we push a rollback before we start doing anything live. I think in practice, many use cases would be totally fine with doing most changes live, and falling back to the rollback if something went wrong. This initial code drop *only* supports live layering of new packages. However, a lot of the base infrastructure is laid for future work. For now, this will be classified as an experimental feature, hence `ex livefs`. Part of: #639 Closes: #652 Approved by: jlebon

rh-atomic-bot · 2017-05-12T14:50:37Z

💔 Test failed - status-atomicjenkins

cgwalters · 2017-05-12T15:00:57Z

@rh-atomic-bot retry

rh-atomic-bot · 2017-05-12T15:01:01Z

⌛ Testing commit 0060833 with merge 95e9aa4...

There are a few different use cases here. First, for layering new packages, there's no good reason for us to force a reboot. Second, we want some support for cherry-picking security updates and allowing admins to restart services. Finally, at some point we should offer support for entirely replacing the running tree if that's what the user wants. Until now we've been very conservative, but there's a spectrum here. In particular, this patch changes things so we push a rollback before we start doing anything live. I think in practice, many use cases would be totally fine with doing most changes live, and falling back to the rollback if something went wrong. This initial code drop *only* supports live layering of new packages. However, a lot of the base infrastructure is laid for future work. For now, this will be classified as an experimental feature, hence `ex livefs`. Part of: #639 Closes: #652 Approved by: jlebon

rh-atomic-bot · 2017-05-12T15:36:25Z

☀️ Test successful - status-atomicjenkins
Approved by: jlebon
Pushing 95e9aa4 to master...

vtolstov · 2017-05-12T16:24:26Z

Great! When the next release ?=) @cgwalters

This is a de-scoping of work I did in preparation for rpm-ostree [live updates](coreos/rpm-ostree#652). Originally I was going to expose this as a public API. However, I decided to do things differently, but the cleanup here for new code style and fd-relative is nice to have anyways. We rework things to use `OstreeDeployment*`, which the caller is expected to already have, rather than `GFile*`s pointing to the config directories.

This is a de-scoping of work I did in preparation for rpm-ostree [live updates](coreos/rpm-ostree#652). Originally I was going to expose this as a public API. However, I decided to do things differently, but the cleanup here for new code style and fd-relative is nice to have anyways. We rework things to use `OstreeDeployment*`, which the caller is expected to already have, rather than `GFile*`s pointing to the config directories. Closes: #741 Approved by: jlebon

cgwalters added the WIP label Mar 2, 2017

cgwalters mentioned this pull request Mar 2, 2017

live updates design #639

Closed

cgwalters force-pushed the live-updates branch from 188d224 to f15edf5 Compare March 2, 2017 20:58

cgwalters mentioned this pull request Mar 10, 2017

implement file triggers #648

Closed

cgwalters force-pushed the live-updates branch from ca048cc to 5dd606f Compare March 11, 2017 20:40

This was referenced Mar 14, 2017

sysroot: Expose 3way /etc merge as API ostreedev/ostree#737

Open

introduce rpm-ostree ex #682

Closed

cgwalters force-pushed the live-updates branch from 5dd606f to d42276b Compare March 16, 2017 22:00

cgwalters force-pushed the live-updates branch 2 times, most recently from 3e5184f to 3018e3d Compare March 20, 2017 19:04

cgwalters force-pushed the live-updates branch from 3018e3d to 8b38111 Compare March 22, 2017 14:33

cgwalters mentioned this pull request Mar 22, 2017

libglnx: bump for new error API #699

Closed

cgwalters force-pushed the live-updates branch 3 times, most recently from 30a139b to 7c178d0 Compare March 24, 2017 21:59

This was referenced Mar 25, 2017

Live updates build 🏗 prep #705

Closed

Live updates infrastructure ⏚ prep #706

Closed

cgwalters changed the title ~~WIP: livefs~~ WIP: livefs 🗱 Mar 25, 2017

cgwalters force-pushed the live-updates branch from 7c178d0 to b0f5d7a Compare March 27, 2017 17:29

cgwalters force-pushed the live-updates branch from b0f5d7a to 22881a6 Compare March 27, 2017 20:49

cgwalters mentioned this pull request Mar 27, 2017

Live updates prep ↨ #709

Closed

jlebon reviewed May 11, 2017

View reviewed changes

cgwalters removed reviewed/needs-rework WIP labels May 11, 2017

cgwalters added 2 commits May 11, 2017 17:43

libtest: Support matching multiple regexps in files

0d42318

For a future patch.

cgwalters mentioned this pull request May 12, 2017

lib/checkout: Fix regression in subpath for regular files ostreedev/ostree#854

Closed

fixup! Introduce ex livefs

0060833

cgwalters force-pushed the live-updates branch from 4f3a70d to 0060833 Compare May 12, 2017 14:12

rh-atomic-bot pushed a commit that referenced this pull request May 12, 2017

libtest: Support matching multiple regexps in files

17626d2

For a future patch. Closes: #652 Approved by: jlebon

rh-atomic-bot closed this in 376a2cc May 12, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce `ex livefs` for layering newly added packages 🗱 #652

Introduce `ex livefs` for layering newly added packages 🗱 #652

cgwalters commented Mar 2, 2017 •

edited

Loading

cgwalters commented Mar 2, 2017

giuseppe commented Mar 2, 2017

giuseppe commented Mar 3, 2017

rh-atomic-bot commented Mar 16, 2017

cgwalters commented Mar 17, 2017

cgwalters commented Mar 20, 2017

rh-atomic-bot commented Mar 20, 2017

rh-atomic-bot commented Mar 22, 2017

rh-atomic-bot commented Mar 27, 2017

rh-atomic-bot commented May 11, 2017

rh-atomic-bot commented May 11, 2017

jlebon commented May 11, 2017

jlebon May 11, 2017

cgwalters commented May 12, 2017

cgwalters commented May 12, 2017

jlebon commented May 12, 2017

rh-atomic-bot commented May 12, 2017

rh-atomic-bot commented May 12, 2017

cgwalters commented May 12, 2017

rh-atomic-bot commented May 12, 2017

rh-atomic-bot commented May 12, 2017

vtolstov commented May 12, 2017

Introduce ex livefs for layering newly added packages 🗱 #652

Introduce ex livefs for layering newly added packages 🗱 #652

Conversation

cgwalters commented Mar 2, 2017 • edited Loading

cgwalters commented Mar 2, 2017

giuseppe commented Mar 2, 2017

giuseppe commented Mar 3, 2017

rh-atomic-bot commented Mar 16, 2017

cgwalters commented Mar 17, 2017

cgwalters commented Mar 20, 2017

rh-atomic-bot commented Mar 20, 2017

rh-atomic-bot commented Mar 22, 2017

rh-atomic-bot commented Mar 27, 2017

rh-atomic-bot commented May 11, 2017

rh-atomic-bot commented May 11, 2017

jlebon commented May 11, 2017

jlebon May 11, 2017

Choose a reason for hiding this comment

cgwalters commented May 12, 2017

cgwalters commented May 12, 2017

jlebon commented May 12, 2017

rh-atomic-bot commented May 12, 2017

rh-atomic-bot commented May 12, 2017

cgwalters commented May 12, 2017

rh-atomic-bot commented May 12, 2017

rh-atomic-bot commented May 12, 2017

vtolstov commented May 12, 2017

Introduce `ex livefs` for layering newly added packages 🗱 #652

Introduce `ex livefs` for layering newly added packages 🗱 #652

cgwalters commented Mar 2, 2017 •

edited

Loading