[merged] Package layering #289

jlebon · 2016-05-20T19:05:20Z

This is a continuation of #107.

I'd first like to start with apologizing in advance for how massive this PR ended up being. I did my best to condense it into only 6 commits:

5c470d7: this is @cgwalters's original work from Package layering #107
827789d: these are fixups on the previous commit, mostly to reconcile it with the updated way we do things in current HEAD
73aefe9: major unpacker rework -- drop the unpack_to_dfd path and use only ostree's newly reworked support for libarchive ([merged] Rework libarchive import support ostreedev/ostree#275)
56b03c3: add printf semantics to the output API
0ccc159: major core rework -- this prepares rpmostree-core.c (mostly RpmOstreeContext) for what will be required for package layering. The commit is larger than it should because I also reorganized the whole file to help me navigate better.
926363e: this is the actual work that moves package layering into the upgrader

The commits are not properly separate from each other, so rpm-ostree will most likely NOT compile in between them. If you'd like to review smaller, easier to follow patches, you can look over the package-layering-wip branch instead, which is the exact same final diff, but more representative of the path I followed. As a result there's a lot of added noise.

This builds upon the earlier prototype in https://github.com/cgwalters/atomic-pkglayer The `.origin` file says for a replicated installation: [origin] refspec=local:rhel-atomic-host/7/x86_64/standard If you then run `rpm-ostree pkg-add strace`, it will result in a new tree with: [origin] baserefspec=local:rhel-atomic-host/7/x86_64/standard [packages] requested=strace; Work still remaining here is to teach `rpm-ostree status` and `rpm-ostree upgrade` about this.

This patch fixes up errors in the parent commit which added pkg-add. Some of them are due to the rebase on top of the unprivileged infrastructure. - fix compile errors (due to libhif changes after rebase) - delete duplicate prototype for rpmostree_sysroot_upgrader_deploy - include allow-older in flags type - fix change_upgrader_refspec to use g_strdup() (this was causing the wrong old refspec to be registered) - in builtin-status.c, check for NULL before joining the packages array - sysroot-upgrader: fix gtype function names - roc_context_prepare_for_root(): delete unused param - assemble_commit(): delete unused param - RpmOstreeSysrootUpgraderFlags: fix docs - Fix sysroot property name and add reboot opt

- Delete unpack_to_dfd path - Get rid of copynpaste stuff and use the newly reworked ostree libarchive API which now supports the callbacks we need

This patch prepares RpmOstreeContext for supporting package layering. A relabel operation is added as well to support relabeling imported packages if the sepolicy of the rootfs we're overlaying onto is different from during import.

- Move the package layering logic away from pkg-add and into the upgrader - Add pkg-delete - Add dry-run option

jlebon · 2016-05-24T15:16:58Z

To make it easier for others to play & test it, I composed an F23 tree. To use:

# ostree remote add jlebon http://jlebon.fedorapeople.org/trees/pkglayer --no-gpg-verify
# rpm-ostree rebase --reboot jlebon:fedora-atomic/f23/x86_64/docker-host

Adding e.g. git:

# rpm-ostree pkg-add --reboot git
...
# git version
git version 2.5.5

jlebon · 2016-05-24T15:17:18Z

Will work on fixing those tests, and add some new tests.

jlebon · 2016-05-24T21:46:24Z

⬆️ Updated to pass tests.

When setting a new refspec to rebase to, we need to also update the origin file. Otherwise the wrong refspec will be written to disk.

When checking if a new upgrade is available, we should be comparing the SHA of the base layer if packages are overlayed. Otherwise we'll always think there are updates.

cgwalters · 2016-05-26T15:02:12Z

@rh-atomic-bot try

rh-atomic-bot · 2016-05-26T15:02:18Z

⌛ Trying commit 6341adc with merge 4c156d9...

This builds upon the earlier prototype in https://github.com/cgwalters/atomic-pkglayer The `.origin` file says for a replicated installation: [origin] refspec=local:rhel-atomic-host/7/x86_64/standard If you then run `rpm-ostree pkg-add strace`, it will result in a new tree with: [origin] baserefspec=local:rhel-atomic-host/7/x86_64/standard [packages] requested=strace; Work still remaining here is to teach `rpm-ostree status` and `rpm-ostree upgrade` about this. Closes: #289 Approved by: <try>

This patch fixes up errors in the parent commit which added pkg-add. Some of them are due to the rebase on top of the unprivileged infrastructure. - fix compile errors (due to libhif changes after rebase) - delete duplicate prototype for rpmostree_sysroot_upgrader_deploy - include allow-older in flags type - fix change_upgrader_refspec to use g_strdup() (this was causing the wrong old refspec to be registered) - in builtin-status.c, check for NULL before joining the packages array - sysroot-upgrader: fix gtype function names - roc_context_prepare_for_root(): delete unused param - assemble_commit(): delete unused param - RpmOstreeSysrootUpgraderFlags: fix docs - Fix sysroot property name and add reboot opt Closes: #289 Approved by: <try>

- Delete unpack_to_dfd path - Get rid of copynpaste stuff and use the newly reworked ostree libarchive API which now supports the callbacks we need Closes: #289 Approved by: <try>

Closes: #289 Approved by: <try>

This patch prepares RpmOstreeContext for supporting package layering. A relabel operation is added as well to support relabeling imported packages if the sepolicy of the rootfs we're overlaying onto is different from during import. Closes: #289 Approved by: <try>

- Move the package layering logic away from pkg-add and into the upgrader - Add pkg-delete - Add dry-run option Closes: #289 Approved by: <try>

Closes: #289 Approved by: <try>

jlebon · 2016-05-26T16:54:51Z

@rh-atomic-bot retry

rh-atomic-bot · 2016-05-26T16:54:57Z

⌛ Trying commit 6341adc with merge e619a5f...

This builds upon the earlier prototype in https://github.com/cgwalters/atomic-pkglayer The `.origin` file says for a replicated installation: [origin] refspec=local:rhel-atomic-host/7/x86_64/standard If you then run `rpm-ostree pkg-add strace`, it will result in a new tree with: [origin] baserefspec=local:rhel-atomic-host/7/x86_64/standard [packages] requested=strace; Work still remaining here is to teach `rpm-ostree status` and `rpm-ostree upgrade` about this. Closes: #289 Approved by: <try>

This patch fixes up errors in the parent commit which added pkg-add. Some of them are due to the rebase on top of the unprivileged infrastructure. - fix compile errors (due to libhif changes after rebase) - delete duplicate prototype for rpmostree_sysroot_upgrader_deploy - include allow-older in flags type - fix change_upgrader_refspec to use g_strdup() (this was causing the wrong old refspec to be registered) - in builtin-status.c, check for NULL before joining the packages array - sysroot-upgrader: fix gtype function names - roc_context_prepare_for_root(): delete unused param - assemble_commit(): delete unused param - RpmOstreeSysrootUpgraderFlags: fix docs - Fix sysroot property name and add reboot opt Closes: #289 Approved by: <try>

This builds upon the earlier prototype in https://github.com/cgwalters/atomic-pkglayer The `.origin` file says for a replicated installation: [origin] refspec=local:rhel-atomic-host/7/x86_64/standard If you then run `rpm-ostree pkg-add strace`, it will result in a new tree with: [origin] baserefspec=local:rhel-atomic-host/7/x86_64/standard [packages] requested=strace; Work still remaining here is to teach `rpm-ostree status` and `rpm-ostree upgrade` about this. Closes: #289 Approved by: <try>

This patch fixes up errors in the parent commit which added pkg-add. Some of them are due to the rebase on top of the unprivileged infrastructure. - fix compile errors (due to libhif changes after rebase) - delete duplicate prototype for rpmostree_sysroot_upgrader_deploy - include allow-older in flags type - fix change_upgrader_refspec to use g_strdup() (this was causing the wrong old refspec to be registered) - in builtin-status.c, check for NULL before joining the packages array - sysroot-upgrader: fix gtype function names - roc_context_prepare_for_root(): delete unused param - assemble_commit(): delete unused param - RpmOstreeSysrootUpgraderFlags: fix docs - Fix sysroot property name and add reboot opt Closes: #289 Approved by: <try>

- Delete unpack_to_dfd path - Get rid of copynpaste stuff and use the newly reworked ostree libarchive API which now supports the callbacks we need Closes: #289 Approved by: <try>

Closes: #289 Approved by: <try>

This patch prepares RpmOstreeContext for supporting package layering. A relabel operation is added as well to support relabeling imported packages if the sepolicy of the rootfs we're overlaying onto is different from during import. Closes: #289 Approved by: <try>

- Move the package layering logic away from pkg-add and into the upgrader - Add pkg-delete - Add dry-run option Closes: #289 Approved by: <try>

Closes: #289 Approved by: <try>

rh-atomic-bot · 2016-05-26T18:56:49Z

☀️ Test successful - status-atomicjenkins
State: approved= try=True

cgwalters · 2016-05-26T19:14:46Z

@rh-atomic-bot do or do not, there is no try-

cgwalters · 2016-05-26T19:15:37Z

Oh yes, that worked. 😎

jlebon · 2016-05-26T19:51:51Z

Hah!

dustymabe · 2016-05-26T19:58:46Z

You guys are having too much fun :)

cgwalters · 2016-05-27T22:18:40Z

Okay, so I finally had some time to play with this and read the code a bit. I pushed one fixup to disable fsync...which makes the whole thing rather dramatically faster 🚤

Kind of avoids the need for:

  /* let's give the user some feedback so they don't think we're blocked */
  rpmostree_output_task_begin ("Checking out tree %.7s", revision);

But we can consider removing that later or something.

I think my vote here is to merge and we can do fixups in master.

@rh-atomic-bot r+ 91bd6c2

rh-atomic-bot · 2016-05-27T22:18:46Z

⌛ Testing commit 91bd6c2 with merge e063754...

This patch fixes up errors in the parent commit which added pkg-add. Some of them are due to the rebase on top of the unprivileged infrastructure. - fix compile errors (due to libhif changes after rebase) - delete duplicate prototype for rpmostree_sysroot_upgrader_deploy - include allow-older in flags type - fix change_upgrader_refspec to use g_strdup() (this was causing the wrong old refspec to be registered) - in builtin-status.c, check for NULL before joining the packages array - sysroot-upgrader: fix gtype function names - roc_context_prepare_for_root(): delete unused param - assemble_commit(): delete unused param - RpmOstreeSysrootUpgraderFlags: fix docs - Fix sysroot property name and add reboot opt Closes: #289 Approved by: cgwalters

- Delete unpack_to_dfd path - Get rid of copynpaste stuff and use the newly reworked ostree libarchive API which now supports the callbacks we need Closes: #289 Approved by: cgwalters

Closes: #289 Approved by: cgwalters

This patch prepares RpmOstreeContext for supporting package layering. A relabel operation is added as well to support relabeling imported packages if the sepolicy of the rootfs we're overlaying onto is different from during import. Closes: #289 Approved by: cgwalters

- Move the package layering logic away from pkg-add and into the upgrader - Add pkg-delete - Add dry-run option Closes: #289 Approved by: cgwalters

Closes: #289 Approved by: cgwalters

rh-atomic-bot · 2016-05-27T22:22:30Z

☀️ Test successful - status-atomicjenkins
Approved by: cgwalters
Pushing e063754 to master...

jlebon · 2016-05-30T15:05:49Z

I pushed one fixup to disable fsync...which makes the whole thing rather dramatically faster 🚤

Wow, what a nice speedup. Awesome!

cgwalters and others added 6 commits May 20, 2016 13:58

unpacker: major rework

73aefe9

- Delete unpack_to_dfd path - Get rid of copynpaste stuff and use the newly reworked ostree libarchive API which now supports the callbacks we need

output: support printf type usage

56b03c3

core: major rework

0ccc159

This patch prepares RpmOstreeContext for supporting package layering. A relabel operation is added as well to support relabeling imported packages if the sepolicy of the rootfs we're overlaying onto is different from during import.

package layering: major rework

926363e

- Move the package layering logic away from pkg-add and into the upgrader - Add pkg-delete - Add dry-run option

jlebon mentioned this pull request May 20, 2016

Package layering #107

Closed

jlebon added 2 commits May 24, 2016 12:17

Makefile-tests.am: add env var to know when in testenv

2a9ae4f

fixup! core: major rework

07828f1

jlebon added 2 commits May 24, 2016 17:50

fixup! package layering: major rework

8e5f054

When setting a new refspec to rebase to, we need to also update the origin file. Otherwise the wrong refspec will be written to disk.

test-basic.sh: make subtests more granular

280ec7c

jlebon force-pushed the package-layering branch from d0faf8a to 280ec7c Compare May 24, 2016 21:50

fixup! package layering: major rework

6341adc

When checking if a new upgrade is available, we should be comparing the SHA of the base layer if packages are overlayed. Otherwise we'll always think there are updates.

rh-atomic-bot pushed a commit that referenced this pull request May 26, 2016

unpacker: major rework

c2a3e30

- Delete unpack_to_dfd path - Get rid of copynpaste stuff and use the newly reworked ostree libarchive API which now supports the callbacks we need Closes: #289 Approved by: <try>

rh-atomic-bot pushed a commit that referenced this pull request May 26, 2016

output: support printf type usage

accb969

Closes: #289 Approved by: <try>

rh-atomic-bot pushed a commit that referenced this pull request May 26, 2016

package layering: major rework

b55deb1

- Move the package layering logic away from pkg-add and into the upgrader - Add pkg-delete - Add dry-run option Closes: #289 Approved by: <try>

rh-atomic-bot pushed a commit that referenced this pull request May 26, 2016

Makefile-tests.am: add env var to know when in testenv

91c7f19

Closes: #289 Approved by: <try>

rh-atomic-bot pushed a commit that referenced this pull request May 26, 2016

test-basic.sh: make subtests more granular

4c156d9

Closes: #289 Approved by: <try>

rh-atomic-bot pushed a commit that referenced this pull request May 26, 2016

unpacker: major rework

a0f147e

- Delete unpack_to_dfd path - Get rid of copynpaste stuff and use the newly reworked ostree libarchive API which now supports the callbacks we need Closes: #289 Approved by: <try>

rh-atomic-bot pushed a commit that referenced this pull request May 26, 2016

output: support printf type usage

ea7fbf4

Closes: #289 Approved by: <try>

rh-atomic-bot pushed a commit that referenced this pull request May 26, 2016

package layering: major rework

7c1cd1a

- Move the package layering logic away from pkg-add and into the upgrader - Add pkg-delete - Add dry-run option Closes: #289 Approved by: <try>

rh-atomic-bot pushed a commit that referenced this pull request May 26, 2016

Makefile-tests.am: add env var to know when in testenv

3357512

Closes: #289 Approved by: <try>

rh-atomic-bot pushed a commit that referenced this pull request May 26, 2016

test-basic.sh: make subtests more granular

7fb9906

Closes: #289 Approved by: <try>

fixup! package layering: major rework

91bd6c2

rh-atomic-bot pushed a commit that referenced this pull request May 27, 2016

unpacker: major rework

ca162de

- Delete unpack_to_dfd path - Get rid of copynpaste stuff and use the newly reworked ostree libarchive API which now supports the callbacks we need Closes: #289 Approved by: cgwalters

rh-atomic-bot pushed a commit that referenced this pull request May 27, 2016

output: support printf type usage

1ccc560

Closes: #289 Approved by: cgwalters

rh-atomic-bot pushed a commit that referenced this pull request May 27, 2016

package layering: major rework

ead1ecd

- Move the package layering logic away from pkg-add and into the upgrader - Add pkg-delete - Add dry-run option Closes: #289 Approved by: cgwalters

rh-atomic-bot pushed a commit that referenced this pull request May 27, 2016

Makefile-tests.am: add env var to know when in testenv

336f765

Closes: #289 Approved by: cgwalters

rh-atomic-bot pushed a commit that referenced this pull request May 27, 2016

test-basic.sh: make subtests more granular

e063754

Closes: #289 Approved by: cgwalters

rh-atomic-bot changed the title ~~Package layering~~ [merged] Package layering May 27, 2016

rh-atomic-bot closed this in beb026f May 27, 2016

jlebon mentioned this pull request Jun 2, 2016

Improving the rpm-ostree testsuite #300

Closed

jlebon deleted the package-layering branch October 3, 2017 19:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[merged] Package layering #289

[merged] Package layering #289

jlebon commented May 20, 2016

jlebon commented May 24, 2016

jlebon commented May 24, 2016

jlebon commented May 24, 2016

cgwalters commented May 26, 2016

rh-atomic-bot commented May 26, 2016

jlebon commented May 26, 2016

rh-atomic-bot commented May 26, 2016

rh-atomic-bot commented May 26, 2016

cgwalters commented May 26, 2016

cgwalters commented May 26, 2016

jlebon commented May 26, 2016

dustymabe commented May 26, 2016

cgwalters commented May 27, 2016

rh-atomic-bot commented May 27, 2016

rh-atomic-bot commented May 27, 2016

jlebon commented May 30, 2016

[merged] Package layering #289

[merged] Package layering #289

Conversation

jlebon commented May 20, 2016

jlebon commented May 24, 2016

jlebon commented May 24, 2016

jlebon commented May 24, 2016

cgwalters commented May 26, 2016

rh-atomic-bot commented May 26, 2016

jlebon commented May 26, 2016

rh-atomic-bot commented May 26, 2016

rh-atomic-bot commented May 26, 2016

cgwalters commented May 26, 2016

cgwalters commented May 26, 2016

jlebon commented May 26, 2016

dustymabe commented May 26, 2016

cgwalters commented May 27, 2016

rh-atomic-bot commented May 27, 2016

rh-atomic-bot commented May 27, 2016

jlebon commented May 30, 2016