Unprivileged operation infrastructure #209

cgwalters · 2016-01-26T19:45:41Z

The first chunk here is fleshing out code for unpacking RPMs as non root and committing them into an OSTree repository. The second commit is using that to make a tech demo container builtin.

cgwalters · 2016-01-26T19:57:08Z

bot, retest this please

This is in preparation for `rpm-ostree container`, which handles unpacking RPMs as non-root. At the moment, I'm copying code in from both ostree's libarchive bits (fixable...may need to export some utility functions) and some functions from libhif (harder, see: http://lists.rpm.org/pipermail/rpm-ecosystem/2016-January/000297.html ) There's lots more cleanup to do here, but I don't want to block on the resolution of the libhif changes.

This is just a tech demo. Example usage: ``` mkdir -p ~/.cache/rpmostree-containers cd ~/.cache/rpmostree-containers rpm-ostree container init cp /etc/yum.repos.d/CentOS-Core.repo rpmmd.repos.d rpm-ostree container assemble bash rpm-ostree container assemble httpd ```

This is a work-in-progress port against the libhif master + some outstanding PRs.

This is really going to be the heart of rpm-ostree, so let's give it a better name.

This helps unify some code more initially between the treecompose and container bits.

It's the default for unprivileged composes if it exists. This is an incremental step towards always using the ostree repo.

It's what more components are standardizing on.

I debated config file formats a lot. JSON is fairly awkward for humans to write, and really painful to parse from C. YAML is nice, but also painful from C. Both are fairly overpowered for what we really need. Keyfiles (desktop spec, `GKeyFile`) have a lot of limitations, but at least it's used by systemd and `.desktop` files, and we already have a parser. We still parse the JSON treefiles, but internally convert them to `GKeyFile` (which is in turn converted to `GVariant` for a canonical form).

We need to do selinux first.

Right now we're doing the /etc -> /usr/etc inside the RPM import, but we might as well do the /usr/local bits in both. Also, use /usr/share/rpm by default for treecompose too so that is unified. Other things like systemd unit files and kernel handling are only going to be used for host side composes.

Sadly, libhif keeps trying to auto-create it. It'll need patching there, and possibly in librpm.

cgwalters · 2016-02-11T16:06:31Z

So the next major architectural steps here are:

Handle SELinux labeling while unprivileged (requires unpacking selinux-policy and caching correctly)
Handle %posts (requires requiring linux-user-chroot or userns)

cgwalters · 2016-02-11T16:07:58Z

Also: The current code now requires git master of https://github.com/rpm-software-management/libhif

Despite this, I'm tempted to merge this to master fairly soon. Any thoughts/objections?

rhatdan · 2016-02-11T17:01:38Z

Not sure you should do anything with SELinux, if this is a container use case. We tend to run our containers with a single label, and since we want apps running as a User. We also don't want to allow a user process to lay down certain labels, for example a normal user should not be creating httpd_exec_t.

cgwalters · 2016-02-11T17:16:02Z

Right, but we should support SELinux labeling for the host case.

jlebon · 2016-02-17T20:47:24Z

The following patch makes all the tests pass here:

diff --git a/Makefile-tests.am b/Makefile-tests.am
index 54a76ea..781c1d4 100644
--- a/Makefile-tests.am
+++ b/Makefile-tests.am
@@ -1,7 +1,5 @@
 include $(top_srcdir)/buildutil/glib-tap.mk

-AM_TESTS_ENVIRONMENT = INSTALLEDTESTS=""
-
 CLEANFILES += \
        tests/setup-session.sh \
        tests/compose/yum/empty tests/compose/yum/repodata/repomd.xml \
@@ -51,16 +49,15 @@ nobase_installed_test_DATA += $(shell find tests/compose/yum/ -type f) \

 installed_test_scripts = \
        tests/test-compose.sh                   \
-       tests/test-compose-no-selinux-tag.sh            \
+       tests/test-compose-no-selinux-tag.sh    \
        tests/test-basic.sh                     \
+       tests/test-ucontainer.sh                \
        $(NULL)

 test_programs = \
        tests/jsonutil                  \
        $(NULL)

-test_scripts = tests/test-ucontainer.sh
-
 installed_test_PROGRAMS += dbus-run-session
 dbus_run_session_SOURCES = tests/dbus-run-session.c

diff --git a/tests/libtest.sh b/tests/libtest.sh
index 7654368..ab5f888 100644
--- a/tests/libtest.sh
+++ b/tests/libtest.sh
@@ -18,17 +18,6 @@
 # Boston, MA 02111-1307, USA.

 SRCDIR=$(dirname $0)
-_cleanup_tmpdir () {
-    if test -f ${test_tmpdir}.test; then
-       rm ${test_tmpdir} -rf
-    fi
-}
-if test -z "${INSTALLEDTESTS:-}"; then
-   test_tmpdir=$(mktemp -d test.XXXXXX)
-   touch ${test_tmpdir}/.test
-   trap _cleanup_tmpdir EXIT
-   cd ${test_tmpdir}
-fi   
 test_tmpdir=$(pwd)

 export G_DEBUG=fatal-warnings

cgwalters · 2016-02-17T21:11:05Z

I just pushed a fixup ⬆️ which makes make check work again uninstalled (and regardless, correctly against the in-tree binary).

cgwalters · 2016-02-18T02:11:28Z

You're right, I had the logic backwards. Pushed fixups.

jlebon · 2016-02-18T21:15:19Z

Finally was able to play around with this branch. Works pretty well, though there are some rough edges from the UX perspective, which is understandable.

I checked over the code, though I didn't comb over it line by line due to the sheer volume. The copy/pasting is unfortunate, though I suppose it's OK since it is all temporary.

Thanks for adding a test. Had a few issues getting them all passing, but I think that's all ironed out now. Would be nice to get some positive cases as well (maybe using the empty pkg + adding another one to test composition).

I noticed you have some commented out code for a "start" subcommand. I suppose it's for chroot'ing into the roots? Is the plan to eventually enable this, or will it be up to external container tools to use the roots as needed?

I'm OK with merging it, though I would prefer to wait until we can update libhif in the tester.

cgwalters · 2016-02-19T13:59:25Z

The big picture here is a little uncertain, but I'm trying to simultaeously reach three related goals:

Package layering
Unprivileged operation for rpm-ostree compose tree for host systems
Unprivileged multi-container rpm base

The second two options are going to depend on an unprivileged container runtime too. There's some more movement on productizing some bits out of https://git.gnome.org/browse/linux-user-chroot and https://cgit.freedesktop.org/xdg-app/xdg-app/tree/common/xdg-app-helper.c

I think rpm-ostree shouldn't itself be a container tool, we'd put the high level logic in the atomic command. So it's more of a shared library. But actually trying to make all of this stuff a stable shared library is a scary thought...we'd really have to be unstable and do version interlock for a while at least?

Realistically, I'd like to return to using some of these bits for package layering first, and leave the other parts as easter eggs to develop later.

cgwalters · 2016-02-23T22:58:20Z

This was merged to master.

This was referenced Jan 26, 2016

Internals mkroot #207

Closed

[WIP] Add a package object cache #146

Closed

This was referenced Jan 28, 2016

repo: Add an API for intercepting file commits and returning cached v… ostreedev/ostree#114

Closed

Package layering #107

Closed

cgwalters force-pushed the unprivileged branch from 850ec46 to d44a30b Compare February 4, 2016 07:52

cgwalters added 12 commits February 10, 2016 18:00

build: Port to master libhif

a7a69c7

This is a work-in-progress port against the libhif master + some outstanding PRs.

Rename rpmostree-hif.[ch] -> core

0c5da93

This is really going to be the heart of rpm-ostree, so let's give it a better name.

Introduce RpmOstreeContext as an object

e8604a6

This helps unify some code more initially between the treecompose and container bits.

libpriv: Make the OSTree repo implicit

28b90c9

It's the default for unprivileged composes if it exists. This is an incremental step towards always using the ostree repo.

tests: Use createrepo_c

eb026e3

It's what more components are standardizing on.

tests: Add test coverage for rpm-ostree container

df9fb2f

treecompose: Make use of cachedir again (just for metadata)

dc30e27

We need to do selinux first.

Use a dummy install root for unprivileged composes

ceb8ee9

Sadly, libhif keeps trying to auto-create it. It'll need patching there, and possibly in librpm.

cgwalters force-pushed the unprivileged branch from 0c38d73 to ceb8ee9 Compare February 10, 2016 20:39

fixup! treecompose: Make use of cachedir again (just for metadata)

d0fadbf

fixup! tests: Add test coverage for rpm-ostree container

0918df3

cgwalters added 2 commits February 17, 2016 18:03

fixup! tests: Add test coverage for rpm-ostree container

b2c6765

fixup! tests: Add test coverage for rpm-ostree container

b5980b9

jlebon mentioned this pull request Feb 23, 2016

unpacker: Fix a 32 bit format string error #215

Merged

cgwalters closed this Feb 23, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unprivileged operation infrastructure #209

Unprivileged operation infrastructure #209

cgwalters commented Jan 26, 2016

cgwalters commented Jan 26, 2016

cgwalters commented Feb 11, 2016

cgwalters commented Feb 11, 2016

rhatdan commented Feb 11, 2016

cgwalters commented Feb 11, 2016

jlebon commented Feb 17, 2016

cgwalters commented Feb 17, 2016

cgwalters commented Feb 18, 2016

jlebon commented Feb 18, 2016

cgwalters commented Feb 19, 2016

cgwalters commented Feb 23, 2016

Unprivileged operation infrastructure #209

Unprivileged operation infrastructure #209

Conversation

cgwalters commented Jan 26, 2016

cgwalters commented Jan 26, 2016

cgwalters commented Feb 11, 2016

cgwalters commented Feb 11, 2016

rhatdan commented Feb 11, 2016

cgwalters commented Feb 11, 2016

jlebon commented Feb 17, 2016

cgwalters commented Feb 17, 2016

cgwalters commented Feb 18, 2016

jlebon commented Feb 18, 2016

cgwalters commented Feb 19, 2016

cgwalters commented Feb 23, 2016