cmdlib: don't use cache qcow for composes; use virtiofs #3720
Conversation
dustymabe
force-pushed
the
dusty-with-cache
branch
from
bc68f03
to
31662a0
Compare
If we ever want to compose over virtiofsd we'll need xattr support.
Now that our OSBuild workflow is using the cache we saw at least one case where the pipeline was running out of space when composing the extensions. Since we had a previous proposal [1] to just drop the cache altogether anyway let's try to at least remove it from the runcompose functions to eliminate the use of it there anyway. [1] coreos#3615
dustymabe
force-pushed
the
dusty-with-cache
branch
from
31662a0
to
100198b
Compare
|
ok this worked (or rather didn't work because it was an invalid test) locally for me initially because I wasn't running in a VM (i.e. I was executing the privileged workflow). Setting I added a commit to add |
|
It failed the same way in CI here. |
|
@dustymabe: The following test failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
Since we are now using the cache qcow2 for OSBuild we need it to be a little large to handle those duties as well as the tree compose ones. I'd like to drop the cache, but hit some trouble there; see coreos#3720.
Since we are now using the cache qcow2 for OSBuild we need it to be a little large to handle those duties as well as the tree compose ones. I'd like to drop the cache, but hit some trouble there; see #3720.
|
PR needs rebase. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
I think the issue there is that the actual compose of the rootfs happens over virtiofs also because rpm-ostree wants to colocate it with the cache repo to get hardlinks. One thing we could do is put the virtiofs mount r/w in |
I'm not sure I understand this comment. Here I am modifying it to use |
Right. The compose just happens wherever the pkgcache repo is. Before (status quo), that was on the cache qcow2. Now, that's over virtiofs. |
|
So rpm-ostree does have support for e.g. applying filecaps at commit time, but it currently keys off of uid != 0 to know this, except that in the supermin VM we are root. And we do need to be root to e.g. do privileged stuff like mount namespaces. But it might work to just add a flag to force the commit modifier path even if uid == 0. E.g. we could try testing with diff --git a/src/libpriv/rpmostree-core.cxx b/src/libpriv/rpmostree-core.cxx
index 9cc872b2..efb77107 100644
--- a/src/libpriv/rpmostree-core.cxx
+++ b/src/libpriv/rpmostree-core.cxx
@@ -3561,7 +3561,7 @@ apply_rpmfi_overrides (RpmOstreeContext *self, int tmprootfs_dfd, DnfPackage *pk
*
* TODO: For non-root `--unified-core` we need to do it as a commit modifier.
*/
- if (getuid () != 0)
+ if (g_getenv ("RPMOSTREE_SKIP_RPMFI_OVERRIDES") || getuid () != 0)
return TRUE; /* 🔚 Early return */
g_auto (rpmfi) fi = NULL;But there may be other things that break. |
|
I think this would be good to pick up again if it's not a lot of work to get working. But long-term, I think it'll get obsoleted by the move to deriving from a shared base image instead. |
|
I looked at this a bit earlier this week and I'll update here with some notes. In order to make it so that you can create files that appear to the VM as UID/GID diff --git a/mantle/platform/qemu.go b/mantle/platform/qemu.go
index e0070acc3..ead654005 100644
--- a/mantle/platform/qemu.go
+++ b/mantle/platform/qemu.go
@@ -1657,7 +1657,10 @@ func (builder *QemuBuilder) VirtioJournal(config *conf.Conf, queryArguments stri
// createVirtiofsCmd returns a new command instance configured to launch virtiofsd.
func createVirtiofsCmd(directory, socketPath string) exec.Cmd {
- args := []string{"--sandbox", "none", "--socket-path", socketPath, "--shared-dir", "."}
+ args := []string{"--mount", "--keep-caps", "--map-users=auto",
+ "--map-groups=auto", "--map-root-user",
+ "/usr/libexec/virtiofsd", "--sandbox", "none",
+ "--socket-path", socketPath, "--shared-dir", "."}
// Work around https://gitlab.com/virtio-fs/virtiofsd/-/merge_requests/197
if os.Getuid() == 0 {
args = append(args, "--modcaps=-mknod:-setfcap")
@@ -1665,7 +1668,7 @@ func createVirtiofsCmd(directory, socketPath string) exec.Cmd {
// We don't need seccomp filtering; we trust our workloads. This incidentally
// works around issues like https://gitlab.com/virtio-fs/virtiofsd/-/merge_requests/200.
args = append(args, "--seccomp=none")
- cmd := exec.Command("/usr/libexec/virtiofsd", args...)
+ cmd := exec.Command("/usr/bin/unshare", args...)
// This sets things up so that the `.` we passed in the arguments is the target directory
cmd.Dir = directory
// Quiet the daemon by defaultOf course To handle xattrs/SELinux we can enable them and map them to a diff --git a/mantle/platform/qemu.go b/mantle/platform/qemu.go
index e0070acc3..43bcba6ab 100644
--- a/mantle/platform/qemu.go
+++ b/mantle/platform/qemu.go
@@ -1665,6 +1665,12 @@ func createVirtiofsCmd(directory, socketPath string) exec.Cmd {
// We don't need seccomp filtering; we trust our workloads. This incidentally
// works around issues like https://gitlab.com/virtio-fs/virtiofsd/-/merge_requests/200.
args = append(args, "--seccomp=none")
+ // OSTree composes want xattrs for SELinux. We need to map all
+ // xattrs inside the guest to `user.virtiofs.*` xattrs on the host.
+ // https://gitlab.com/virtio-fs/virtiofsd/-/blob/main/doc/xattr-mapping.md
+ args = append(args, "--xattr")
+ args = append(args, "--xattrmap=:map::user.virtiofs.:")
+ args = append(args, "--security-label")
cmd := exec.Command("/usr/libexec/virtiofsd", args...)
// This sets things up so that the `.` we passed in the arguments is the target directory
cmd.Dir = directoryWhere it doesn't work is (apparently) There was an effort in the past years to lift this limitation on the So I think we're at a bit of a dead-end unfortunately unless we gain |
Now that our OSBuild workflow is using the cache we saw at least one case where the pipeline was running out of space. Since we had a previous proposal [1] to just drop the cahce altogether anyway let's try to at least remove it from the runcompose functions to eliminate the use of it there anyway.
[1] #3615