coreos/toolbox v2

[dustymabe]

We'd like to carry https://github.com/coreos/toolbox forward for FCOS and maybe give it a revamp as part of the process. Let's discuss design decisions for a v2 of coreos/toolbox in this ticket.

[dustymabe]

Discussed in the meeting today and bgilbert had a few notes of things we'd like to change:

13:04:35       bgilbert | off the top of my head, dropping the rkt dependency, adding a mechanism for command-line arguments, probably changing the mount point for /
13:05:56       bgilbert | not exposing systemd-nspawn syntax in the config API

Maybe we can make a write up of what toolbox does today and a proposal for what we'd like to change/improve.

[dm0-]

There is still the open bug coreos/bugs#1610. Maybe the redesign can include mount propagation as a requirement from the start.

[dustymabe]

on a related note can we enable issues in https://github.com/coreos/toolbox like we did for ignition ?

[dm0-]

Okay, I've enabled them.

[glevand]

The CL toolbox has the user config parameter TOOLBOX_DOCKER_IMAGE, so I could set it to my favorite distro, and not be forced to use the default...

[dustymabe]

The CL toolbox has the user config parameter TOOLBOX_DOCKER_IMAGE, so I could set it to my favorite distro, and not be forced to use the default...

Yeah I like that too. Choosing a different distro or a different container with more things layered into it would be the two use cases I can think of.

[debarshiray]

We have been working on a pet toolbox container for Silverblue [1, 2] based on rootless podman and buildah. It's also loosely inspired from https://github.com/coreos/toolbox. The current status is basically chasing down bugs in rootless podman and getting them fixed.

[1] https://pagure.io/releng/issue/7698
[2] https://discussion.fedoraproject.org/t/an-image-for-a-fedora-toolbox-container/286

[yuqi-zhang]

From the discussions today we have identified that there are differences in the Silverblue/FCOS/RHCOS usecases for toolbox. The key difference is that the Silverblue toolbox will be designed for development, and is currently exploring rootless options (more details: https://fedoraproject.org/wiki/Workstation/Desktop_Containers). For FCOS/RHCOS it is more of a simple wrapper (does not need rootless) mainly designed for troubleshooting (via SSH).

There is also the consideration of multiple toolboxes, or at least the ability to switch out the base container for toolbox, so people can have more customization options on package selection, etc.

At the moment /bin/sh should be enough, but we may explore go/rust as alternatives if more functionality is required.

[dustymabe]

an interesting way to have both is to have the tool assume if user is root then do you are "troubleshooting" and if you are non-root then you are "developing". Could be an interesting way to fill both use cases with one tool.

[debarshiray]

I put up my toolbox script at https://github.com/debarshiray/fedora-toolbox

I had initially considered forking coreos/toolbox and adding a commit on top, but then I wasn't sure what to do with the governance and process related bits in there. So, I went with a separate repository. It's really just a toy, so nothing is set in stone or anything. We can move the repository, rename the script, whatever. I just wanted something better than a tarball to point people at.

[miabbott]

I saw this show up over the weekend:

https://discussion.fedoraproject.org/t/what-workflows-do-developers-using-silverblue-use/248/14

Might be something to investigate to see if anything could be used for the toolbox script/container idea.

[yuqi-zhang]

I also created a (WIP) script for RHCOS at: coreos/toolbox#44 , since the use case will be different for RHCOS vs, Silverblue, this script is more for debugging the host and was kept similar to the original.

[dustymabe]

@yuqi-zhang - any updates on this? I see the upstream PR got merged. What do we need to do to get that into Fedora CoreOS?

[debarshiray]

an interesting way to have both is to have the tool assume if user is root
then do you are "troubleshooting" and if you are non-root then you are
"developing". Could be an interesting way to fill both use cases with one
tool.

For what it's worth, fedora-toolbox has a hidden --sudo flag to switch to rootful use. However, not being intimately familiar with CoreOS use-cases, I haven't done anything further in that direction.

[debarshiray]

There is also the consideration of multiple toolboxes, or
at least the ability to switch out the base container for toolbox,
so people can have more customization options on package
selection, etc.

Some of this already works with fedora-toolbox today. See --help.

[yuqi-zhang]

@dustymabe could you clarify on what you mean by getting it FCOS? The script in the coreos/toolbox repo is built into RHCOS systems via https://github.com/projectatomic/rpmdistro-gitoverlay , and afaik that is the only use of it so far (and the only way we planned to build it). If you want to start the same build process for FCOS that is an option.

@debarshiray Based on #38 (comment) I made the assumption that we were going to split the toolbox(s) for *COS/Silverblue, and thus I created the very small and mostly 1-to-1 functionality port from the existing coreos/toolbox. But since the functionality is included in https://github.com/debarshiray/fedora-toolbox as well, mayhaps moving forward we can look to move to using this one as well for *COS systems? What do you think @dustymabe

Sorry for the delay, for some reason I did not get pinged.

[dustymabe]

@dustymabe could you clarify on what you mean by getting it FCOS? The script in the coreos/toolbox repo is built into RHCOS systems via https://github.com/projectatomic/rpmdistro-gitoverlay , and afaik that is the only use of it so far (and the only way we planned to build it). If you want to start the same build process for FCOS that is an option.

Yeah I was thinking of getting that as an rpm into fedora and then pulling it into FCOS.

@debarshiray Based on #38 (comment) I made the assumption that we were going to split the toolbox(s) for *COS/Silverblue, and thus I created the very small and mostly 1-to-1 functionality port from the existing coreos/toolbox. But since the functionality is included in https://github.com/debarshiray/fedora-toolbox as well, mayhaps moving forward we can look to move to using this one as well for *COS systems? What do you think @dustymabe

There is some discussion based on that over in containers/toolbox#8 . My opinion is that it would be nice if they were shared, but if it doesn't make sense to do that then it doesn't make sense and that's ok. Can we carry on that discussion over in containers/toolbox#8 ?

[debarshiray]

@debarshiray Based on #38 (comment)
I made the assumption that we were going to split the toolbox(s)

Yes, we did identify the differences during our chat in #fedora-coreos but my understanding was that we still wanted to share it as much as possible. I assumed that people were busy with other aspects of CoreOS and haven't had a chance to take a close look at fedora-toolbox.

for *COS/Silverblue, and thus I created the very small and mostly 1-to-1
functionality port from the existing coreos/toolbox. But since the functionality
is included in https://github.com/debarshiray/fedora-toolbox as well, mayhaps
moving forward we can look to move to using this one as well for *COS
systems?

I don't know if fedora-toolbox in its current state will work out of the box for CoreOS. I have been working on it from the Silverblue angle, and even though I am aware of CoreOS' needs, it's better if somebody more closely involved in CoreOS would try it out, give feedback, etc..

[debarshiray]

There is some discussion based on that over in debarshiray/fedora-toolbox#8 . My opinion is that it would be nice if they were shared, but if it doesn't make sense to do that then it doesn't make sense and that's ok. Can we carry on that discussion over in debarshiray/fedora-toolbox#8 ?

Yes, let's move over to debarshiray/fedora-toolbox#8

[mike-nguyen]

I was able to get toolbox running on FCOS. There are two issues with the latest toolbox rpm (toolbox-0.0.10-1.fc30.noarch).

The first issue is that the /dev/dri bind mount does not exist in FCOS:

toolbox: trying to create container fedora-toolbox-30
Error: error checking path "/dev/dri": stat /dev/dri: no such file or directory
toolbox: failed to create container fedora-toolbox-30

Removing the bind mount in toolbox allows toolbox container to be created.

The second issue is that the toolbox enter command has to be run twice because of a possible podman cp issue.

Error: unable to find user core: no matching entries in passwd file
foolbox: copying /etc/profile.d/toolbox.sh to container fedora-toolbox-30
Error: cannot copy into running rootless container with pause set - pass --pause=false to force copying
foolbox: unable to copy /etc/profile.d/toolbox.sh to container fedora-toolbox-30

The latest toolbox master has --pause=false to fix the problem but I am seeing another error message that I commented on [0]. Running toolbox enter twice still works around it.

The latest toolbox master adds an addition issue because it leverages flatpak to keep the container's /etc/timezone and /etc/localtime synchronized with the host[1]. Unfortunately flatpak is not installed on FCOS.

toolbox: calling org.freedesktop.Flatpak.SessionHelper.RequestSession
Error connecting: Cannot autolaunch D-Bus without X11 $DISPLAY
toolbox: failed to call org.freedesktop.Flatpak.SessionHelper.RequestSession

I'm mainly concerned about the /etc/timezone and /etc/localtime issue because I'm not sure if there is another solution that will work for both Silverblue and FCOS. The other issues seem like they be handled or fixed.

[0] https://github.com/debarshiray/toolbox/issues/198#issuecomment-502867168
[1] debarshiray/toolbox@8db414d

[mike-nguyen]

Do we want to continue down the path of using toolbox or using https://github.com/coreos/toolbox since the use case is much different?

[cgwalters]

I am also working on https://github.com/cgwalters/coretoolbox FWIW

[mike-nguyen]

The size increase might not be so bad anymore. :)

See: flatpak/flatpak#3052

Currently the list is down to:

	linux-vdso.so.1 (0x00007ffeb1371000)
	libgio-2.0.so.0 => /lib64/libgio-2.0.so.0 (0x00007f9fe17c3000)
	libgobject-2.0.so.0 => /lib64/libgobject-2.0.so.0 (0x00007f9fe176d000)
	libglib-2.0.so.0 => /lib64/libglib-2.0.so.0 (0x00007f9fe164d000)
	libarchive.so.13 => /lib64/libarchive.so.13 (0x00007f9fe1590000)
	libxml2.so.2 => /lib64/libxml2.so.2 (0x00007f9fe1426000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f9fe1404000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f9fe123c000)
	libgmodule-2.0.so.0 => /lib64/libgmodule-2.0.so.0 (0x00007f9fe1236000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f9fe1230000)
	libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f9fe11bd000)
	libffi.so.6 => /lib64/libffi.so.6 (0x00007f9fe11b2000)
	libz.so.1 => /lib64/libz.so.1 (0x00007f9fe1198000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f9fe1169000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f9fe114f000)
	libmount.so.1 => /lib64/libmount.so.1 (0x00007f9fe10f2000)
	libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007f9fe0e12000)
	libacl.so.1 => /lib64/libacl.so.1 (0x00007f9fe0e07000)
	liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f9fe0dde000)
	libzstd.so.1 => /lib64/libzstd.so.1 (0x00007f9fe0d3b000)
	liblz4.so.1 => /lib64/liblz4.so.1 (0x00007f9fe0d1c000)
	libbz2.so.1 => /lib64/libbz2.so.1 (0x00007f9fe0d09000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f9fe0b85000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f9fe1997000)
	libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007f9fe0af6000)
	libblkid.so.1 => /lib64/libblkid.so.1 (0x00007f9fe0aa1000)
	libuuid.so.1 => /lib64/libuuid.so.1 (0x00007f9fe0a98000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f9fe0a8e000)
	libattr.so.1 => /lib64/libattr.so.1 (0x00007f9fe0a86000)

There are a few more things that can be trimmed.

This is awesome @debarshiray!

[debarshiray]

The list is now down to:

	linux-vdso.so.1 (0x00007ffce0fb1000)
	libgio-2.0.so.0 => /lib64/libgio-2.0.so.0 (0x00007efe98239000)
	libgobject-2.0.so.0 => /lib64/libgobject-2.0.so.0 (0x00007efe981e3000)
	libglib-2.0.so.0 => /lib64/libglib-2.0.so.0 (0x00007efe980c3000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007efe980a1000)
	libc.so.6 => /lib64/libc.so.6 (0x00007efe97edb000)
	libgmodule-2.0.so.0 => /lib64/libgmodule-2.0.so.0 (0x00007efe97ed5000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007efe97ecd000)
	libpcre.so.1 => /lib64/libpcre.so.1 (0x00007efe97e5a000)
	libffi.so.6 => /lib64/libffi.so.6 (0x00007efe97e4f000)
	libz.so.1 => /lib64/libz.so.1 (0x00007efe97e35000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x00007efe97e08000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007efe97dee000)
	libmount.so.1 => /lib64/libmount.so.1 (0x00007efe97d8f000)
	/lib64/ld-linux-x86-64.so.2 (0x00007efe9840d000)
	libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007efe97d00000)
	libblkid.so.1 => /lib64/libblkid.so.1 (0x00007efe97cad000)
	libuuid.so.1 => /lib64/libuuid.so.1 (0x00007efe97ca4000)
	librt.so.1 => /lib64/librt.so.1 (0x00007efe97c9a000)

Does this look better now?

[mike-nguyen]

@debarshiray should be much better. Every bit helps!

[dustymabe]

looks like we need to reconvene and make a decision on this.

[dustymabe]

some data points:

Today on f30 coreos this would add:

Added:
  abattis-cantarell-fonts-0.111-2.fc30.noarch
  adobe-source-code-pro-fonts-2.030.1.050-6.fc30.noarch
  dconf-0.32.0-1.fc30.x86_64
  flatpak-session-helper-1.4.2-4.fc30.x86_64
  fontpackages-filesystem-1.44-24.fc30.noarch
  glib-networking-2.60.3-1.fc30.x86_64
  gsettings-desktop-schemas-3.32.0-1.fc30.x86_64
  libXau-1.0.9-1.fc30.x86_64
  libmodman-2.0.1-19.fc30.x86_64
  libproxy-0.4.15-13.fc30.x86_64
  libsoup-2.66.2-1.fc30.x86_64
  toolbox-0.0.12-1.fc30.noarch

Today on a rawhide build of FCOS this would add:

Added:
  abattis-cantarell-fonts-0.111-3.fc31.noarch
  adobe-source-code-pro-fonts-2.030.1.050-7.fc31.noarch
  dconf-0.34.0-1.fc32.x86_64
  flatpak-session-helper-1.4.2-5.fc31.x86_64
  fontpackages-filesystem-1.44-25.fc31.noarch
  glib-networking-2.62.0-1.fc32.x86_64
  gsettings-desktop-schemas-3.34.0-1.fc32.x86_64
  libXau-1.0.9-2.fc31.x86_64
  libmodman-2.0.1-20.fc31.x86_64
  libproxy-0.4.15-15.fc32.x86_64
  libsoup-2.67.93-1.fc32.x86_64
  toolbox-0.0.13-1.fc32.noarch

[dustymabe]

@debarshiray is there a new toolbox rpm out with the decreased set of deps? or maybe a copr somewhere we can use?

[lucab]

My 2c: the dconf package brings in a binary which is directly available in PATH (/usr/bin/dconf). Those are usually hard to mask, and hard to get rid of in the future.

[dustymabe]

Discussed in the Fedora CoreOS meeting today. :

#info we want to ship something like toolbox and toolbox is 
something that already exists so we will ship it once we've
iterated on the dependencies in #38. Will work with debarshiray
to see if there is an updated rpm we can evaluate.

[debarshiray]

@debarshiray is there a new toolbox rpm out with the decreased
set of deps? or maybe a copr somewhere we can use?

These dependencies are all coming in via the flatpak-session-helper package.

Today on f30 coreos this would add:

Added:
  abattis-cantarell-fonts-0.111-2.fc30.noarch
  adobe-source-code-pro-fonts-2.030.1.050-6.fc30.noarch
  dconf-0.32.0-1.fc30.x86_64
  flatpak-session-helper-1.4.2-4.fc30.x86_64
  fontpackages-filesystem-1.44-24.fc30.noarch
  glib-networking-2.60.3-1.fc30.x86_64
  gsettings-desktop-schemas-3.32.0-1.fc30.x86_64
  libXau-1.0.9-1.fc30.x86_64
  libmodman-2.0.1-19.fc30.x86_64
  libproxy-0.4.15-13.fc30.x86_64
  libsoup-2.66.2-1.fc30.x86_64
  toolbox-0.0.12-1.fc30.noarch

Today on a rawhide build of FCOS this would add:

Added:
  abattis-cantarell-fonts-0.111-3.fc31.noarch
  adobe-source-code-pro-fonts-2.030.1.050-7.fc31.noarch
  dconf-0.34.0-1.fc32.x86_64
  flatpak-session-helper-1.4.2-5.fc31.x86_64
  fontpackages-filesystem-1.44-25.fc31.noarch
  glib-networking-2.62.0-1.fc32.x86_64
  gsettings-desktop-schemas-3.34.0-1.fc32.x86_64
  libXau-1.0.9-2.fc31.x86_64
  libmodman-2.0.1-20.fc31.x86_64
  libproxy-0.4.15-15.fc32.x86_64
  libsoup-2.67.93-1.fc32.x86_64
  toolbox-0.0.13-1.fc32.noarch

With flatpak/flatpak#3052 in place, most of those shouldn't be necessary. Specifically the fonts, dconf and libXau should be gone, but quite possibly a few more of the others should go as well. We brought down the number of ELF linkages to 19 from a whopping 62. :)

The current list is:

	linux-vdso.so.1 (0x00007ffce0fb1000)
	libgio-2.0.so.0 => /lib64/libgio-2.0.so.0 (0x00007efe98239000)
	libgobject-2.0.so.0 => /lib64/libgobject-2.0.so.0 (0x00007efe981e3000)
	libglib-2.0.so.0 => /lib64/libglib-2.0.so.0 (0x00007efe980c3000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007efe980a1000)
	libc.so.6 => /lib64/libc.so.6 (0x00007efe97edb000)
	libgmodule-2.0.so.0 => /lib64/libgmodule-2.0.so.0 (0x00007efe97ed5000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007efe97ecd000)
	libpcre.so.1 => /lib64/libpcre.so.1 (0x00007efe97e5a000)
	libffi.so.6 => /lib64/libffi.so.6 (0x00007efe97e4f000)
	libz.so.1 => /lib64/libz.so.1 (0x00007efe97e35000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x00007efe97e08000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007efe97dee000)
	libmount.so.1 => /lib64/libmount.so.1 (0x00007efe97d8f000)
	/lib64/ld-linux-x86-64.so.2 (0x00007efe9840d000)
	libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007efe97d00000)
	libblkid.so.1 => /lib64/libblkid.so.1 (0x00007efe97cad000)
	libuuid.so.1 => /lib64/libuuid.so.1 (0x00007efe97ca4000)
	librt.so.1 => /lib64/librt.so.1 (0x00007efe97c9a000)

Sadly, there hasn't been a new upstream Flatpak release containing flatpak/flatpak#3052

We can certainly consider carrying the patches downstream in Fedora to tide you over in the short-term.

[cgwalters]

I'd still prefer we didn't ship any toolbox until we've resolved the implementation of which one; particularly the discussion in https://github.com/debarshiray/toolbox/pull/100

[dustymabe]

@debarshiray
Sadly, there hasn't been a new upstream Flatpak release containing flatpak/flatpak#3052

maybe we can poke and see if they'll be ready for one soon?

@debarshiray
We can certainly consider carrying the patches downstream in Fedora to tide you over in the short-term.

I think it would be nice to get a scratch build of a package with that change just so we can see what the deps will be.

@cgwalters
I'd still prefer we didn't ship any toolbox until we've resolved the implementation of which one; particularly the discussion in debarshiray/toolbox#100

That discussion looks deep. Would you and @debarshiray want to lay out the points at the meeting tomorrow?

[debarshiray]

I think it would be nice to get a scratch build of a package with that change
just so we can see what the deps will be.

I have backported the patches to Fedora 32 or Rawhide and submitted a build.

Here are some scratch builds:

• F32: https://koji.fedoraproject.org/koji/taskinfo?taskID=37721635
• F31: https://koji.fedoraproject.org/koji/taskinfo?taskID=37721648

If it works fine, we can do official builds for F31 and such too.

[cgwalters]

I moved the "cgwalters/coretoolbox vs debarshiray/toolbox" issue to https://github.com/debarshiray/toolbox/issues/265

[dustymabe]

@debarshiray
Here are some scratch builds

Thanks. I'll take a look

@cgwalters
I moved the "cgwalters/coretoolbox vs debarshiray/toolbox" issue to debarshiray/toolbox#265

We discussed this in the Fedora CoreOS meeting today. I have added a comment to the issue you opened.

[dustymabe]

@debarshiray
Here are some scratch builds

Thanks. I'll take a look

OK this looks much better!

Added:
  flatpak-session-helper-1.4.2-6.fc32.x86_64
  toolbox-0.0.14-1.fc32.noarch

Thanks @debarshiray! An official build for f31 would be great

[cgwalters]

I'd still prefer we didn't ship it until we've resolved the discussions on project direction.

[dustymabe]

I'd still prefer we didn't ship it until we've resolved the discussions on project direction.

I thought that's what we did already.

[debarshiray]

Added:
  flatpak-session-helper-1.4.2-6.fc32.x86_64
  toolbox-0.0.14-1.fc32.noarch

Thanks @debarshiray! An official build for f31 would be great

Fedora 31: https://bodhi.fedoraproject.org/updates/FEDORA-2019-7c75e9dbcf
Fedora 30: https://bodhi.fedoraproject.org/updates/FEDORA-2019-7d457deb52

[keithy]

#186

I was surprised to find toolbox in FCOS since I thought this is supposed to be a minimal OS and was expecting things to be removed rather than added.

Could we consider moving to providing a standard means to install 3rd party packages outside of overlays, such that those packages can be called in via ignition. Toolbox would only be one of such packages.

I set about developing such an install mechanism which includes a script and a unit that could be included as standard (or enabled via ignition)

https://github.com/keithy/ign-tool/wiki/plug:install-packages

[dustymabe]

Maybe you're mistaking what toolbox is?

toolbox is a convenient interactive tool for running containers. It has nothing to do with installing packages in the host. For Fedora CoreOS the most useful piece is being able to run a container that has some extra debugging tools installed in case your system is in a bad state and you need to debug it but proper tooling isn't there already.

Ideally your systems privision automatically (via ignition) and you never need to run toolbox interactively to debug your system.

Does that help?