Use installed toolbox instead of binary from host

[kuba3351]

I hit an issue (#821) that was caused by GLIBC mismatch between host and container, so toolbox binary didn't worked inside container. I have an idea to prevent this issue by not using toolbox binary from host, but install toolbox binary in the container using dnf. It will install the binary that is always able to run in the container, so I think this issue should not happen again.

This breaks compatibility with images that does not have installed toolbox, so I think a fllback logic should be implemented that checks if there is an /usr/bin/toolbox in the container, and if not, will mount the binary from host like before to have backward compatibility.

What do you think about?

[softwarefactory-project-zuul]

Build failed.

• unit-test : SUCCESS in 2m 33s
• system-test-fedora-33 : FAILURE in 8m 22s
• system-test-fedora-34 : FAILURE in 9m 04s
• system-test-fedora-rawhide : FAILURE in 8m 45s

[debarshiray]

I already commented on #821 about the overall problem, and the way to address it.

As for this specific pull request ...

I am apprehensive of involving RPM transactions as part of the container creation process. It makes things non-deterministic. Among other things, RPM packages have scriptlets, which get run as part of the transaction on user machines. That opens up a huge possibility for things to go wrong.

That's why OSTree-based OSes don't run RPM scriptlets on user machines. Scriptlets are discouraged, and if really necessary are run on the server, so that the outcome is the same for everybody.

Secondly, using the toolbox binary from the host was an explicit design choice. It means that the communication between the host and the container doesn't need to be a stable protocol, which is a huge benefit. This is also why online updates are implicitly unsupported. It means that we can change Toolbox, and expect that the user will reboot the machine as part of the update, which will restart all containers with the new version of Toolbox, keeping both sides synchronized.

Finally, fallback logic comes with the downside of exploding the test matrix or, even worse, being impossible to test. In this case, we will end up creating two subtly different kinds of Toolbox containers. Those that have toolbox mounted from the host, and those that have it installed natively inside the container. Over time, they will have different backwards compatibility requirements - backwards compatibility is tricky enough, this will be a lot worse.

Go does have the downside of having huge binaries, but the upside of that is that the binaries are mostly self-contained. So let's try to take advantage of that as much as possible. With a programming language like C, it would be a lot harder to mount in the binary across OSes from the host to the container, like we do now, because of the pervasive use of shared libraries.

[debarshiray]

Anyway, thanks for stopping by, and taking the time to work on this problem. I really appreciate that.