ci: Build & Publish Fedora Toolbx images with GitHub Packages

[HarryMichal]

For the sake of greater control over the testing of images and for having an
infrustructure for hosting images that are not endorsed by the distirbutions.

[debarshiray]

Umm... what's the plan here?

I am not sure we want to have a separate place to publish images that are not endorsed by the distributions? See my comment on #759

It won't be prudent to suggest any level of official support for an OS distribution unless we have some serious commitment from that community, and in that case, we use the registry chosen by that distribution. Anything that's part of the Toolbx source code repository is official, and in this case will hold up our CI because breakages will need to be fixed.

What we need is a way to test our image definitions in images so that we get immediate feedback that changes to the images work with the toolbox tool. Otherwise, we need to wait for the changes to propagate through the distribution's build system and updates, which can take a few days or weeks in the case of Fedora, and several months for RHEL.

However, we don't need to separately publish these, and we would still need to continue testing the images published through the distributions' chosen registries.

[HarryMichal]

Umm... what's the plan here?

Mainly the plan is to have an easy-to-operate platform for testing images. Despite hosting fedora-toolbox images on Fedora infra we keep them in sync in this repository. I presume the same will happen to other supported images (of course, with the exception of closed downstream versions like UBI). So the testing may very well happen here rather than "upstream". I even suspect that despite usually having an OCI image infrastructure (or borrowing it from likes of Docker) distributions lack a testing infrastructure for these images.

I am not sure we want to have a separate place to publish images that are not endorsed by the distributions? See my comment on #759

I added the publish step mostly because I could 😅. It was not required but I wanted to showcase what this relatively simple pipeline is capable of doing for us, including the very last step.

It won't be prudent to suggest any level of official support for an OS distribution unless we have some serious commitment from that community, and in that case, we use the registry chosen by that distribution.

Agreed.

What we need is a way to test our image definitions in images so that we get immediate feedback that changes to the images work with the toolbox tool. Otherwise, we need to wait for the changes to propagate through the distribution's build system and updates, which can take a few days or weeks in the case of Fedora, and several months for RHEL.

The pipeline by this seems to me like the perfect place for the testing.

However, we don't need to separately publish these, and we would still need to continue testing the images published through the distributions' chosen registries.

I agree with not publishing the images but the testing on the distribution level depends on the testing infrastructure of the distributions. The easiest way about testing is to do it here and asynchronously report potential problems to distributions.

[debarshiray]

(of course, with the exception of closed downstream versions like UBI).

The definitions of the UBI images is public.

[debarshiray]

I am not sure we want to have a separate place to publish images that are
not endorsed by the distributions? See my comment on #759

I added the publish step mostly because I could sweat_smile. It was not
required but I wanted to showcase what this relatively simple pipeline is
capable of doing for us, including the very last step.

Ok. Publishing a separate set of images that users can actually use, can become a problem. We shouldn't do that.

What we need is a way to
test our image definitions
in images so that we get immediate feedback that changes to the images
work with the toolbox tool. Otherwise, we need to wait for the changes to
propagate through the distribution's build system and updates, which can take
a few days or weeks in the case of Fedora, and several months for RHEL.

The pipeline by this seems to me like the perfect place for the testing.

Why not a simple podman build ?

(Remember that our test-suite is used by downstream distributors.)

I don't want to depend on a wide cross-section of remote infrastructure -- each with their own cryptic YAML files and hard to decipher logs. It's really really painful when things break or need to be changed -- documentations is sometimes non-existent, logs require magic tricks to get to. It also takes us away from our desire to be able to run the entire test suite locally from ninja test.

We should really make our test suite easier to use. Consolidating on a smaller set of things, making it possible to run it locally, etc. seem like steps in that direction.

However, we don't need to separately publish these, and we would still need to
continue testing the images published through the distributions' chosen registries.

I agree with not publishing the images but the testing on the distribution level
depends on the testing infrastructure of the distributions. The easiest way about
testing is to do it here and asynchronously report potential problems to distributions.

It's not about testing on the distribution level. We need to test what the users see (and will see in the near future).

At any given time, vast majority of users are going to be using the official images from the distributions. So we need to keep testing those images from those URLs, because we can't break them. Rebuilding the images as part of the test suite doesn't replace this, because differences in the Podman stack or Docker versus Podman and such can start to matter.

Then there's the case, where we might be trying to make some changes to the images, and we need immediate feedback if those changes will work or not. We also need that feedback across all supported distributions. This is already a bit non-trivial right now, and it's going to get much worse as we add more distributions. That's why we carry a copy of the image definitions in our code repository, because otherwise things would be even worse. To improve things, we should build images from our in-tree definitions and run the test suite on them too.

Finally, there are several things that can trip us up at some unknown time in the future. The distro's base images might have changed in ways that we didn't expect, toolchain problems, etc.. Rebuilding the images as part of our tests will protect us from such surprises by flagging them well in advance.

Ideally, we would also build the images against the toolbox(1) from the latest release.

The main point is that Toolbx consists of two main parts -- the toolbox(1) tool and the OCI images (which are themselves made from the base image). It's good to have some overlap when testing to prevent things from falling through the cracks.

[debarshiray]

This, linked to from https://pagure.io/fedora-ci/general/issue/311, is one of the reasons why I want to consolidate our test suite around ninja test.

While there's a lot of value in triggering the CI automatically for each pull request, merge, and such, when things go wrong, it's invaluable to be able to run it locally. Otherwise it feels like there's a wall between the human and the test suite.