[master] feat: build Fedora 41 RPM packages (carry #1055)

[thaJeztah]

• depends on [master] install_rpm_containerd: add support for dnf5 #1059
• carries feat: build Fedora 41 RPM packages #1055
• closes feat: build Fedora 41 RPM packages #1055

- What I did

It's nearing that time again. A new Fedora release.
This is part 2 of 2 for packaging Docker CE for Fedora 41. Part 1 is awaiting review in the docker/containerd-packaging repository.

Fedora have not (yet) set a concrete date for the EOL of Fedora 41, however it will be in or around November 2025.

- How I did it

The same as the previous two releases - added a line for Fedora 41 in the Jenkinsfile, updated the list of OSes in the Makefile and created a new Dockerfile using the new Fedora 41 image.

- How to verify it

I have ran make fedora from the ./rpms directory, and a RPM file was successfully created.

- Description for the changelog
Enable Docker CE builds for Fedora 41

- A picture of a cute animal (not mandatory but encouraged)

[thaJeztah]

Let's see if CI is happy, but it' possible we may need something similar as;

• [main] rpm: install binutils-gold to work around build failure on Fedora 41 on arm64 containerd-packaging#390

[thaJeztah]

Interesting; compiling works, but looks like the verify script needs an update;

othing to do.
+ install_rpm_containerd
+ '[' prod = stage ']'
+ REPO_URL=https://download.docker.com/linux/fedora/docker-ce.repo
+ echo '[DEBUG] Installing engine dependencies from https://download.docker.com/linux/fedora/docker-ce.repo'
+ dnf --version
[DEBUG] Installing engine dependencies from https://download.docker.com/linux/fedora/docker-ce.repo
dnf5 version 5.2.5.0
dnf5 plugin API version 2.0
libdnf5 version 5.2.5.0
libdnf5 plugin API version 2.0

Loaded dnf5 plugins:
  name: builddep
  version: 1.0.0
  API version: 2.0

  name: changelog
  version: 1.0.0
  API version: 2.0

  name: config-manager
  version: 0.1.0
  API version: 2.0

  name: copr
  version: 0.1.0
  API version: 2.0

  name: needs_restarting
  version: 1.0.0
  API version: 2.0

  name: repoclosure
  version: 1.0.0
  API version: 2.0
+ dnf config-manager --add-repo https://download.docker.com/linux/fedora/docker-ce.repo
Unknown argument "--add-repo" for command "config-manager". Add "--help" for more information about the arguments.
make: *** [Makefile:95: verify] Error 2
script returned exit code 2

[thaJeztah]

Ugh... looks like Fedora 41 ships with dnf5 (https://github.com/rpm-software-management/dnf5), which is a complete rewrite of the command?
https://www.reddit.com/r/Fedora/comments/1f3o6s4/problems_when_installing_docker_engine_with_dnf/

On Fedora 40, dnf config-manager is an optional install;

docker run -it --rm fedora:41

dnf config-manager --help
No such command: config-manager. Please use /usr/bin/dnf --help
It could be a DNF plugin command, try: "dnf install 'dnf-command(config-manager)'"

dnf install 'dnf-command(config-manager)'

dnf config-manager --help
usage: dnf config-manager [-c [config file]] [-q] [-v] [--version] [--installroot [path]] [--nodocs] [--noplugins] [--enableplugin [plugin]] [--disableplugin [plugin]] [--releasever RELEASEVER] [--setopt SETOPTS] [--skip-broken] [-h] [--allowerasing] [-b | --nobest] [-C]
[-R [minutes]] [-d [debug level]] [--debugsolver] [--showduplicates] [-e ERRORLEVEL] [--obsoletes] [--rpmverbosity [debug level name]] [-y] [--assumeno] [--enablerepo [repo]] [--disablerepo [repo] | --repo [repo]] [--enable | --disable] [-x [package]]
[--disableexcludes [repo]] [--repofrompath [repo,path]] [--noautoremove] [--nogpgcheck] [--color COLOR] [--refresh] [-4] [-6] [--destdir DESTDIR] [--downloadonly] [--comment COMMENT] [--bugfix] [--enhancement] [--newpackage] [--security]
[--advisory ADVISORY] [--bz BUGZILLA] [--cve CVES] [--sec-severity {Critical,Important,Moderate,Low}] [--forcearch ARCH] [--save] [--add-repo URL] [--dump] [--dump-variables] [--set-enabled | --set-disabled]
[section ...]

manage dnf configuration options and repositories

General DNF options:
  -c [config file], --config [config file]
                        config file location
  -q, --quiet           quiet operation
  -v, --verbose         verbose operation
  --version             show DNF version and exit
  --installroot [path]  set install root
  --nodocs              do not install documentations
  --noplugins           disable all plugins
  --enableplugin [plugin]
                        enable plugins by name
  --disableplugin [plugin]
                        disable plugins by name
  --releasever RELEASEVER
                        override the value of $releasever in config and repo files
  --setopt SETOPTS      set arbitrary config and repo options
  --skip-broken         resolve depsolve problems by skipping packages
  -h, --help, --help-cmd
                        show command help
  --allowerasing        allow erasing of installed packages to resolve dependencies
  -b, --best            try the best available package versions in transactions.
  --nobest              do not limit the transaction to the best candidate
  -C, --cacheonly       run entirely from system cache, don't update cache
  -R [minutes], --randomwait [minutes]
                        maximum command wait time
  -d [debug level], --debuglevel [debug level]
                        debugging output level
  --debugsolver         dumps detailed solving results into files
  --showduplicates      show duplicates, in repos, in list/search commands
  -e ERRORLEVEL, --errorlevel ERRORLEVEL
                        error output level
  --obsoletes           enables dnf's obsoletes processing logic for upgrade or display capabilities that the package obsoletes for info, list and repoquery
  --rpmverbosity [debug level name]
                        debugging output level for rpm
  -y, --assumeyes       automatically answer yes for all questions
  --assumeno            automatically answer no for all questions
  --enablerepo [repo]   Temporarily enable repositories for the purpose of the current dnf command. Accepts an id, a comma-separated list of ids, or a glob of ids. This option can be specified multiple times.
  --disablerepo [repo]  Temporarily disable active repositories for the purpose of the current dnf command. Accepts an id, a comma-separated list of ids, or a glob of ids. This option can be specified multiple times, but is mutually exclusive with `--repo`.
  --repo [repo], --repoid [repo]
                        enable just specific repositories by an id or a glob, can be specified multiple times
  --enable              enable repos with config-manager command (automatically saves)
  --disable             disable repos with config-manager command (automatically saves)
  -x [package], --exclude [package], --excludepkgs [package]
                        exclude packages by name or glob
  --disableexcludes [repo], --disableexcludepkgs [repo]
                        disable excludepkgs
  --repofrompath [repo,path]
                        label and path to an additional repository to use (same path as in a baseurl), can be specified multiple times.
  --noautoremove        disable removal of dependencies that are no longer used
  --nogpgcheck          disable gpg signature checking (if RPM policy allows)
  --color COLOR         control whether color is used
  --refresh             set metadata as expired before running the command
  -4                    resolve to IPv4 addresses only
  -6                    resolve to IPv6 addresses only
  --destdir DESTDIR, --downloaddir DESTDIR
                        set directory to copy packages to
  --downloadonly        only download packages
  --comment COMMENT     add a comment to transaction
  --bugfix              Include bugfix relevant packages, in updates
  --enhancement         Include enhancement relevant packages, in updates
  --newpackage          Include newpackage relevant packages, in updates
  --security            Include security relevant packages, in updates
  --advisory ADVISORY, --advisories ADVISORY
                        Include packages needed to fix the given advisory, in updates
  --bz BUGZILLA, --bzs BUGZILLA
                        Include packages needed to fix the given BZ, in updates
  --cve CVES, --cves CVES
                        Include packages needed to fix the given CVE, in updates
  --sec-severity {Critical,Important,Moderate,Low}, --secseverity {Critical,Important,Moderate,Low}
                        Include security relevant packages matching the severity, in updates
  --forcearch ARCH      Force the use of an architecture

Config-manager command-specific options:
  --save                save the current options (useful with --setopt)
  --add-repo URL        add (and enable) the repo from the specified file or url
  --dump                print current configuration values to stdout
  --dump-variables      print variable values to stdout
  --set-enabled         enable repos (automatically saves)
  --set-disabled        disable repos (automatically saves)

  section               configuration sections to manage, "main" or repoid

Fedora 41:

docker run -it --rm fedora:41 dnf config-manager --help
Usage:
  dnf5 [GLOBAL OPTIONS] config-manager <COMMAND> ...

Description:
  Manage main and repositories configuration, variables and add new repositories.

Commands:
  addrepo                       Add repositories from the specified configuration file or define a new repository using user options
  setopt                        Set configuration and repositories options
  unsetopt                      Unset/remove configuration and repositories options
  setvar                        Set variables
  unsetvar                      Unset/remove variables

[thaJeztah]

so I started updating the verify script to work with dnf5; ran into some issues;

docker-ce-packaging/install-containerd-helpers

Lines 29 to 30 in d757a76

	dnf config-manager --set-disabled docker-ce-*
	dnf config-manager --set-enabled docker-ce-test

It looks like dnf itself still has --enablerepo and --disablerepo flags (aliases for --enable-repo and --disable-repo;

--enable-repo=REPO_ID,...              Enable additional repositories. List option. Supports globs, can be specified multiple times.
  --disable-repo=REPO_ID,...             Disable repositories. List option. Supports globs, can be specified multiple times.

But if those are the equivalent to the dnf / dnf4 command, they won't persist changes, so it's only for a single invocation;

--enablerepo [repo]   Temporarily enable repositories for the purpose of the current dnf command. Accepts an id, a comma-separated list of ids, or a glob of ids. This option can be specified multiple
                        times.
  --disablerepo [repo]  Temporarily disable active repositories for the purpose of the current dnf command. Accepts an id, a comma-separated list of ids, or a glob of ids. This option can be specified
                        multiple times, but is mutually exclusive with `--repo`.

So the alternative is to use dnf5 config-manger setopt repo_id.enabled=0; https://github.com/rpm-software-management/dnf5/blob/58cfa3f3e5768665f2d6a1b6f06eb1ec3451a5fc/doc/commands/repo.8.rst#examples

• Document how to enable and disable repositories rpm-software-management/dnf5#1213

However, that doesn't support glob / wildcards, so we may need to do a dnf repo list to get the IDs of the repositories to disable;

dnf repo list --all docker-ce-*
repo id                                                                                                        repo name                                                                                                                                                         status
docker-ce-stable                                                                                               Docker CE Stable - aarch64                                                                                                                                       enabled
docker-ce-stable-debuginfo                                                                                     Docker CE Stable - Debuginfo aarch64                                                                                                                            disabled

Perhaps using --json;

dnf repo list --json --enabled docker-ce-*
[
  {
    "id":"docker-ce-stable",
    "name":"Docker CE Stable - aarch64",
    "is_enabled":true
  }
]

However, I ran into an issue; it looks like dnf5 is not compatible with the existing .repo files we have;

dnf config-manager addrepo --from-repofile="https://download.docker.com/linux/fedora/docker-ce.repo"
Error in added repository configuration file. Cannot set repository option "#1=
": Option "#1" not found

After some debugging, it looks like the problem is in empty lines in the file; individual entries work (so it's not an issue in any of those), but the empty lines between them; i.e., this fails;

[docker-ce-stable]
name=Docker CE Stable - $basearch
baseurl=https://download.docker.com/linux/fedora/$releasever/$basearch/stable
enabled=1
gpgcheck=1
gpgkey=https://download.docker.com/linux/fedora/gpg

[docker-ce-stable-debuginfo]
name=Docker CE Stable - Debuginfo $basearch
baseurl=https://download.docker.com/linux/fedora/$releasever/debug-$basearch/stable
enabled=0
gpgcheck=1
gpgkey=https://download.docker.com/linux/fedora/gpg

dnf config-manager addrepo --from-repofile=./test.repo --overwrite
Error in added repository configuration file. Cannot set repository option "#1=
": Option "#1" not found

And this works;

[docker-ce-stable]
name=Docker CE Stable - $basearch
baseurl=https://download.docker.com/linux/fedora/$releasever/$basearch/stable
enabled=1
gpgcheck=1
gpgkey=https://download.docker.com/linux/fedora/gpg
[docker-ce-stable-debuginfo]
name=Docker CE Stable - Debuginfo $basearch
baseurl=https://download.docker.com/linux/fedora/$releasever/debug-$basearch/stable
enabled=0
gpgcheck=1
gpgkey=https://download.docker.com/linux/fedora/gpg

[thaJeztah]

And it looks like it's a confirmed bug, and .... 😂 😂 the reproducer is.. our .repo file;

• config-manager addrepo fails to add docker repository config rpm-software-management/dnf5#1603
• Pull request (not merged yet); config-manager: Fix addrepo from-repofile with empty/comment lines rpm-software-management/dnf5#1671

So; confirmed at least that the issue is because of the empty lines, but not sure yet when the fix will ship, so possibly we have to fix it up;

• replace the repo-files with empty lines removed?
• download to a temporary file and strip empty lines?
• manually add the repositories (but this requires all options to be set separately)

[thaJeztah]

On a fun sidetone; I discovered that dnf5 has support for releasever_major - wondering if we could use that for the RHEL 8/9 issue that we ran into with minor versions (which we tried to solve with redirects); not sure if it's also supported by dnf4 though;

• Add support for $releasever_major, $releasever_minor, and limited shell expansion rpm-software-management/dnf5#710

[jrohel]

@thaJeztah

I discovered that dnf5 has support for releasever_major
not sire if it's also supported by dnf4 though

Yes, it is supported in dnf4 since version 4.18.0 PR rpm-software-management/dnf#1989

wondering if we could use that for the RHEL 8/9

No. If I know correctly, in RHEL 8 there is an older version of dnf4 4.7.0 and in RHEL 9 there is an older version of dnf4 4.14.0.

Note: It can be used in RHEL 10, there is dnf4 4.20.0.

[jrohel]

@thaJeztah

However, that doesn't support glob / wildcards, so we may need to do a dnf repo list to get the IDs of the repositories to disable;

Globs are supported in the repository id. And it documentation for config-manager https://dnf5.readthedocs.io/en/latest/dnf5_plugins/config-manager.8.html is written Reposiory ID may contain globs.

See the examples at the end of the documentation. I purposely wrote the examples with globs in the id repository, hoping it will help someone.

[thaJeztah]

Yes, it is supported in dnf4 since version 4.18.0 PR rpm-software-management/dnf#1989

Ah, thanks! I'm not a user of a rpm-based distro myself, so sometimes need to do a bit more digging for the right place to search

No. If I know correctly, in RHEL 8 there is an older version of dnf4 4.7.0 and in RHEL 9 there is an older version of dnf4 4.14.0.

Note: It can be used in RHEL 10, there is dnf4 4.20.0.

Yes, that's usually the fun challenge; we're packaging for various distros and distro-versions, so where possible, we try to use options that are common between versions. Scripts we use in packaging don't always have to be "polished", we can do some detection there, but for our documentation (and installation script), it m more important, because, well, users don't read documentation (so may be copying the wrong commands), and we also know many users try to install packages on various derivative distros (most of which we don't actively "support", and at least don't test)

Globs are supported in the repository id. And it documentation for config-manager https://dnf5.readthedocs.io/en/latest/dnf5_plugins/config-manager.8.html is written Reposiory ID may contain globs.

See the examples at the end of the documentation. I purposely wrote the examples with globs in the id repository, hoping it will help someone.

Oh, perfect! Thank you for that; I didn't spot that part, that's useful.

Interesting; looks like setopt is taking a slightly different approach, and doesn't update the original .repo file (/etc/yum.repos.d/docker-ce.repo) but instead uses override files (/usr/share/dnf5/repos.override.d or /etc/dnf/repos.override.d).

[jrohel]

The dnf5 config-manager bug affects empty lines and comment lines (starting with the # character).

[jrohel]

tr -s '\n' only removes empty lines. Comment lines must also be removed. May be replace tr -s '\n' with grep '^[^#]'.

[thaJeztah]

Thanks! Yes, I was considering comment-lines as well, but our .repo files don't have them, so I ultimately only accounted for the empty lines.

It should (hopefully 🤞) just be temporary, and we can revert the commit once fixed.

FWIW: I lifted those commits to a separate PR (mostly for visibility); and we just merged them; #1059

[thaJeztah]

@vvoland this one should be ready for review; I pushed a commit to install the binutils-gold, but mostly as a precaution; happy to drop it as well (at least made it a separate commit).

[p5]

Apologies, if I saw this thread I'd have helped out.
Thank you for pushing this through!

[thaJeztah]

You're welcome! I had to carry the PR because Jenkins is configured to not allow external contributors to make changes in the Jenkinsfile itself, so it wouldn't test it in CI.

The dnf5 change made me go a bit down the rabbit-hole though 😂

[p5]

Is there something I should have ran from my local machine before submitting the PR in the first place? Some way that I could have identified the dnf5 issue before CI?

The build was fine, but I wasn't aware of any validation scripts

Edit:
Ah, looking at the repo, there's a ./verify script with instructions on how to run this inside a Docker container. Will be sure to run this next time