Skip to content

Fedora

Jump to bottom Edit New page
Henryk Paluch edited this page Aug 11, 2024 · 13 revisions

Fedora Notes

Fedora 39 Server notes

WARNING! In my case Fedora installed useless grub manager called sdubby. It is absolutely useless in BIOS mode - because it is hardcoded to to support EFI/systemd-boot only. So I have to replace it with standard grubby using:

dnf install --allowerasing grubby

Hoping that it is bug (not ill intention):

https://bugzilla.redhat.com/show_bug.cgi?id=2243872#c72

I thought with this bug resolved, grubby instead of sdubby will be installed by default. But it's not true for all cases. Currently a clean install of Fedora-Server-dvd-x86_64-40-20240314.n.0 will install sdubby but Fedora-Server-dvd-x86_64-39-1.5 has grubby installed. And I tried install Fedora 39, 40 and 41 on beaker machines, all cases have sdubby installed.

Added comment there: https://bugzilla.redhat.com/show_bug.cgi?id=2243872#c75

TIP: There are now my optimization scripts available on: https://github.com/hpaluch/fedora-files

Then I could finally remove bloat arguments using commands:

dnf install --allowerasing grubby
grubby --update-kernel=ALL --remove-args=resume=UUID=YOUR_UUID_HERE

And add few optimizations:

# disable nagging disk resume (what's the use on Fedora *Server*???)
grubby --update-kernel=ALL --args=noresume
# disable mitigations to get back normal performance on Opteron G2
grubby --update-kernel=ALL --args=mitigations=off
# stop stupid renaming of network cards
grubby --update-kernel=ALL --args=net.ifnames=0
# if you don't need GUI:
grubby --update-kernel=ALL --args=nomodeset

WARNING! When using net.ifnames=0 you should also rename files and their content in /etc/NetworkManager/system-connections (!). Only then you may reboot...

Set proper default editor:

dnf install --allowerasing vim-default-editor

Security issue:

Translating LFS docbook

Trying to build HTML and pdf version of these docbooks:

Have to install these RPMs:

dnf install git-lfs java-17-openjdk-headless \
    make libxslt tidy docbook5-schemas \
    docbook5-style-xsl xml-common \
    docbook-dtds docbook-style-xsl docbook5-style-xsl

Controversial:

  • for PDF generation you can install fop just with dnf install fop
  • but it will install lot of mad dependencise (including pipewire for sound support!)
  • I rather downloaded tar.gz from https://xmlgraphics.apache.org/fop/download.html and unpacked it into /opt

Start is easy:

mkdir -p ~/projects
cd ~/projects
git clone https://git.linuxfromscratch.org/lfs.git lfs-git
cd lfs-git/
make book
# TODO: warning: 
# stylesheets/lfs-xsl/lfs-l10n.xml:2: warning: failed to load external entity "http://cdn.docbook.org/release/xsl-nons/current/common/l10n.dtd"
# l:i18n SYSTEM "http://cdn.docbook.org/release/xsl-nons/current/common/l10n.dtd"
# above is include in: docbook-style-xsl

# PDF version of LFS Book:
/opt/fop-2.9/fop/fop --help
PATH=/opt/fop-2.9/fop:$PATH make pdf
# look on: ~/public_html/lfs-book/LFS-BOOK.pdf

# Now trying BLFS:
cd ~/projects/
git clone https://git.linuxfromscratch.org/blfs.git blfs-git
cd blfs-git/
make html
# TODO: missing pdf target

Fedora 39

Installed Fedora 39 in default GUI as VM under Proxmox in UEFI mode (I use UEFI because one bare-metal machine uses UEFI).

Speeding up Fedora 39 Workstation Installation under Proxmox VE + SPICE console:

In my case Fedora 39 Workstation installation took forever (package installation was acceptable but following system configuration took more than 1 hour).

Why? Because in case of SPICE under Proxmox VE it will run as gnome-shell Wayland in software mode. Thus both gnome-shell and overloaded kernel DRI eat basically all CPU cycles causing saturated kernel (system) and user CPU.

You can verify this by simply pressing Alt-F2 while installing Fedora Workstation - dialog "Run Command" Should appear - fill-in gnome-terminal and press ENTER. Then run top in terminal.

In my case there was horrible scene - where gnome-shell + overloaded DRI kernel processes saturated VM to around 350% (load average around 3.5) - so for example dracut will take forever to finish creation of initrd. Poor dracut was unable to grab more than 3% of CPU (even when I tried to renice gnome-shell to provide space)....

Workaround: press directly Ctrl-Alt-F3 (or use keyboard menu in SPICE console if needed) to witch back from GUI to normal console. Login prompt should appear. Enter username liveuser (no password needed). Now run top - and watch CPU usage - dracut should be finally able to finish in reasonable time...

Once you see that there is no dracut and no pigz it should be safe to use Ctrl-Alt-F2 to switch back to gnome-shell GUI and finish installation by pressing Reboot button...

WARNING! Fedora 39 was extremely slow (both GUI and CLI and both as VM and bare metal). For example just dnf search ANYTHING caused Python process to eat all CPU for several minutes (!). I think that these two things helped to restore normal performance.

  1. Disable CPU mitigations using: 
    grubby --update-kernel=ALL --args=mitigations=off
  2. Remove ZRAM nonsense (you should have regular swap partition): 
    sudo dnf remove zram-generator zram-generator-defaults
  3. Finally reboot using sudo reboot
  4. In my case it helped to restore acceptable performance of Fedora...

Optional: GRUB adjustemnt - I want to see boot messages:

sudo grubby --update-kernel=ALL --remove-args=rhgb
sudo grubby --update-kernel=ALL --remove-args=quiet

Important - disable OOMD - otherwise it will kill your Firefox or LibreOffice with unsaved work:

sudo systemctl mask --now systemd-oomd

After reboot it was terribly slow

  • packagekitd eating all CPU
  • gnome-software-center eating all CPU
  • /usr/libexec/sssd/sssd_kcm eating CPU - not all, but still considerable bloat

Tried:

# pin elfutils
sudo dnf install elfutils
sudo dnf remove abrt

dnf remove PackageKit gnome-software
# who needs logo on bootup that covers kernel messags?
dnf remove plymouth\*

In case of sssd it is risky, you have to:

dnf remove sssd-\*
reboot
# other stuff
sudo systemctl mask dnf-makecache.timer unbound-anchor.timer plocate-updatedb.timer raid-check.timer

On other side had to enable and start SSH server

systemctl enable --now sshd

Other bloat:

dnf remove bluez-obexd
# Swapping RAM to RAM...
dnf remove zram-generator zram-generator-defaults
# if you don't use wifi
dnf remove wpa_supplicant
dnf remove ModemManager fwupd-plugin-modem-manager
systemctl mask --now fwupd

Risky:

dnf remove sssd\*
# but rather reininstall:
# (binutils contains "strip" commands that removes debug and other data)
dnf install dracut-squash binutils
  • WARNING - if you remove sssd you have to also comment-out all references to it under /etc/pam.d/* !

Enabling ZSTD for dracut:

  • install ztsd: 
    dnf install zstd
  • create custom file /etc/dracut.conf.d/custom.conf with content: 
    compress="zstd"
  • WARNING! Even if I regenerate initramfs with dracut -fv the command below: 
    $ file /boot/initramfs-`uname -r`.img

/boot/initramfs-6.5.6-300.fc39.x86_64.img: ASCII cpio archive (SVR4 with no CRC)
  • still shows CPIO archive (no compression)...

GNOME tracks your location

Private:

  • even if you disable Location services on GNOME "first run wizard" it will happily ignore you and still call location services (DNS query location.services.mozilla.com)
  • see GNOME how to disable geoclue service.

But in case of Fedora - even after reboot GNOME is still contacting fedoraproject.org several times.

Using sway

The only way to make F39 speed acceptable was to use rather Sway (simplest Wayland "compositor" = "X-Server" and Window Manager in one process). And there is nice swaybar in /usr/share/sway/config.d/90-bar.conf

# Read `man 5 sway-bar` for more information about this section.
#
# Requires: waybar

bar {
    position top
    swaybar_command waybar
}

SELinux - recovery

It may easily happen, for example when cloning Fedora with plain cpio, there there will be missing SELinux contexts required for proper SELinux operation. Here is trick how to relabel otherwise unbootable filesystem (enforce mode will quickly panic because missing critical permissions):

  1. Boot kernel with selinux=0 which completely disables SELinux. Now run fixfiles ‐F onboot (it will create file /.autorelabel telling boot scripts to relabel filesystem) and reboot
  2. On reboot, boot kernel with enforcing=0 parameter - this will enable SELinux in permissive mode - it is required to provide API to relabel filesystem (but we still may not use enforcing mode, because of missing permissions). The system may appear to hang, but it actually relabels whole filesystem - at the end it will automatically reboot.
  3. Now just boot Fedora/SELinux normally - it should work again in fully enforcing mode.

Tracing DNS

There are 2 options with systemd-resolverd:

Is Fedora rolling?

Although Fedora is NOT rolling distribution I was surprised by so many upgrades

  • last updates installed on 2024-04-07
  • tis upgrade run on: 2024-04-21
  • see: 
    $ fgrep `date '+%Y-%m-%d'` /var/log/dnf.log | fgrep Upgraded | wc -l

777
  • 777 packages are upgraded!
  • when looked into changelog of wget 
    $ rpm -q --changelog  wget | head

* Tue Mar 05 2024 Michal Ruprich <mruprich@redhat.com> - 1.21.4-1
- New version 1.21.4

* Sat Jul 22 2023 Fedora Release Engineering <releng@fedoraproject.org> - 1.21.3-7
- Rebuilt for https://fedoraproject.org/wiki/Fedora_39_Mass_Rebuild
  • that is interesting (but not primary reason for upgrade)
  • another interesting URL: https://kojipkgs.fedoraproject.org/mass-rebuild/
  • found it there: https://koji.fedoraproject.org/koji/buildinfo?buildID=2415437

Sometimes I'm right confused, because:

  • package qemu-audio-spice
  • upgrade log: 
    2024-04-21T15:23:56+0200 SUBDEBUG Upgrade: qemu-audio-spice-2:8.1.3-5.fc39.x86_64
2024-04-21T15:31:20+0200 SUBDEBUG Upgraded: qemu-audio-spice-2:8.1.0-1.fc39.x86_64
  • changelog: 
    * Mon Apr 08 2024 Cole Robinson <crobinso@redhat.com> - 8.1.3-5
- Fix some qemu-user-static usage (bz 2256916)

* Wed Feb 21 2024 Richard W.M. Jones <rjones@redhat.com> - 2:8.1.3-4
- Fix user-emulation of FIFREEZE and FITHAW ioctls

* Tue Feb 13 2024 Mauro Matteo Cascella <mcascell@redhat.com> - 2:8.1.3-3
- Stack buffer overflow in virtio_net_flush_tx (CVE-2023-6693) (rhbz#2256436)

* Tue Jan 23 2024 Paolo Bonzini <pbonzini@redhat.com> - 8.1.3-2
- Workaround upstream issue 2092

* Tue Nov 28 2023 Cole Robinson <crobinso@redhat.com> - 8.1.3-1
- Rebase to qemu 8.1.3

* Tue Sep 26 2023 Cole Robinson <crobinso@redhat.com> - 8.1.1-1
- Rebase to qemu 8.1.1

* Thu Aug 24 2023 Cole Robinson <crobinso@redhat.com> - 8.1.0-2
- Make qemu-docs noarch

* Wed Aug 23 2023 Cole Robinson <crobinso@redhat.com> - 8.1.0-1
- Rebase to qemu 8.1.0 GA
  • so sudden jump from 8.1.0-1 released in Aug 23, 2023 to 8.1.3-5 released in Apr 08, 2024

RPM source fetching and development

First install required tools (for building all RPM packages):

dnf install rpmdevtools dnf-utils rpm-build

As non-privileged user setup RPM directories and macros with:

# run as non-root user
rpmdev-setuptree

Now you can download source package (for example source of systemd) with:

# run as non-root user
dnf download systemd --source

Now unpack src.rpm with:

rpm -ivh systemd-*.src.rpm

Install build dependencies for systemd package:

sudo yum-builddep systemd

To just unpack sources:

rpmbuild -bp ~/rpmbuild/SPECS/systemd.spec

etting SOURCE_DATE_EPOCH=1709251200
error: Failed build dependencies:
	bpftool is needed by systemd-254.10-1.fc39.x86_64
	xen-devel is needed by systemd-254.10-1.fc39.x86_64

Ooops, trying to fix it:

sudo dnf install bpftool xen-devel
rpmbuild -bp ~/rpmbuild/SPECS/systemd.spec
# Now it should proceed without problems

To build both src and binary RPM, just use:

rpmbuild -ba ~/rpmbuild/SPECS/systemd.spec

Phantom UARTS in /dev

There are over 30 phantom serial ports under /dev which also slows down systemd:

Appending boot parameter 8250.nr_uarts=2 will do the trick!

sudo grubby --update-kernel=ALL --args=8250.nr_uarts=2

After reboot you should see only two devices starting with /dev/ttyS* :-)

BPF quick example

BPF can be used to trace both kernel and user space programs (if they provides tracepoints).

Quick-example - execsnoop:

  • install 
    sudo dnf install bcc-tools
  • now run (ignore warnings): 
    sudo /usr/share/bcc/tools/execsnoop -xtU
  • here is what I see when I log via SSH as fadmin user:
7.251   0     sshd             20692   790       0 /usr/sbin/sshd -D -R
7.418   0     unix_chkpwd      20694   20692     0  fadmin 
7.525   1000  bash             20696   20695     0 /bin/bash
7.529   1000  id               20697   20696     0 /usr/bin/id -un
7.533   1000  hostnamectl      20699   20698     0 /usr/bin/hostnamectl --transient
7.688   0     systemd-hostnam  20700   1         0 /usr/lib/systemd/systemd-hostnamed
7.735   1000  grepconf.sh      20701   20696     0 /usr/libexec/grepconf.sh -c
7.738   1000  grep             20702   20701     0 /usr/bin/grep -qsi ^COLOR.*none /etc/GREP_COLORS
7.744   1000  dircolors        20704   20703     0 /usr/bin/dircolors --sh /etc/DIR_COLORS
7.747   1000  grep             20705   20696     0 /usr/bin/grep -qi ^COLOR.*none /etc/DIR_COLORS
7.751   1000  grepconf.sh      20706   20696     0 /usr/libexec/grepconf.sh -c
7.754   1000  grep             20707   20706     0 /usr/bin/grep -qsi ^COLOR.*none /etc/GREP_COLORS
7.757   1000  grepconf.sh      20708   20696     0 /usr/libexec/grepconf.sh -c
7.760   1000  grep             20709   20708     0 /usr/bin/grep -qsi ^COLOR.*none /etc/GREP_COLORS
7.766   1000  cat              20711   20710     0 /usr/bin/cat /dev/null /etc/debuginfod/elfutils.urls
7.766   1000  tr               20712   20710     0 /usr/bin/tr \n  
7.771   1000  locale           20714   20713     0 /usr/bin/locale
7.774   1000  sed              20715   20696     0 /usr/bin/sed -r -e s/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d /etc/locale.conf
7.781   1000  readlink         20717   20716     0 /usr/bin/readlink /proc/20696/exe
7.784   1000  basename         20716   20696     0 /usr/bin/basename /usr/bin/bash

So we can see that we spend precious time by parsing colors and other stuff... There is also similar example for monitoring open files:

sudo /usr/share/bcc/tools/opensnoop -TUF

Combining output of both commands you may see that these color checks come from /etc/profile.d/color*.sh

Or DNS request latencies (latencies of calls like gethostbyname and similar)

sudo /usr/share/bcc/tools/gethostlatency 
TIME      PID     COMM                  LATms HOST
19:33:46  21122   lynx                   4.92 www.google.com
19:33:46  21123   lynx                   2.85 www.google.com

Or interesting example - trace TCP Connects + related DNS:

sudo dnf install python3-cachetools python3-dnslib
# and then
sudo /usr/share/bcc/tools/tcpconnect -tdU

/virtual/main.c:198:35: error: no member named 'iov' in 'struct iov_iter'
    if (buflen > msghdr->msg_iter.iov->iov_len)

Hmmm, not good example...

How it is different from strace or ltrace: - BPF is system wide, while strace normally attaches only specific process.

Usign chrony in "one shot mode"

Normally there is running chronyd all the time (another security issue). I strongly prefer to just once synchronize clock and exit. Here is How I did it:

  • disable service with: systemctl mask --now chronyd
  • create /etc/rc.d/rc.local with contents: 
    #!/bin/bash
set -euo pipefail
# synchronize clock once and exit
chronyd -q
exit 0
  • make it executable (not sure if needed): chmod +x /etc/rc.d/rc.local
  • you can try it right now, if it works as expceted: /etc/rc.d/rc.local

Nouveau and firmware

In case of my NVidia GT218 card there is no suitable firmware included with Fedora 40. When X11 or Wayland attempts to use 3D acceleration there will be following error reported by kernel:

kernel: nouveau 0000:06:00.0: Direct firmware load for nouveau/nva8_fuc084 failed with error -2
kernel: nouveau 0000:06:00.0: Direct firmware load for nouveau/nva8_fuc084d failed with error -2

Unfortunately package nvidia-gpu-firmware-20240410-1.fc40.noarch does not provide it. I have to grab it from Gentoo at least these files:

$ ls -l /lib/firmware/nouveau/nva8_fuc084 

lrwxrwxrwx. 1 root root 8 May  8 17:09 /lib/firmware/nouveau/nva8_fuc084 -> nva3_bsp

If you have somewhere Gentoo installed you can follow guide on: https://github.com/hpaluch/gentoo-files/tree/master/desktop-profile Basically it is contained in package sys-firmware/nvidia-firmware-340.32-r1:0 - please note that it is important that version is 340 or lower (because NVidia compassionately dropped support for GT218 from more recent driver package).

To play video (at least free H264) you may install:

dnf install mpv openh264 mozilla-openh264 vdpauinfo mesa-vdpau-drivers inxi \
    libva-utils libva-nvidia-driver

To see information on Video acceleration try vdpauinfo or vainfo (these are 2 standards for video acceleration). To see general details on GPU driver try inxi -G --display or similar command. To play video try mpv VIDEO.mp4

In my case vainfo shows disappointing results - only MPEG2 acceleration. While vdpauinfo is much better. But I have issues to find suitable program. Because using mpv --vo=vdpau VIDEO.mp4 will yield error that libvdpau_nvidia.so not found...

Also tried mpv --vo=gpu -hwdec=vdpau VIDEO.mp4 but never got satisfactory results...

I'm unable to use VAAPI, because it supports MPEG2 only while vdpauinfo lists also H264 and other codecs...

LVM: new disks ignored

When you insert new disk with LVM (after Fedora install) its PVs/VGs will be not visible:

# pvs
  PV         VG     Fmt  Attr PSize    PFree   
  /dev/sdb3  vgs350 lvm2 a--  <296.09g <237.29g

# pvs -A

  PV         VG      Fmt  Attr PSize    PFree   
  /dev/sda3  pveiron lvm2 a--    <2.00t 1005.00g
  /dev/sdb3  vgs350  lvm2 a--  <296.09g <237.29g

Solution is described here: https://github.com/coreos/fedora-coreos-tracker/issues/1517 and here: https://serverfault.com/questions/1109273/why-is-pvscan-skipping-my-devices

$ lvmdevices 

  Device /dev/sdb3 IDTYPE=sys_wwid IDNAME=t10.ATA_ST3320620AS_5QF3SLHZ DEVNAME=/dev/sdb3 PVID=sVTrqaiJNJmu3UfcBxwazy5BX6fAyWqC PART=3

$ lvmdevices --adddev /dev/sda3
$ pvs

  PV         VG      Fmt  Attr PSize    PFree   
  /dev/sda3  pveiron lvm2 a--    <2.00t 1005.00g
  /dev/sdb3  vgs350  lvm2 a--  <296.09g <237.29g
$ vgs

  VG      #PV #LV #SN Attr   VSize    VFree   
  pveiron   1  55   0 wz--n-   <2.00t 1005.00g
  vgs350    1   2   0 wz--n- <296.09g <237.29g

$ vgchange -a y pveiron

Now my PV /dev/sda3 with VG pveiron is visible!

Copyright © Henryk Paluch. All rights reserved.

Creative Commons License This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License

Add a custom sidebar
Clone this wiki locally