Fresh 20.04 to 22.04 upgrade fails

[jacobgkau]

Steps to reproduce:

• Perform a fresh install of the last Pop!_OS 20.04 Intel iso (number 35)
• sudo apt update && sudo apt full-upgrade
• reboot
• pop-upgrade release upgrade
• reboot

The system has the upgrade complete/rebooting message much quicker than expected, and the system reboots back into version 20.04, but with all sources pointing to Jammy.

Full offline upgrader log: upgrade.log

The following output is near the top of the log file:

Output:

The following packages will be marked as automatically installed:
  libgcrypt20 lsb-release gnupg apt libc6 libstdc++6 dpkg ca-certificates
  libdbus-1-3 libexpat1 libglib2.0-0 libuuid1 dbus gir1.2-glib-2.0 libcrypt1
  libsystemd0 libgcc-s1 kmod lsb-base libncursesw6 libtinfo6 python3
  python3-yaml libudev1 init-system-helpers zlib1g util-linux base-files
  pop-upgrade linux-system76 sessioninstaller coreutils dmsetup dosfstools
  e2fsprogs locales mount libcryptsetup12 libgnutls30 python3-gi libsqlite3-0
  shared-mime-info liblzma5 adduser bash systemd-sysv libpam-modules
  libpam-runtime procps ucf debconf libaudit1 libpam0g libselinux1
  libpop-upgrade-gtk libxml2 libcom-err2 libp11-kit0 libgirepository-1.0-1
  systemd kernelstub libelf1 libcap2 libseccomp2 linux-generic bzip2 libslang2
  python3-dbus libzstd1 libfribidi0 xdg-user-dirs openssl libreadline8 udev
  libnss-systemd libpam-systemd liblz4-1 libacl1 libbz2-1.0 libblkid1
  libfdisk1 libapparmor1 libgpg-error0 libkmod2 libmount1 systemd-timesyncd
  libip4tc2 networkd-dispatcher sudo iproute2 libdevmapper1.02.1 file
  libpython3-stdlib passwd liblocale-gettext-perl libext2fs2 libpcre3 cron
  debianutils rsyslog mime-support gpgv libapt-pkg6.0 ubuntu-keyring apt-utils
  sensible-utils libdb5.3 libattr1 mawk libcap-ng0 xz-utils isc-dhcp-client
  iputils-ping libpopt0 netbase libidn2-0 libbsd0 libsmartcols1 bsdutils less
  eject fdisk libnewt0.52 logrotate libncurses6 libcap2-bin tzdata netplan.io
  python3-netifaces software-properties-common libmnl0 console-setup
  console-setup-linux xkb-data keyboard-configuration kbd libgmp10 wamerican
  perl-base whiptail debconf-i18n libtext-iconv-perl libtext-wrapi18n-perl
  libtext-charwidth-perl libyaml-0-2 distro-info-data libsodium23 tar
  python3-pkg-resources libss2 logsave poppler-data libmagic1 sed
  libunistring2 libpcre2-8-0 hunspell-ar hunspell-de-at-frami
  hunspell-de-ch-frami hunspell-de-de-frami hunspell-en-au hunspell-en-ca
  hunspell-en-gb hunspell-en-us hunspell-en-za hunspell-es hunspell-it
  hunspell-pt-br hunspell-pt-pt hunspell-ru libxtables12 libatm1
  libisc-export1105 isc-dhcp-common readline-common libargon2-1
  libaudit-common libc-bin python3.8 libpam-cap libestr0 libfastjson4
  libglib2.0-data libtasn1-6 libmagic-mgc libpython3.8-stdlib libnetplan0
  libpam-modules-bin libprocps8 libsemanage-common netcat-openbsd login
  python3-six xxd vim-common python3-minimal python3-cffi-backend python3-nacl
  python3-pymacaroons ubuntu-advantage-tools vim-tiny wbrazilian wbritish
  wfrench witalian wngerman wogerman wportuguese wspanish wswiss gcc-10-base
  libffi7 libssl1.1 libjson-c4 libicu66 libhogweed5 libnettle7
  libdns-export1109 libmpdec2 libpython3.8-minimal libsemanage1 libsepol1
  python3.8-minimal
Do you want to continue? [y/N] Y
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
fakeroot gcc-12-base gnome-keyring libc-bin libc-dev-bin libc6-dev
libegl-mesa0 libfakeroot libffi8 libgbm1 libgcrypt20 libgl1-mesa-dri
libglapi-mesa libglib2.0-bin libglib2.0-dev libglib2.0-dev-bin libglx-mesa0
libgmp10 libgnutls30 libhogweed6 libllvm14 libmount-dev libnettle8
libnsl-dev libnsl2 libnss-nis libnss-nisplus libselinux1 libselinux1-dev
libsepol-dev libsepol2 libssl3 libstdc++6 libtirpc-common libtirpc-dev
libtirpc3 libwayland-client0 libx11-6 libx11-xcb1 locales
network-manager-pptp network-manager-pptp-gnome rpcsvc-proto zlib1g-dev
Suggested packages:
glibc-doc rng-tools libgirepository1.0-dev libglib2.0-doc libxml2-utils
gnutls-bin libteam-utils
Recommended packages:
libc-devtools
The following packages will be REMOVED:
libsepol1-dev
E: Internal error, InstallPackages was called with broken packages!

The E: Internal error, InstallPackages was called with broken packages! error then repeats after later steps in the log file.

On my test VM, after I booted back into 20.04, the Jammy sources were valid. Performing a regular apt upgrade (as many users are unaware of full-upgrade), I noticed that I was asked whether or not to upgrade various files in /etc, including our /etc/os-release and /etc/lsb-release files, with the default choice being "no":

This explains why some users have appeared on Mattermost with mostly-upgraded systems but with those files still reporting that they're on the prior release.

After finishing the apt upgrade, there were still a number of packages that needed to be upgraded using apt full-upgrade. Doing that and then rebooting, I dropped into a busybox prompt. Not sure of the actual end state of the system, but we should probably focus on the earlier pop-upgrade failure first (as everything after that is dependent on how the user tries to proceed.)

[jacobgkau]

I attempted to replicate this again today, running upgrade.sh manually. The script did not appear to fail. After it finished, /etc/os-release still said 20.04. After rebooting, I am in 22.04, and all packages are up-to-date, but pop-upgrade still offers to "upgrade" to 22.04 since /etc/os-release is wrong.

This feels like something is wrong with the pop-default-settings packaging causing those files not to be updated. However, a possible workaround might simply be force reinstalling the /etc/os-release/etc/lsb-release files as described here at the end of the upgrade.sh script.

[jacobgkau]

pop-os/default-settings#163 touched the conffiles configuration of pop-default-settings. I'm beginning to suspect this issue was caused by that PR.

[mmstick]

Perhaps it'll be fixed by renaming it back to conffiles from pop-default-settings.conffiles.

[jacobgkau]

I see in the upgrade.sh output where pop-upgrade is leaving the old pop-default-settings conf files by default:

Perhaps it'll be fixed by renaming it back to conffiles from pop-default-settings.conffiles.

I think the breaking change in pop-os/default-settings@7fcad35 might have been that we used to override dh_installdeb and copy our conffiles file over manually after the automatic version has been run. Without doing that, dh_installdeb is going to include any file under /etc by default, and we leave it there (by not copying over it.)

So maybe the naming doesn't matter so much as that override. (The new naming is correct if we want to have it use that and then add its auto-generated ones; doesn't matter so much if we use the "correct" name or not if we're overriding?)

[jacobgkau]

I'm working on this in a branch: https://github.com/pop-os/default-settings/tree/conffiles_jammy

[jacobgkau]

/etc/os-release is displaying correctly with the fix in pop-os/default-settings#165.