supervise.sh running in a loop

[Farfaday]

Hi,
Today I found out that the two scripts /opt/bin/supervise.sh /opt/bin/bootstrap and /opt/bin/supervise.sh /opt/bin/setup were constantly running in a loop on all our worker nodes. So constantly we have apt update / upgrade / etc. running.

Part of /var/log/syslog, this is flowing in a loop:

systemd[1]: Finished Update APT News.
systemd[1]: esm-cache.service: Succeeded.
systemd[1]: Finished Update the local ESM caches.
supervise.sh[3482641]: Fetched 222 kB in 1s (206 kB/s)
supervise.sh[3482641]: Reading package lists...
supervise.sh[3481021]: + apt-get install -y --allow-downgrades 'containerd.io=1.6*'
supervise.sh[3483659]: Reading package lists...
supervise.sh[3483659]: Building dependency tree...
supervise.sh[3483659]: Reading state information...
supervise.sh[3483659]: The following held packages will be changed:
supervise.sh[3483659]:   containerd.io
supervise.sh[3483659]: The following packages will be upgraded:
supervise.sh[3483659]:   containerd.io
supervise.sh[3483659]: 1 upgraded, 0 newly installed, 0 to remove and 94 not upgraded.
supervise.sh[3483659]: E: Held packages were changed and -y was used without --allow-change-held-packages.
supervise.sh[3277]: + sleep 1
supervise.sh[3277]: + /opt/bin/setup
supervise.sh[3483816]: + systemctl is-active ufw
supervise.sh[3483817]: inactive
supervise.sh[3483816]: + systemctl mask ufw
systemd[1]: Reloading.
supervise.sh[3483816]: + systemctl restart systemd-modules-load.service
systemd[1]: systemd-modules-load.service: Succeeded.
systemd[1]: Stopped Load Kernel Modules.
systemd[1]: Stopping Load Kernel Modules...
systemd[1]: Starting Load Kernel Modules...
systemd[1]: Finished Load Kernel Modules.
supervise.sh[3483816]: + sysctl --system
supervise.sh[3483850]: * Applying /etc/sysctl.d/10-console-messages.conf ...
supervise.sh[3483850]: kernel.printk = 4 4 1 7
supervise.sh[3483850]: * Applying /etc/sysctl.d/10-ipv6-privacy.conf ...
supervise.sh[3483850]: net.ipv6.conf.all.use_tempaddr = 2
supervise.sh[3483850]: net.ipv6.conf.default.use_tempaddr = 2
supervise.sh[3483850]: * Applying /etc/sysctl.d/10-kernel-hardening.conf ...
supervise.sh[3483850]: kernel.kptr_restrict = 1
supervise.sh[3483850]: * Applying /etc/sysctl.d/10-link-restrictions.conf ...
supervise.sh[3483850]: fs.protected_hardlinks = 1
supervise.sh[3483850]: fs.protected_symlinks = 1
supervise.sh[3483850]: * Applying /etc/sysctl.d/10-magic-sysrq.conf ...
supervise.sh[3483850]: kernel.sysrq = 176
supervise.sh[3483850]: * Applying /etc/sysctl.d/10-network-security.conf ...
supervise.sh[3483850]: net.ipv4.conf.default.rp_filter = 2
supervise.sh[3483850]: net.ipv4.conf.all.rp_filter = 2
supervise.sh[3483850]: * Applying /etc/sysctl.d/10-ptrace.conf ...
supervise.sh[3483850]: kernel.yama.ptrace_scope = 1
supervise.sh[3483850]: * Applying /etc/sysctl.d/10-zeropage.conf ...
supervise.sh[3483850]: vm.mmap_min_addr = 65536
supervise.sh[3483850]: * Applying /usr/lib/sysctl.d/50-default.conf ...
supervise.sh[3483850]: net.ipv4.conf.default.promote_secondaries = 1
supervise.sh[3483850]: sysctl: setting key "net.ipv4.conf.all.promote_secondaries": Invalid argument
supervise.sh[3483850]: net.ipv4.ping_group_range = 0 2147483647
supervise.sh[3483850]: net.core.default_qdisc = fq_codel
supervise.sh[3483850]: fs.protected_regular = 1
supervise.sh[3483850]: fs.protected_fifos = 1
supervise.sh[3483850]: * Applying /usr/lib/sysctl.d/50-pid-max.conf ...
supervise.sh[3483850]: kernel.pid_max = 4194304
supervise.sh[3483850]: * Applying /etc/sysctl.d/99-cloudimg-ipv6.conf ...
supervise.sh[3483850]: net.ipv6.conf.all.use_tempaddr = 0
supervise.sh[3483850]: net.ipv6.conf.default.use_tempaddr = 0
supervise.sh[3483850]: * Applying /etc/sysctl.d/99-sysctl.conf ...
supervise.sh[3483850]: * Applying /etc/sysctl.d/k8s.conf ...
supervise.sh[3483850]: kernel.panic_on_oops = 1
supervise.sh[3483850]: kernel.panic = 10
supervise.sh[3483850]: net.ipv4.ip_forward = 1
supervise.sh[3483850]: vm.overcommit_memory = 1
supervise.sh[3483850]: fs.inotify.max_user_watches = 1048576
supervise.sh[3483850]: fs.inotify.max_user_instances = 8192
supervise.sh[3483850]: * Applying /usr/lib/sysctl.d/protect-links.conf ...
supervise.sh[3483850]: fs.protected_fifos = 1
supervise.sh[3483850]: fs.protected_hardlinks = 1
supervise.sh[3483850]: fs.protected_regular = 2
supervise.sh[3483850]: fs.protected_symlinks = 1
supervise.sh[3483850]: * Applying /etc/sysctl.conf ...
supervise.sh[3483851]: ++ command -v hostnamectl
supervise.sh[3483816]: + '[' -x /usr/bin/hostnamectl ']'
supervise.sh[3483816]: + '[' -s /etc/machine-name ']'
supervise.sh[3483852]: ++ cat /etc/machine-name
supervise.sh[3483816]: + machine_name=ston-k1d-worker-965b66456-ljlkt
supervise.sh[3483816]: + hostnamectl set-hostname ston-k1d-worker-965b66456-ljlkt
supervise.sh[3483816]: + apt-get update
systemd[1]: Starting Update APT News...
systemd[1]: Starting Update the local ESM caches...
supervise.sh[3483865]: Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
supervise.sh[3483865]: Hit:2 http://az-1.clouds.archive.ubuntu.com/ubuntu focal InRelease
supervise.sh[3483865]: Hit:3 http://az-1.clouds.archive.ubuntu.com/ubuntu focal-updates InRelease
supervise.sh[3483865]: Hit:4 https://download.docker.com/linux/ubuntu focal InRelease
supervise.sh[3483865]: Get:5 http://az-1.clouds.archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
systemd[1]: apt-news.service: Succeeded.
systemd[1]: Finished Update APT News.
systemd[1]: esm-cache.service: Succeeded.
systemd[1]: Finished Update the local ESM caches.
supervise.sh[3483865]: Fetched 222 kB in 1s (227 kB/s)
supervise.sh[3483865]: Reading package lists...
supervise.sh[3483816]: + DEBIAN_FRONTEND=noninteractive
supervise.sh[3483816]: + apt-get -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold install -y curl ca-certificates ceph-common cifs-utils conntrack e2fsprogs ebtables ethtool glusterfs-client iptables jq kmod openssh-client nfs-common socat util-linux ipvsadm
supervise.sh[3484443]: Reading package lists...
supervise.sh[3484443]: Building dependency tree...
supervise.sh[3484443]: Reading state information...
supervise.sh[3484443]: conntrack is already the newest version (1:1.4.5-2).
supervise.sh[3484443]: ebtables is already the newest version (2.0.11-3build1).
supervise.sh[3484443]: ethtool is already the newest version (1:5.4-1).
supervise.sh[3484443]: ipvsadm is already the newest version (1:1.31-1).
supervise.sh[3484443]: socat is already the newest version (1.7.3.3-2).
supervise.sh[3484443]: glusterfs-client is already the newest version (7.2-2build1).
supervise.sh[3484443]: ca-certificates is already the newest version (20230311ubuntu0.20.04.1).
supervise.sh[3484443]: ceph-common is already the newest version (15.2.17-0ubuntu0.20.04.4).
supervise.sh[3484443]: cifs-utils is already the newest version (2:6.9-1ubuntu0.2).
supervise.sh[3484443]: curl is already the newest version (7.68.0-1ubuntu2.18).
supervise.sh[3484443]: e2fsprogs is already the newest version (1.45.5-2ubuntu1.1).
supervise.sh[3484443]: iptables is already the newest version (1.8.4-3ubuntu2.1).
supervise.sh[3484443]: kmod is already the newest version (27-1ubuntu2.1).
supervise.sh[3484443]: nfs-common is already the newest version (1:1.3.4-2.5ubuntu3.4).
supervise.sh[3484443]: openssh-client is already the newest version (1:8.2p1-4ubuntu0.7).
supervise.sh[3484443]: util-linux is already the newest version (2.34-0.1ubuntu9.4).
supervise.sh[3484443]: jq is already the newest version (1.6-1ubuntu0.20.04.1).
supervise.sh[3484443]: 0 upgraded, 0 newly installed, 0 to remove and 95 not upgraded.
supervise.sh[3483816]: + apt-get update
systemd[1]: Starting Update APT News...
systemd[1]: Starting Update the local ESM caches...
supervise.sh[3484446]: Hit:1 http://az-1.clouds.archive.ubuntu.com/ubuntu focal InRelease
supervise.sh[3484446]: Hit:2 http://az-1.clouds.archive.ubuntu.com/ubuntu focal-updates InRelease
supervise.sh[3484446]: Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
supervise.sh[3484446]: Get:4 http://az-1.clouds.archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
supervise.sh[3484446]: Hit:5 https://download.docker.com/linux/ubuntu focal InRelease
systemd[1]: apt-news.service: Succeeded.
systemd[1]: Finished Update APT News.
systemd[1]: esm-cache.service: Succeeded.
systemd[1]: Finished Update the local ESM caches.
supervise.sh[3484446]: Fetched 222 kB in 1s (213 kB/s)
supervise.sh[3484446]: Reading package lists...
supervise.sh[3483816]: + apt-get install -y apt-transport-https ca-certificates curl software-properties-common lsb-release
supervise.sh[3485253]: Reading package lists...
supervise.sh[3485253]: Building dependency tree...
supervise.sh[3485253]: Reading state information...
supervise.sh[3485253]: lsb-release is already the newest version (11.1.0ubuntu2).
supervise.sh[3485253]: ca-certificates is already the newest version (20230311ubuntu0.20.04.1).
supervise.sh[3485253]: curl is already the newest version (7.68.0-1ubuntu2.18).
supervise.sh[3485253]: software-properties-common is already the newest version (0.99.9.11).
supervise.sh[3485253]: apt-transport-https is already the newest version (2.0.9).
supervise.sh[3485253]: 0 upgraded, 0 newly installed, 0 to remove and 95 not upgraded.
supervise.sh[3485263]: + curl -fsSL https://download.docker.com/linux/ubuntu/gpg
supervise.sh[3485264]: + apt-key add -
supervise.sh[3485264]: Warning: apt-key output should not be parsed (stdout is not a terminal)
supervise.sh[3485264]: OK
supervise.sh[3485409]: ++ lsb_release -cs
supervise.sh[3483816]: + add-apt-repository 'deb https://download.docker.com/linux/ubuntu focal stable'
systemd[1]: Starting Update APT News...
systemd[1]: Starting Update the local ESM caches...

I guess this issue is very similar to #1049
Maybe the option --allow-change-held-packages should be added, as in https://github.com/kubermatic/machine-controller/pull/1073/files

We are using Ubuntu focal workers with machine-controler v1.54.4

Thanks!

[embik]

Hi @Farfaday, apologies for the late reply. Is this still an issue for you? Have you found any solution? Judging by the log output, is it possible that your image already had containerd installed in an older version?

[Farfaday]

Hi @embik,
That could be possible. In the meantime, we changed the image used by our workers and also let them be replaced/rotated by the machine deployment, so I unfortunately cannot reproduce that issue.
Thank you!