Skip to content

Commit

Permalink
OCPBUGS-19303: Changed OKD/FCOS workaround to also support Agent-base…
Browse files Browse the repository at this point in the history
…d Installer

OKD/FCOS uses FCOS as its bootimage, i.e. when booting cluster nodes
the first time during installation. FCOS does not provide tools such
as OpenShift Client (oc) or crio.service which Agent-based Installer
uses at the rendezvous host, e.g. to launch the bootstrap control
plane.

RHCOS and SCOS include these tools, but FCOS has to pivot the root fs
[1] to okd-machine-os [2] first in order to make those tools available.

Pivoting uses 'rpm-ostree rebase' but the rendezvous host is booted
the first time the node boots from a FCOS Live ISO where the root fs
and /sysroot are mounted read-only. Thus 'rpm-ostree rebase' fails and
necessary tools will not be available, causing the setup to stall.

Until rpm-ostree has implemented support for rebasing Live ISOs [3],
this patch adapts the workaround for SNO installations [4] to also
support Agent-based Installer.

In particular, the Go conditional {{- if .BootstrapInPlace }} which
is used to mark a SNO install has been replaced with a shell if-else
which checks at runtime whether the system is launched from are on a
Live ISO.
Most code in the OpenShift ecosystem is written with RHCOS in mind
and often assumes that tools like oc or crio.service are available.
These assumptions can be satisfied by applying this workaround to all
Live ISO boots. It will not remove functionality or overwrite
configuration files in /etc and thus side effects should be minimal.

The Go conditional {{- if .BootstrapInPlace }} in the release-image-\
pivot.service has been dropped completely. This service is only used
in OKD only, so OCP will not be impacted at all. The 'Before=' option
will not cause systemd to fail if a service does not exist. So, in
case bootkube.service or kubelet.service do not exist, the option will
have no effect.
When bootkube.service or kubelet.service do exist, it must always be
ensured that release-image-pivot.service is started first because it
might reboot the system or change /usr in the Live ISO use case.
So it is safe to drop the Go conditional and ask systemd to always
launch release-image-pivot.service before bootkube.service and
kubelet.service.

[0] https://github.com/openshift/installer/blob/master/data/data/bootstrap/files/usr/local/bin/bootkube.sh.template
[1] https://github.com/openshift/installer/blob/master/data/data/bootstrap/files/usr/local/bin/bootstrap-pivot.sh.template
[2] https://github.com/openshift/okd-machine-os
[3] coreos/rpm-ostree#4547
[4] openshift#7445

(cherry picked from commit b2bbc85)
  • Loading branch information
JM1 committed Jan 11, 2024
1 parent 72d4834 commit 09dabb2
Show file tree
Hide file tree
Showing 3 changed files with 70 additions and 21 deletions.
81 changes: 64 additions & 17 deletions data/data/bootstrap/files/usr/local/bin/bootstrap-pivot.sh.template
Original file line number Diff line number Diff line change
Expand Up @@ -42,25 +42,72 @@ if [ ! -f /opt/openshift/.pivot-done ]; then
record_service_stage_start "rebase-to-okd-os-image"
{{if .IsFCOS -}}
mnt="$(podman image mount "${MACHINE_OS_IMAGE}")"
{{- if or (.BootstrapInPlace) (eq .Invoker "agent-installer") }}
# SNO setup boots into Live ISO which cannot be rebased
# https://github.com/coreos/rpm-ostree/issues/4547
mkdir /var/mnt/{upper,worker}
mount -t overlay overlay -o "lowerdir=/usr:$mnt/usr" /usr
mount -t overlay overlay -o "lowerdir=/etc:$mnt/etc,upperdir=/var/mnt/upper,workdir=/var/mnt/worker" /etc
systemctl daemon-reload

# Workaround for SELinux denials when launching crio.service from overlayfs
setenforce Permissive
# The bootstrap host during SNO installation and the rendezvous host of Agent-based Installer both boot into a Live
# ISO which cannot be rebased. Until rpm-ostree supports this live rebase [0], the following workaround will mount the
# proper OKD/FCOS Machine OS image over the existing mount at /usr and copy new config files to /etc.
# [0] https://github.com/coreos/rpm-ostree/issues/4547
if grep -q coreos.liveiso= /proc/cmdline; then
mount -t tmpfs -o size=50% none /var/mnt/
rsync -aHAXx "$mnt/" /var/mnt/
mount -t overlay overlay -o lowerdir=/usr:/var/mnt/usr /usr
rsync -rlt --ignore-existing /var/mnt/etc/ /etc/

systemctl start crio.service
# No reboot necessary because SNO setup will reboot system
{{ else }}
pushd "${mnt}/bootstrap"
# shellcheck disable=SC1091
. ./pre-pivot.sh
popd
{{ end -}}
# Agent-based Installer will launch a ephemeral control plane at the rendezvous host which will create and publish
# Ignition configs for the other master nodes. These Ignition configs must match what the in-cluster control plane
# would generate else machine config operator will fail [0]. Because the rendezvous host is booted with a FCOS Live
# ISO without any OKD/FCOS related changes, we have to copy the manifests from OKD Machine OS manually to the
# bootstrap manifests folder of the rendezvous host.
# [0] https://access.redhat.com/solutions/4970731
mkdir -p /var/opt/openshift/manifests
cp -av /var/mnt/manifests/*.* /var/opt/openshift/manifests/

# Load new systemd unit files and configuration such as crio.service after mounting the content of OKD/FCOS Machine
# OS over /usr and copying new files to /etc
systemctl daemon-reload

# Apply presets from OKD Machine OS
systemctl preset-all

# On OKD/FCOS prior to commit e859a66 [0] systemd-resolved is used by default and NetworkManager's DNS handling is
# disabled. In this case, CoreDNS fails to listen to 127.0.0.53:53 when Agent-based Installer boots its the
# rendezvous host with a Fedora CoreOS bootimage because by default FCOS' systemd-resolved already listens to this
# port. OKD/FCOS disables resolved's stub listener [1] but the resolved must be restarted for this setting to take
# effect.
# On OKD/FCOS since commit e859a66 [0] systemd-resolved is disabled by default and NetworkManager's DNS handling is
# used. However, the bootimage is vanilla FCOS and thus uses systemd-resolved by default. The latter has to be
# disabled after rebasing to OKD Machine OS and NetworkManager as well as the service to fix /etc/resolv.conf have
# to be started.
# [0] https://github.com/openshift/okd-machine-os/commit/e859a6643330596a8a282aeb4bf853763a2d219e
# [1] https://github.com/openshift/okd-machine-os/blob/28dec35d60ea07069366b22ebdcb296d429b15e9/overlay.d/99okd/etc/systemd/resolved.conf.d/okd-no-dns-stub.conf
if [ -e /etc/systemd/resolved.conf.d/okd-no-dns-stub.conf ]; then
systemctl restart systemd-resolved.service
else
systemctl disable --now systemd-resolved.service
fi

if systemctl list-unit-files -q fix-resolvconf.service >/dev/null; then
systemctl stop NetworkManager.service
systemctl start fix-resolvconf.service
systemctl start NetworkManager.service
nmcli general reload dns-full
fi

# Workaround for SELinux denials when launching crio.service from overlayfs
setenforce Permissive

# crio.service is not part of FCOS but of OKD Machine OS. It will loaded after systemctl daemon-reload above but has
# to be started manually
systemctl start crio.service

# No reboot necessary because setup will reboot the system automatically
else
pushd "${mnt}/bootstrap"
# shellcheck disable=SC1091
. ./pre-pivot.sh
popd
fi
record_service_stage_success
{{else if .IsSCOS -}}
chmod 0644 /etc/containers/registries.conf
rpm-ostree rebase --experimental "ostree-unverified-registry:${MACHINE_OS_IMAGE}"
Expand Down
6 changes: 6 additions & 0 deletions data/data/bootstrap/systemd/units/kubelet.service.template
Original file line number Diff line number Diff line change
@@ -1,7 +1,13 @@
[Unit]
Description=Kubernetes Kubelet
Wants=rpc-statd.service crio.service release-image.service
{{if .IsOKD -}}
Wants=release-image-pivot.service
{{end -}}
After=crio.service release-image.service
{{if .IsOKD -}}
After=release-image-pivot.service
{{end -}}

[Service]
Type=notify
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,7 @@
Description=Pivot bootstrap to the OpenShift Release Image
Wants=release-image.service
After=release-image.service
{{- if or (.BootstrapInPlace) (eq .Invoker "agent-installer") }}
Before=bootkube.service kubelet.service
{{ else }}
Before=bootkube.service
{{ end -}}

[Service]
Type=oneshot
Expand Down

0 comments on commit 09dabb2

Please sign in to comment.