OCPBUGS#10640: Added clarification point to disk partition BM doc

[dfitzmau]

OCPBUGS-10640

Version(s):
4.16, 4.15 and 4.14

For 4.13 and 4.12, ensure you do not remove:

For disk sizes larger than 100GB, and especially disk sizes larger than 1TB, create a separate /var partition. See "Creating a separate /var partition" and this link:https://access.redhat.com/solutions/5587281[Red Hat Knowledgebase article] for more information.

Link to docs preview:
Disk partioning

• SME has approved this change.
• QE has approved this change.

Additional information:

[openshift-ci-robot]

@dfitzmau: This pull request references Jira Issue OCPBUGS-10640, which is invalid:

• expected the bug to target the "4.15.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Version(s):

Issue:

Link to docs preview:

QE review:

• QE has approved this change.

Additional information:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci-robot]

@dfitzmau: No Jira issue is referenced in the title of this pull request.
To reference a jira issue, add 'XYZ-NNN:' to the title of this pull request and request another refresh with /jira refresh.

In response to this:

OCPBUGS-10640

Version(s):
4.11 to 4.15

Link to docs preview:

QE review:

• QE has approved this change.

Additional information:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ocpdocs-previewbot]

🤖 Tue Mar 19 17:09:23 - Prow CI generated the docs preview:
https://67707--ocpdocs-pr.netlify.app

[andreaskaris]

That doesn't quite cut it for me.

Just as a reminder:
nodefs monitors the disk that contains /var/lib/kubelet
imagefs monitors the disk that contains /var/lib/containers

Thus, in the default configuration, nodefs monitors /, imagefs monitors /.

My issues with the doc:

Looking at https://docs.openshift.com/container-platform/4.14/installing/installing_bare_metal/installing-bare-metal.html#installation-user-infra-machines-advanced_disk_installing-bare-metal

• " This is officially supported for mounting /var or a subdirectory of /var, such as /var/lib/etcd, on a separate partition, but not both."
  This sentence IMO is problematic. We should at least advise against mounting /var in its entirety. When you mount /var in full to a separate partition, the following happens:
  kubelet monitors imagefs = nodefs = /var. Kubelet stops monitoring the root disk.
  The better scheme is to mount /var/lib/containers to a separate partition, in that case:
  imagefs = /var/lib/containers, nodefs=/, therefore imagefs monitors the container location and nodefs monitors the entire root filesystem including logs, etc

• "The use of custom partitions could result in those partitions not being monitored by OpenShift Container Platform or alerted on. If you are overriding the default partitioning, see Understanding OpenShift File System Monitoring (eviction conditions) for more information about how OpenShift Container Platform monitors your host file systems."
  The documentation already contains this - correct - warning, I think I missed that when I created https://issues.redhat.com/browse/OCPBUGS-10640. Be it as it may, I would like to see our recommendations keep this in mind and be rewritten in the spirit of this warning (this means instead of highlighting mounting /var, I'd recommend highlighting mounting /var/lib/containers instead).

• "For disk sizes larger than 100GB, and especially larger than 1TB, create a separate /var partition."
  That advice is misguided, and the same warning appears 2x in the same document (redundant), once with, once without a link to https://access.redhat.com/solutions/5587281
  a) for "large" disk sizes - who considers 100GB large in 2023?
  b) Look at https://access.redhat.com/solutions/5587281: it talks about resizing disks ("The affected Filesystem has been resized (extended)."), and OCP file systems are (nearly?) never resized. If we absolutely want to keep the warning because we're being extra cautious, we should add to it that this only happens with resized disks.
  --> Large partition + xfs + rhel8 + resized = problematic
  --> Large partition + xfs + rhel8 = not problematic
  Investigate if the issue persists in RHEL9, because if not, starting with OCP 4.13 we might drop the warning altogether if RHEL 9 fixed this.

• "The following procedure sets up a separate /var partition"
  I'd change the entire example to /var/lib/containers instead of /var

[kannon92]

My issues with the doc:

Looking at https://docs.openshift.com/container-platform/4.14/installing/installing_bare_metal/installing-bare-metal.html#installation-user-infra-machines-advanced_disk_installing-bare-metal

• " This is officially supported for mounting /var or a subdirectory of /var, such as /var/lib/etcd, on a separate partition, but not both."
  This sentence IMO is problematic. We should at least advise against mounting /var in its entirety. When you mount /var in full to a separate partition, the following happens:
  kubelet monitors imagefs = nodefs = /var. Kubelet stops monitoring the root disk.

I don't know if this is true. I don't think Kubelet will ever stop monitoring the root disk. It will monitor the disk that /var/lib/kubelet is on and consider that the root filesystem. So pods, volumes, ephemeral storage and logs will be on this filesystem.

The better scheme is to mount /var/lib/containers to a separate partition, in that case:
imagefs = /var/lib/containers, nodefs=/, therefore imagefs monitors the container location and nodefs monitors the entire root filesystem including logs, etc

This is correct.

The other issue we should keep in mind is that we may allow users to modify the location for /var/lib/containers. They can change the graphroot in storage.conf. I wrote an upstream blog post about this. Currently we don't really provide support for it but I just want to bring more awareness to this fact.

The blog post I wrote (which is still WIP) is here

[dfitzmau]

Thanks, @kannon92 , @andreaskaris , and @djuran . From reading the comments, I'll take it the bullet points under "There are two cases where you might want to override the default partitioning when installing RHCOS on an OpenShift Container Platform cluster node:" are problematic and need to be removed and replaced with suggestive text that points towards mounting /var/lib/containers in a separate partition?

[andreaskaris]

I don't know if this is true. I don't think Kubelet will ever stop monitoring the root disk. It will monitor the disk that /var/lib/kubelet is on and consider that the root filesystem. So pods, volumes, ephemeral storage and logs will be on this filesystem.

So some of the confusion might be coming from my wording, and if so, I'm sorry about that. What I mean to say is that (and correct me if I'm wrong):

kubelet considers the root filesystem the disk that /var/lib/kubelet is on.
Let's say / is mounted on /dev/sda1, and /var is mounted on /dev/sda2.
In that case, kubelet considers the root filesystem /var (because that's what /var/lib/kubelet is under) and monitors the contents of /dev/sda2 as nodefs. It will also monitor imagefs = /var/lib/containers, but that's also on /var. So kubelet will only monitor /var/, thus /dev/sda2.
The kubelet will no longer monitor the filesystem that's holding /root, /etc/, /usr, etc. assuming that all of these reside on /.

[kannon92]

I don't know if this is true. I don't think Kubelet will ever stop monitoring the root disk. It will monitor the disk that /var/lib/kubelet is on and consider that the root filesystem. So pods, volumes, ephemeral storage and logs will be on this filesystem.

So some of the confusion might be coming from my wording, and if so, I'm sorry about that. What I mean to say is that (and correct me if I'm wrong):

kubelet considers the root filesystem the disk that /var/lib/kubelet is on. Let's say / is mounted on /dev/sda1, and /var is mounted on /dev/sda2. In that case, kubelet considers the root filesystem /var (because that's what /var/lib/kubelet is under) and monitors the contents of /dev/sda2 as nodefs. It will also monitor imagefs = /var/lib/containers, but that's also on /var. So kubelet will only monitor /var/, thus /dev/sda2. The kubelet will no longer monitor the filesystem that's holding /root, /etc/, /usr, etc. assuming that all of these reside on /.

That is all correct.

[dfitzmau]

Hi @andreaskaris and @kannon92 , I updated the PR based on your feedback. I'm not too versed in OCP storage components, so please feel free to suggest better sentences. I assume I should remove the https://access.redhat.com/solutions/5587281 link for 4.14 + docs as it applies to RHEL 8?

As this section is highly confusing its current format, maybe simplifying it would be better?

[dfitzmau]

/retest

[kcarmichael08]

Couple of suggestions. Nice job! (Added labels but wasn't sure about 4.16, so didn't add that one.)

[openshift-ci]

@dfitzmau: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[gpei]

/lgtm

[dfitzmau]

/label merge-review-needed

[ShaunaDiaz]

/remove-label merge-review-needed

[ShaunaDiaz]

/cherrypick enterprise-4.14

[ShaunaDiaz]

/cherrypick enterprise-4.15

[ShaunaDiaz]

/cherrypick enterprise-4.16

[openshift-cherrypick-robot]

@ShaunaDiaz: new pull request created: #73615

In response to this:

/cherrypick enterprise-4.14

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-cherrypick-robot]

@ShaunaDiaz: new pull request created: #73616

In response to this:

/cherrypick enterprise-4.15

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-cherrypick-robot]

@ShaunaDiaz: new pull request created: #73617

In response to this:

/cherrypick enterprise-4.16

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.