-
Notifications
You must be signed in to change notification settings - Fork 401
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Move Ignition file generation to controller/common, fix Compression #3128
Move Ignition file generation to controller/common, fix Compression #3128
Conversation
79a0d9c
to
b2eb20c
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(Just a quick look as someone not familiar with the problem space, going just by the diff and the problem description.)
ACK; apart from the bug fix, the consolidation of copy&pasted data is certainly a great improvement.
I wonder about the change to use dataurl.EncodeBytes
. It does seem to me to be an improvement, or at worst immaterial, I’m just tempted to think that the people explicitly setting a MIME type and ASCII could have had a reason for doing so. Or… maybe not?
pkg/controller/common/helpers.go
Outdated
}, | ||
FileEmbedded1: ign3types.FileEmbedded1{ | ||
Contents: ign3types.Resource{ | ||
Source: StrToPtr(dataurl.EncodeBytes(contents)), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- This is doing MIME type auto detection (notably using
text/plain
; charset=utf-8
, orapplication/content-stream
, while the old code was usually manually settingtext/plain
. Does that matter? - AFAICS this sets base64 data encoding to
base64
, but all existing data manually sets ASCII. Does that matter? It does seem to me that base64 is a better default (binary-safe and we don’t have to worry about special cases).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See below
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well, hmm...actually it does mean that these files will start appearing as base64 inside MachineConfig, which will mean people need to base64 decode them manually if relevant.
It will also be a "spurious" change on upgrades, e.g. the machineconfig hash will be different, but the physical files will not.
I am not sure if either of these really matter though.
This is only interpreted by Ignition, which ends up calling |
b2eb20c
to
fc3bff9
Compare
Thanks for the thoughtful review! Addressed all comments. The e2e test failure I think is just a flake, cc #3039 |
See coreos/butane#332 (comment) Basically, everywhere we make a "raw" Ignition file type, we need to be explicitly setting the compression field. This is prep for updating butane and using it for some of our templates, which will start compressing files automatically. The problem is Ignition config merging is a bit broken with respect to an unset compression field. We need to make it the explicit empty string. Now, when I was digging into this I noticed that we'd ended up with our API to make Ignition files in the `test/helpers`. We shouldn't have *core* code importing APIs from `test/`. This moves the APIs into `controller/common`. Port the kubelet config and container runtime config bits to this - notice how much the code shrinks!
fc3bff9
to
da5184f
Compare
/lgtm (The |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: cgwalters, mtrmac The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
/retest-required Please review the full test history for this PR and help us cut down flakes. |
2 similar comments
/retest-required Please review the full test history for this PR and help us cut down flakes. |
/retest-required Please review the full test history for this PR and help us cut down flakes. |
@cgwalters: all tests passed! Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
Today's offhand thought: when we can hard switch to layering, the role of Ignition in the MCO will shrink...perhaps to nearly nil. If we can move everything we need to bootstrap into the pointer config, the MCS goes away. How we generate the content that goes into our images becomes very flexible - the stock templates don't have to be Ignition or butane or whatever necessarily, etc. |
When da5184f (Move Ignition file generation to controller/common, fix Compression, 2022-04-29, openshift#3128) created the NewIgnFileBytes* helpers, the call-sites which had previously explicitly set file modes began leaving them unset. Updating from 4.10 to 4.11 and picking up da5184f, clusters which include ContainerRuntimeConfig (or possibly KubeletConfig) which flow through the NewIgnFileBytes* helpers would have rendered MachineConfig entries with null/unset mode values. While this is not a problem for Ignition v3 clients, where there is an explicit 0644 default: $ git clone --depth 1 --branch main https://github.com/coreos/ignition.git $ cd ignition $ git --no-pager log --oneline -1 7652b83 (grafted, HEAD -> main, origin/main, origin/HEAD) Merge pull request openshift#1409 from bgilbert/notes $ git --no-pager grep '_mode_.*file' docs docs/configuration-v3_0.md: * **_mode_** (integer): the file's permission mode. Note that the mode must be properly specified as a **decimal** value (i.e. 0644 -> 420). Setuid/setgid/sticky bits are not supported. If not specified, the permission mode for files defaults to 0644 or the existing file's permissions if `overwrite` is false, `contents.source` is unspecified, and a file already exists at the path. docs/configuration-v3_1.md: * **_mode_** (integer): the file's permission mode. Note that the mode must be properly specified as a **decimal** value (i.e. 0644 -> 420). Setuid/setgid/sticky bits are not supported. If not specified, the permission mode for files defaults to 0644 or the existing file's permissions if `overwrite` is false, `contents.source` is unspecified, and a file already exists at the path. docs/configuration-v3_2.md: * **_mode_** (integer): the file's permission mode. Note that the mode must be properly specified as a **decimal** value (i.e. 0644 -> 420). Setuid/setgid/sticky bits are not supported. If not specified, the permission mode for files defaults to 0644 or the existing file's permissions if `overwrite` is false, `contents.source` is unspecified, and a file already exists at the path. docs/configuration-v3_3.md: * **_mode_** (integer): the file's permission mode. Note that the mode must be properly specified as a **decimal** value (i.e. 0644 -> 420). Setuid/setgid/sticky bits are not supported. If not specified, the permission mode for files defaults to 0644 or the existing file's permissions if `overwrite` is false, `contents.source` is unspecified, and a file already exists at the path. docs/configuration-v3_4_experimental.md: * **_mode_** (integer): the file's permission mode. Note that the mode must be properly specified as a **decimal** value (i.e. 0644 -> 420). Setuid/setgid/sticky bits are supported. If not specified, the permission mode for files defaults to 0644 or the existing file's permissions if `overwrite` is false, `contents.source` is unspecified, and a file already exists at the path. it is a problem for Ignition v2 clients, where there is no such default: $ git clone --depth 1 --branch spec2x https://github.com/coreos/ignition.git $ cd ignition $ git --no-pager log --oneline -1 4db553d (grafted, HEAD -> spec2x, origin/spec2x) Merge pull request openshift#1067 from bgilbert/spec2y $ git --no-pager grep '_mode_.*file' doc doc/configuration-v2_0.md: * **_mode_** (integer): the file's permission mode. Note that the mode must be properly specified as a **decimal** value (i.e. 0644 -> 420). doc/configuration-v2_1.md: * **_mode_** (integer): the file's permission mode. Note that the mode must be properly specified as a **decimal** value (i.e. 0644 -> 420). doc/configuration-v2_2.md: * **_mode_** (integer): the file's permission mode. Note that the mode must be properly specified as a **decimal** value (i.e. 0644 -> 420). doc/configuration-v2_3.md: * **_mode_** (integer): the file's permission mode. Note that the mode must be properly specified as a **decimal** value (i.e. 0644 -> 420). doc/configuration-v2_4.md: * **_mode_** (integer): the file's permission mode. Note that the mode must be properly specified as a **decimal** value (i.e. 0644 -> 420). doc/configuration-v2_5-experimental.md: * **_mode_** (integer): the file's permission mode. Note that the mode must be properly specified as a **decimal** value (i.e. 0644 -> 420). There is also no in-cluster component responsible for updating boot image configuration, or even complaining about stale boot image configuration, in Machines or MachineSets. This causes problems when 4.10 clusters with older boot images (at least older than 4.6 [1], I haven't pinned down a specific version): 1. Cluster updates to 4.11. 2. Incoming machine-config operator renders fresh MachineConfig from any ContainerRuntimeConfig or KubeletConfig. 3. Those rendered MachineConfig contain null/unset modes. 4. A new Machine is created with the old Ignition v2 boot image. 5. The boot image Ignition lays down a file with mode 0, because Ignition v2 does not define a default mode. 6. The boot image pivots into the modern machine-os-content image. 7. The incoming (modern) machine-config daemon notices the mode 0 file, compares it with its Ignition-v3-like 0644 default mode, and complains with [2]: unexpected on-disk state validating against rendered-worker-44f2c74623e4d3bbe9557a9e82102c01: mode mismatch for file: "/etc/crio/crio.conf.d/01-ctrcfg-pidsLimit"; expected: -rw-r--r--/420/0644; received: ----------/0/0 Folks should probably update to more modern boot images, but this commit restores the explicit mode to avoid breaking on this mode-mismatch issue for folks who are still running Ignition v2 boot images. [1]: https://bugzilla.redhat.com/show_bug.cgi?id=2102004#c28 [2]: https://bugzilla.redhat.com/show_bug.cgi?id=2102004#c0
When da5184f (Move Ignition file generation to controller/common, fix Compression, 2022-04-29, openshift#3128) created the NewIgnFileBytes* helpers, the call-sites which had previously explicitly set file modes began leaving them unset. Updating from 4.10 to 4.11 and picking up da5184f, clusters which include ContainerRuntimeConfig (or possibly KubeletConfig) which flow through the NewIgnFileBytes* helpers would have rendered MachineConfig entries with null/unset mode values. While this is not a problem for Ignition v3 clients, where there is an explicit 0644 default: $ git clone --depth 1 --branch main https://github.com/coreos/ignition.git $ cd ignition $ git --no-pager log --oneline -1 7652b83 (grafted, HEAD -> main, origin/main, origin/HEAD) Merge pull request openshift#1409 from bgilbert/notes $ git --no-pager grep '_mode_.*file' docs docs/configuration-v3_0.md: * **_mode_** (integer): the file's permission mode. Note that the mode must be properly specified as a **decimal** value (i.e. 0644 -> 420). Setuid/setgid/sticky bits are not supported. If not specified, the permission mode for files defaults to 0644 or the existing file's permissions if `overwrite` is false, `contents.source` is unspecified, and a file already exists at the path. docs/configuration-v3_1.md: * **_mode_** (integer): the file's permission mode. Note that the mode must be properly specified as a **decimal** value (i.e. 0644 -> 420). Setuid/setgid/sticky bits are not supported. If not specified, the permission mode for files defaults to 0644 or the existing file's permissions if `overwrite` is false, `contents.source` is unspecified, and a file already exists at the path. docs/configuration-v3_2.md: * **_mode_** (integer): the file's permission mode. Note that the mode must be properly specified as a **decimal** value (i.e. 0644 -> 420). Setuid/setgid/sticky bits are not supported. If not specified, the permission mode for files defaults to 0644 or the existing file's permissions if `overwrite` is false, `contents.source` is unspecified, and a file already exists at the path. docs/configuration-v3_3.md: * **_mode_** (integer): the file's permission mode. Note that the mode must be properly specified as a **decimal** value (i.e. 0644 -> 420). Setuid/setgid/sticky bits are not supported. If not specified, the permission mode for files defaults to 0644 or the existing file's permissions if `overwrite` is false, `contents.source` is unspecified, and a file already exists at the path. docs/configuration-v3_4_experimental.md: * **_mode_** (integer): the file's permission mode. Note that the mode must be properly specified as a **decimal** value (i.e. 0644 -> 420). Setuid/setgid/sticky bits are supported. If not specified, the permission mode for files defaults to 0644 or the existing file's permissions if `overwrite` is false, `contents.source` is unspecified, and a file already exists at the path. it is a problem for Ignition v2 clients, where there is no such default: $ git clone --depth 1 --branch spec2x https://github.com/coreos/ignition.git $ cd ignition $ git --no-pager log --oneline -1 4db553d (grafted, HEAD -> spec2x, origin/spec2x) Merge pull request openshift#1067 from bgilbert/spec2y $ git --no-pager grep '_mode_.*file' doc doc/configuration-v2_0.md: * **_mode_** (integer): the file's permission mode. Note that the mode must be properly specified as a **decimal** value (i.e. 0644 -> 420). doc/configuration-v2_1.md: * **_mode_** (integer): the file's permission mode. Note that the mode must be properly specified as a **decimal** value (i.e. 0644 -> 420). doc/configuration-v2_2.md: * **_mode_** (integer): the file's permission mode. Note that the mode must be properly specified as a **decimal** value (i.e. 0644 -> 420). doc/configuration-v2_3.md: * **_mode_** (integer): the file's permission mode. Note that the mode must be properly specified as a **decimal** value (i.e. 0644 -> 420). doc/configuration-v2_4.md: * **_mode_** (integer): the file's permission mode. Note that the mode must be properly specified as a **decimal** value (i.e. 0644 -> 420). doc/configuration-v2_5-experimental.md: * **_mode_** (integer): the file's permission mode. Note that the mode must be properly specified as a **decimal** value (i.e. 0644 -> 420). There is also no in-cluster component responsible for updating boot image configuration, or even complaining about stale boot image configuration, in Machines or MachineSets. This causes problems when 4.10 clusters with older boot images (at least older than 4.6 [1], I haven't pinned down a specific version): 1. Cluster updates to 4.11. 2. Incoming machine-config operator renders fresh MachineConfig from any ContainerRuntimeConfig or KubeletConfig. 3. Those rendered MachineConfig contain null/unset modes. 4. A new Machine is created with the old Ignition v2 boot image. 5. The boot image Ignition lays down a file with mode 0, because Ignition v2 does not define a default mode. 6. The boot image pivots into the modern machine-os-content image. 7. The incoming (modern) machine-config daemon notices the mode 0 file, compares it with its Ignition-v3-like 0644 default mode, and complains with [2]: unexpected on-disk state validating against rendered-worker-44f2c74623e4d3bbe9557a9e82102c01: mode mismatch for file: "/etc/crio/crio.conf.d/01-ctrcfg-pidsLimit"; expected: -rw-r--r--/420/0644; received: ----------/0/0 Folks should probably update to more modern boot images, but this commit restores the explicit mode to avoid breaking on this mode-mismatch issue for folks who are still running Ignition v2 boot images. [1]: https://bugzilla.redhat.com/show_bug.cgi?id=2102004#c28 [2]: https://bugzilla.redhat.com/show_bug.cgi?id=2102004#c0
See coreos/butane#332 (comment)
Basically, everywhere we make a "raw" Ignition file type, we need
to be explicitly setting the compression field. This is prep
for updating butane and using it for some of our templates, which
will start compressing files automatically.
The problem is Ignition config merging is a bit broken with respect
to an unset compression field. We need to make it the explicit
empty string.
Now, when I was digging into this I noticed that we'd ended up with
our API to make Ignition files in the
test/helpers
. We shouldn'thave core code importing APIs from
test/
.This moves the APIs into
controller/common
.Port the kubelet config and container runtime config bits to this -
notice how much the code shrinks!