Replace pod for instances with changed file security context

[nicmorales9]

Description

Should solve #208 by adding a check to replacement workflow. It also adds a feature flag (default off) and an e2e test.

Type of change

• New feature (non-breaking change which adds functionality)

Discussion

Are there any design details that you would like to discuss further?

Testing

Please describe the tests that you ran to verify your changes. Unit tests?
Manual testing?
Unit test, e2e tests
e2e test output (focus tested on the new one)

 {"level":"info","ts":"2024-05-21T17:54:19Z","logger":"controller","msg":"Attempting to run sub-reconciler","namespace":"nicmorales","cluster":"nicdev","reconciler":"controllers.replaceMisconfiguredProcessGroups"}
  {"level":"info","ts":"2024-05-21T17:54:19Z","logger":"controller","msg":"Replace process group","namespace":"nicmorales","cluster":"nicdev","reconciler":"controllers.replaceMisconfiguredProcessGroups","reason":"either FSGroup or FSGroupChangePolicy has changed for the pod SecurityContext"}
  {"level":"info","ts":"2024-05-21T17:54:19Z","logger":"controller","msg":"Replace process group","namespace":"nicmorales","cluster":"nicdev","reconciler":"controllers.replaceMisconfiguredProcessGroups","reason":"either FSGroup or FSGroupChangePolicy has changed for the pod SecurityContext"}
  {"level":"info","ts":"2024-05-21T17:54:19Z","logger":"controller","msg":"Replace process group","namespace":"nicmorales","cluster":"nicdev","reconciler":"controllers.replaceMisconfiguredProcessGroups","reason":"either FSGroup or FSGroupChangePolicy has changed for the pod SecurityContext"}
  {"level":"info","ts":"2024-05-21T17:54:19Z","logger":"controller","msg":"Replace process group","namespace":"nicmorales","cluster":"nicdev","reconciler":"controllers.replaceMisconfiguredProcessGroups","reason":"either FSGroup or FSGroupChangePolicy has changed for the pod SecurityContext"}
  {"level":"info","ts":"2024-05-21T17:54:19Z","logger":"controller","msg":"Replace process group","namespace":"nicmorales","cluster":"nicdev","reconciler":"controllers.replaceMisconfiguredProcessGroups","reason":"either FSGroup or FSGroupChangePolicy has changed for the pod SecurityContext"}
  {"level":"info","ts":"2024-05-21T17:54:19Z","logger":"controller","msg":"Removals have been updated in the cluster status","namespace":"nicmorales","cluster":"nicdev","reconciler":"controllers.replaceMisconfiguredProcessGroups"}
  {"level":"info","ts":"2024-05-21T17:54:19Z","logger":"controller","msg":"Subreconciler finished run","namespace":"nicmorales","cluster":"nicdev","reconciler":"controllers.replaceMisconfiguredProcessGroups","duration_seconds":0.031225991}

Do we need to perform additional testing once this is merged, or perform in a larger testing environment?
Any unique kubernetes environment should be checked for securityContext server-side config changes before enabling this feature.

Documentation

Did you update relevant documentation within this repository?
I think so! Added docstrings + section in docs/manual/replacements_and_deletions.md.

If this change is adding new functionality, do we need to describe it in our user manual?
I think the addition in docs/manual/replacements_and_deletions.md covers it but let me know if not.

If this change is adding or removing subreconcilers, have we updated the core technical design doc to reflect that?
N/A

If this change is adding new safety checks or new potential failure modes, have we documented and how to debug potential issues?
I think they are straightforward enough, but let me know if not

Follow-up

Are there any follow-up issues that we should pursue in the future?

Does this introduce new defaults that we should re-evaluate in the future?

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 1cc8248
• Duration 0:03:00
• Result: ❌ FAILED
• Error: Error while executing command: IMG=${REGISTRY}/${OPERATOR_IMAGE} make container-build container-push. Reason: exit status 2
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 61a2a82
• Duration 0:03:00
• Result: ❌ FAILED
• Error: Error while executing command: IMG=${REGISTRY}/${OPERATOR_IMAGE} make container-build container-push. Reason: exit status 2
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 7022a81
• Duration 0:03:27
• Result: ❌ FAILED
• Error: Error while executing command: IMG=${REGISTRY}/${OPERATOR_IMAGE} make container-build container-push. Reason: exit status 2
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 5c4b38f
• Duration 2:19:28
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 6973575
• Duration 2:33:03
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: adc0fc4
• Duration 2:41:18
• Result: ❌ FAILED
• Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: c840622
• Duration 2:23:30
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: c99f3b9
• Duration 2:37:21
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: d1f15b3
• Duration 2:32:46
• Result: ❌ FAILED
• Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 7bb571b
• Duration 2:25:57
• Result: ❌ FAILED
• Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: aadc211
• Duration 0:04:47
• Result: ❌ FAILED
• Error: reference not found for primary source and source version aadc211efc0f69bd7ca71d127cf28924e7441fe9
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: c18a3e8
• Duration 0:10:29
• Result: ❌ FAILED
• Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 9509765
• Duration 2:41:10
• Result: ❌ FAILED
• Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: eee8a80
• Duration 2:20:15
• Result: ❌ FAILED
• Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 6ca31a7
• Duration 2:19:52
• Result: ❌ FAILED
• Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[nicmorales9]

I believe the only failure was unrelated to the test as it passed when I reran locally and I believe I have seen it flake before:

[FAIL] Operator Upgrades one process is marked for removal and is stuck in terminating state [It] Upgrade from 7.1.57 to 7.3.33 [e2e, pr]
  /codebuild/output/src477579299/src/github.com/FoundationDB/fdb-kubernetes-operator/e2e/test_operator_upgrades/operator_upgrades_test.go:573

rerunning

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 6ca31a7
• Duration 2:33:51
• Result: ❌ FAILED
• Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 11d61ed
• Duration 2:30:28
• Result: ❌ FAILED
• Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: fb29007
• Duration 2:36:57
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 013aa37
• Duration 2:37:45
• Result: ❌ FAILED
• Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 7358c69
• Duration 2:29:28
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[johscheuer]

LGTM 👍

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: dc26323
• Duration 2:20:25
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)