Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CASM-4908 Runtime container image signature validation #5455

Draft
wants to merge 1 commit into
base: release/1.6
Choose a base branch
from

Conversation

mtupitsyn
Copy link
Contributor

@mtupitsyn mtupitsyn commented Oct 10, 2024

Summary and Scope

During initial testing of image signature validation, it was discovered that Kyverno tries to contact https://artifactory.alogl60.net/ for image verification, and this blocks deployments in air-gapped environments, even in Audit mode (CASMTRIAGE-7283). We need to set Kyverno to contact local registry instead, for both images and their respective signatures. This will allow us to turn on signature validation in runtime (during initial deployments, upgrades and in background on running clusters).

Proposed solution involves these key steps:

  • Deploy new Kyverno cluster policy prepend-registry, which will automatically add registry.local/ to the beginning of image spec for any new pod (if it doesn't already start with registry.local/).
  • Add a mirroring rule to containerd configuration, so that images with names starting from registry.local/ are looked in https://pit.nmn first and in https://registry.local/ second. This rule is needed to support a switch from PIT Nexus to Cloud Nexus during initial install. It is similar to already existing rule for image names starting from artifactory.algol60.net, which now becomes obsolete.
  • Move Kyverno and policies deployment into separate manifest, and deploy it early in install/upgrade pipeline, thus ensuring that image name mutation and signature validation happen to all deployments after Kyverno.
  • For the duration of fresh install, when images are downloaded from PIT Nexus, put a temporary hosts record override into CoreDNS ConfigMap. This override will point to PIT Nexus instead of Cloud Nexus. It is needed for Kyverno admission controller to look for images and their signatures in the right location during fresh install (when Cloud Nexus is not yet deployed).

This change consists of the following PR's:

Issues and Related PRs

Testing

Tested on:

  • Virtual Shasta

Test description:

  • Created custom builds of CSM and docs-csm with changes outlined above
  • Performed multiple automated deployments on vShasta in different combinations: fresh install and upgrade, with validationFailureAction set to Audit and Enforce.

Risks and Mitigations

None known ATM.

Pull Request Checklist

  • Version number(s) incremented, if applicable
  • Copyrights updated
  • License file intact
  • Target branch correct
  • Testing is appropriate and complete, if applicable
  • HPC Product Announcement prepared, if applicable

@@ -62,23 +62,9 @@ spec:
podman tag skopeo:csm-{{ getEnv "CSM_RELEASE" }} quay.io/skopeo/stable:latest

podman run --rm --network host -v {{ getEnv "CSM_PATH" }}/docker:/images:ro \
quay.io/skopeo/stable sync --scoped --src dir --dest docker --dest-tls-verify=false --dest-creds admin:admin123 \
quay.io/skopeo/stable sync --all --scoped --src dir --dest docker --dest-tls-verify=false --dest-creds admin:admin123 \
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This step actually duplicates upload, performed by setup-nexus.sh script for metal nexus, so I'm am not sure if we need it at all. But if we do upload, we must include --all option. Otherwise, skopeo only uploads amd64 manifest without signature, which overrides original multi-platform manifest and detaches existing signature.

@mtupitsyn mtupitsyn force-pushed the feature/prepend-registry branch from 5dba86b to ee66d94 Compare October 11, 2024 21:45
* Kyverno may have been moved to separate manifest - support both locations
* Kyverno should be installed after restart of services which are not yet upgraded
* Strip registry.local/ prefix when uploading images to Nexus for pre-caching
* 'Initialiaze Boostrap Registry' phase of fresh install did not handle multi-arch images correctly
@mtupitsyn mtupitsyn force-pushed the feature/prepend-registry branch from ee66d94 to 9870b76 Compare October 17, 2024 21:40
Copy link
Contributor

github-actions bot commented Nov 7, 2024

This pull-request has not had activity in over 20 days and is being marked as stale.

@github-actions github-actions bot added the Stale Hasn't had activity in over 30 days label Nov 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Stale Hasn't had activity in over 30 days
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant