The source code was taken from: https://github.com/paulbouwer/hello-kubernetes
Repository with images: https://quay.io/repository/petr_ruzicka/myc-hello-kubernetes?tab=tags
- Multiple Dockerfile files / sources in one git repository
- Multi-arch container images for each Dockerfile
- Image should be signed (cosign)
- SBOM (Software Bill Of Materials)
- SLSA Provenance
- Build process should never overwrite existing tags (1.2.3) in container repository (doesn't apply for "latest" or "br-")
- Container image should be scanned for "Critical" vulnerabilities before it is pushed to container registry
- There should be a way to push container to registry also with CRITICAL
vulnerabilities (how to progress when there are CRITICAL vulnerabilities which
can not be fixed "immediately")
- Set
container_image_vulnerability_checks=false
- Set
- I want to build test container for specific PR / branch with expiration date
- I want to build only
latest
tag manually (formmain
branch) - Newly released images must create/update following tags in container
repository (example):
1
1.2
1.2.1
latest
Build commands:
docker build -f ./src/app/Dockerfile -t myc-hello-kubernetes:latest src/app
docker build -f ./src/app/Dockerfile-node-18-alpine -t myc-hello-kubernetes:latest-alpine src/app
docker build -f ./src/app/Dockerfile-node-18-debian-slim -t myc-hello-kubernetes:latest-debian-slim src/app
docker build -f ./src/app/Dockerfile-nodejs18-distroless -t myc-hello-kubernetes:latest-distroless src/app
docker build -f ./src/app/Dockerfile-nodejs-16-ubi -t myc-hello-kubernetes:latest-ubi src/app
Run commands:
docker run -p 8080:8080 --rm myc-hello-kubernetes:latest
docker run -p 8080:8080 --rm myc-hello-kubernetes:latest-alpine
docker run -p 8080:8080 --rm myc-hello-kubernetes:latest-debian-slim
docker run -p 8080:8080 --rm myc-hello-kubernetes:latest-distroless
docker run -p 8080:8080 --rm myc-hello-kubernetes:latest-ubi
curl http://localhost:8080/
Debug container:
docker run -it --rm --entrypoint=/bin/sh --user root -p 8080:8080 myc-hello-kubernetes:latest
Run in Kubernetes:
kubectl run myc-hello-kubernetes --image=quay.io/petr_ruzicka/myc-hello-kubernetes
Example repositories:
- https://quay.io/repository/cilium/cilium?tab=tags
- https://quay.io/repository/metallb/controller?tab=tags
- https://quay.io/repository/kubescape/kubescape?tab=tags
- https://quay.io/repository/costoolkit/elemental-operator?tab=tags&tag=latest
- https://hub.docker.com/r/bridgecrew/checkov/tags
Notes:
- Attest and verify artifacts
- https://github.com/marco-lancini/utils/blob/main/ci/github/docker-build-sign-sbom/reusable-docker.yml
Test commands:
IMAGE="bridgecrew/checkov"
IMAGE="quay.io/petr_ruzicka/malware-cryptominer-container:2.0.0"
IMAGE="ghcr.io/external-secrets/external-secrets:v0.7.1"
docker buildx imagetools inspect "${IMAGE}"
cosign verify "${IMAGE}" | jq
cosign verify "${IMAGE}" | jq -r '.[].optional| .Issuer + " | " + .Subject + " | " + .githubWorkflowRef + " | https://rekor.tlog.dev/?logIndex=" + (.Bundle.Payload.logIndex|tostring)'
cosign triangulate "${IMAGE}"
cosign tree "${IMAGE}"
cosign verify-attestation --type slsaprovenance "${IMAGE}"
cosign verify-attestation --type slsaprovenance "${IMAGE}" | jq '.payload |= @base64d | .payload | fromjson'
cosign verify-attestation --type cyclonedx "${IMAGE}"
cosign verify-attestation --type cyclonedx "${IMAGE}" | jq '.payload |= @base64d | .payload | fromjson'
trivy image --debug --sbom-sources rekor "${IMAGE}"
cosign verify-attestation --type cyclonedx "${IMAGE}" | jq -r '.payload' | base64 -d | jq -r '.predicate.Data' > /tmp/sbom.cdx.json
trivy sbom /tmp/sbom.cdx.json
trivy image --scanners license "${IMAGE}"
trivy image --compliance docker-cis "${IMAGE}"
trivy image --scanners none --image-config-scanners config "${IMAGE}"
trivy image --scanners none --image-config-scanners secret "${IMAGE}"
Test commands with other images:
IMAGE="registry.k8s.io/kube-apiserver-amd64:v1.24.0"
IMAGE="quay.io/petr_ruzicka/malware-cryptominer-container:2.0.0"
IMAGE="quay.io/cilium/cilium:v1.13.0"
IMAGE="quay.io/metallb/controller:latest"
IMAGE="quay.io/costoolkit/elemental-operator:v1.1.0"
IMAGE="ghcr.io/fluxcd/source-controller:v0.34.0"
IMAGE="bridgecrew/checkov"
skopeo inspect --raw "docker://${IMAGE}"
cosign verify "${IMAGE}" | jq
cosign verify "${IMAGE}" | jq -r '.[].optional| .Issuer + "-" + .Subject'
cosign triangulate "${IMAGE}"
cosign tree "${IMAGE}"
rekor-cli get --uuid 68a53d0e75463d805dc9437dda5815171502475dd704459a5ce3078edba96226 --format json | jq -r .Attestation | base64 --decode | jq
curl -s https://rekor.sigstore.dev/api/v1/log/entries/68a53d0e75463d805dc9437dda5815171502475dd704459a5ce3078edba96226 | jq
rekor-cli search --email petr.ruzicka@gmail.com
# https://rekor.tlog.dev/?email=petr.ruzicka@gmail.com
rekor-cli get --log-index 8757761
docker manifest inspect "${IMAGE}"
cosign download sbom --platform=linux/amd64 ghcr.io/google/ko
cosign download sbom --platform=linux/amd64 cgr.dev/chainguard/node
cosign download sbom "${IMAGE}"
# Image Manifest
curl -s -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' "https://quay.io/v2/jetstack/cert-manager-controller/manifests/v1.9.1" | jq
# Manifest List
curl -s -H 'Accept: application/vnd.docker.distribution.manifest.list.v2+json' "https://quay.io/v2/jetstack/cert-manager-controller/manifests/v1.9.1" | jq
curl -s -H 'Accept: application/vnd.oci.image.index.v1+json' "https://quay.io/v2/jetstack/cert-manager-controller/manifests/v1.9.1" | jq
# Verify SBOM attestation
cosign verify-attestation --type cyclonedx bridgecrew/checkov | jq '.payload |= @base64d | .payload | fromjson | select(.predicateType == "https://cyclonedx.org/schema") | .predicate.Data'
cosign verify-attestation --type slsaprovenance --key https://ftp.suse.com/pub/projects/security/keys/container–key.pem registry.suse.com/bci/golang@sha256:35bc38ce40811b587a56bcfa328ef077c0703732e3bbedf4dbdf47f612cca04b | jq # DevSkim: ignore DS117838
cosign verify-attestation --type slsaprovenance ghcr.io/thomasvitale/band-service@sha256:388e8d292b55a7934bdaf11277ea9f33c3533258de92eb4b12085717dbdbd875 | jq '.payload |= @base64d | .payload | fromjson'
docker manifest inspect quay.io/jetstack/cert-manager-controller:v1.9.1
slsa-verifier verify-image "ghcr.io/chgl/kube-powertools@sha256:b76dc742957ae883f3ed0c4fe89b54d5c9a9de69a8bc531f9ee12ec995dab10d" --source-uri github.com/chgl/kube-powertools
# https://unifiedguru.com/highlights-from-the-buildkit-v0-11-release-docker/
docker buildx imagetools inspect moby/buildkit:latest --format '{{ json .Provenance }}'
docker buildx imagetools inspect moby/buildkit:latest --format '{{ json (index .Provenance "linux/amd64").SLSA.invocation.configSource }}'
docker buildx imagetools inspect moby/buildkit:latest --format '{{ (index .Provenance "linux/amd64").SLSA.builder.id }}'
docker buildx imagetools inspect moby/buildkit:latest --format '{{ range (index .SBOM "linux/amd64").SPDX.packages }}{{ println .name }}{{ end }}'
docker buildx imagetools inspect ghcr.io/fluxcd/source-controller:v0.34.0 --format "{{ json (index .Provenance \"linux/amd64\").SLSA}}"
docker buildx imagetools inspect ghcr.io/fluxcd/source-controller:v0.34.0 --format "{{ json (index .SBOM \"linux/amd64\").SPDX}}"
docker sbom fluxcd/source-controller:v0.34.0
# https://registry-ui.chainguard.app/?image=quay.io/petr_ruzicka/malware-cryptominer-container:1
# https://registry-ui.chainguard.app/?image=cgr.dev/chainguard/nginx:latest
cosign verify --certificate-github-workflow-repository cilium/cilium --certificate-oidc-issuer https://token.actions.githubusercontent.com --certificate-github-workflow-name "Image Release Build" --certificate-github-workflow-ref refs/tags/v1.13.0 quay.io/cilium/cilium:v1.13.0 | jq
cosign verify --certificate-github-workflow-repository cilium/cilium --certificate-oidc-issuer https://token.actions.githubusercontent.com --attachment sbom quay.io/cilium/cilium:v1.13.0 | jq
I was looking for the vulnerability which affects only the specific architecture
(for example: arm
):
https://security.snyk.io/vuln/SNYK-DEBIAN10-GLIBC-564233
trivy image --severity HIGH,CRITICAL --platform=linux/amd64 node:18.10.0-buster-slim | grep CVE-2020-6096
trivy image --severity HIGH,CRITICAL --platform=linux/arm64 node:18.10.0-buster-slim | grep CVE-2020-6096
-
Build container images with tag latest form main branch and ignore vulnerability scanners
gh workflow run container-build.yml -f container_registry_push=true -f container_image_expires_after=30 -f container_image_skip_vulnerability_checks=true sleep 100
-
Tag "main" branch
TAG="8.0.55" git tag "v${TAG}-beta.0" && git push origin "v${TAG}-beta.0" sleep 10 WORKLOAD_ID=$(gh run list --workflow=container-build.yml --limit 1 --json databaseId | jq -r '.[].databaseId') gh run watch "${WORKLOAD_ID}" # In case some containers has security vulnerabilities they will not be pushed to Container Registry by default # If you want to do a force push please use something like: gh workflow run container-build.yml --ref="v${TAG}-beta.0" -f container_registry_push=true -f container_image_expires_after=365 -f container_image_skip_vulnerability_checks=true sleep 100 git tag "v${TAG}-beta.1" && git push origin "v${TAG}-beta.1" sleep 10 WORKLOAD_ID=$(gh run list --workflow=container-build.yml --limit 1 --json databaseId | jq -r '.[].databaseId') gh run watch "${WORKLOAD_ID}" gh workflow run container-build.yml --ref="v${TAG}-beta.1" -f container_registry_push=true -f container_image_expires_after=365 -f container_image_skip_vulnerability_checks=true sleep 100 git tag "v${TAG}-rc.0" && git push origin "v${TAG}-rc.0" sleep 10 WORKLOAD_ID=$(gh run list --workflow=container-build.yml --limit 1 --json databaseId | jq -r '.[].databaseId') gh run watch "${WORKLOAD_ID}" gh workflow run container-build.yml --ref="v${TAG}-rc.0" -f container_registry_push=true -f container_image_expires_after=365 -f container_image_skip_vulnerability_checks=true sleep 100 git tag "v${TAG}-rc.1" && git push origin "v${TAG}-rc.1" sleep 10 WORKLOAD_ID=$(gh run list --workflow=container-build.yml --limit 1 --json databaseId | jq -r '.[].databaseId') gh run watch "${WORKLOAD_ID}" gh workflow run container-build.yml --ref="v${TAG}-rc.1" -f container_registry_push=true -f container_image_expires_after=365 -f container_image_skip_vulnerability_checks=true sleep 100 git tag "v${TAG}" && git push origin "v${TAG}" sleep 10 WORKLOAD_ID=$(gh run list --workflow=container-build.yml --limit 1 --json databaseId | jq -r '.[].databaseId') gh run watch "${WORKLOAD_ID}" gh workflow run container-build.yml --ref="v${TAG}" -f container_registry_push=true -f container_image_expires_after=365 -f container_image_skip_vulnerability_checks=true
-
Run build process form "main" branch - expires in 300 days
gh workflow run container-build.yml -f container_registry_push=true -f container_image_expires_after=300 -f container_image_skip_vulnerability_checks=true sleep 10 WORKLOAD_ID=$(gh run list --workflow=container-build.yml --limit 1 --json databaseId | jq -r '.[].databaseId') gh run watch "${WORKLOAD_ID}"
-
Run build process form "fix" branch - expires in 30 days (manual workflow execution is needed)
git checkout -b fix && git push gh workflow run container-build.yml --ref=fix -f container_registry_push=true -f container_image_expires_after=30 -f container_image_skip_vulnerability_checks=true sleep 10 WORKLOAD_ID=$(gh run list --workflow=container-build.yml --limit 1 --json databaseId | jq -r '.[].databaseId') gh run watch "${WORKLOAD_ID}" git checkout main && git branch -d fix && git push origin -d fix
-
Run build process form "fix2" branch form PR - expires in 30 days (manual workflow execution is needed)
git checkout -b fix2 date > date.txt git add date.txt && git commit -m "fix(container-build): date" && git push PR_URL=$(gh pr create --fill) gh workflow run container-build.yml --ref=fix2 -f container_registry_push=true -f container_image_expires_after=30 -f container_image_skip_vulnerability_checks=true sleep 10 WORKLOAD_ID=$(gh run list --workflow=container-build.yml --limit 1 --json databaseId | jq -r '.[].databaseId') gh run watch "${WORKLOAD_ID}" date > date.txt git add date.txt && git commit -m "fix(container-build): date2" && git push gh workflow run container-build.yml --ref=fix2 -f container_registry_push=true -f container_image_expires_after=30 -f container_image_skip_vulnerability_checks=true sleep 10 WORKLOAD_ID=$(gh run list --workflow=container-build.yml --limit 1 --json databaseId | jq -r '.[].databaseId') gh run watch "${WORKLOAD_ID}" gh pr close --delete-branch "${PR_URL}"
-
Scheduled build - expires in 300 days