Image scanned from "metadata:annotations:kubectl.kubernetes.io/last-applied-configuration" instead of "spec"

[ngraf]

Description

trivy k8s scans the irrelevant image that is mentioned in metadata:annotations:kubectl.kubernetes.io/last-applied-configuration instead of the image used in spec of kind:Deployment.

The scenario where this bug becomes a real problem is, if you do the following:

1. Create a deployment via kubectl apply
2. Do follow-up updates of that deployment with other tools, e.g. helm that do not create/update the metadata:annotations:kubectl.kubernetes.io/last-applied-configuration annotation
   .. this is what happened to me. It took me a looong time, to find out why trivy gives me unexpected CVEs.

Desired Behavior

Images from spec are scanned.

Actual Behavior

Images from metadata:annotations:kubectl.kubernetes.io/last-applied-configuration are scanned.

Reproduction Steps

1. Make sure you have a deployment present in k8s cluster, that looks like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  annotations:
     kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app":"nginx"},"name":"nginx-deployment","namespace":"default"},"spec":{"replicas":1,"selector":{"matchLabels":{"app":"nginx"}},"template":{"metadata":{"labels":{"app":"nginx"}},"spec":{"containers":[{"image":"nginx:1.13.0","name":"nginx","ports":[{"containerPort":80}]}]}}}}
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

2. Do a trivy k8s scan that targets this deployment
3. The trivy scan will unexpectedly contain CVEs that relate to "nginx:1.13.0" instead of "nginx.1.14.2"

Target

Kubernetes

Scanner

Vulnerability

Output Format

JSON

Mode

None

Debug Output

$ trivy k8s deployments --namespace foo --context MY_CONTEXT -s CRITICAL --report summary --debug                                                                                                                                                                                                                                                                                                      [ 24/09/19 | 12:35pm ]
2024-09-19T12:37:56.069+0200	DEBUG	Severities: ["CRITICAL"]
2024-09-19T12:37:56.069+0200	DEBUG	Ignore statuses	{"statuses": null}
2024-09-19T12:37:56.777+0200	DEBUG	cache dir:  /Users/MY_USER/Library/Caches/trivy
2024-09-19T12:37:56.777+0200	DEBUG	DB update was skipped because the local DB is the latest
2024-09-19T12:37:56.777+0200	DEBUG	DB Schema: 2, UpdatedAt: 2024-09-19 06:12:22.986098678 +0000 UTC, NextUpdate: 2024-09-19 12:12:22.986098548 +0000 UTC, DownloadedAt: 2024-09-19 10:34:23.822328 +0000 UTC
11 / 11 [------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 2 p/s

Summary Report for MY_CONTEXT


Workload Assessment
┌───────────┬──────────────────────────────────┬─────────────────┬───────────────────┬─────────┐
│ Namespace │             Resource             │ Vulnerabilities │ Misconfigurations │ Secrets │
│           │                                  ├─────────────────┼───────────────────┼─────────┤
│           │                                  │        C        │         C         │    C    │
├───────────┼──────────────────────────────────┼─────────────────┼───────────────────┼─────────┤
│ foo       │ Deployment/MY_SERVICE_A          │        1        │                   │         │
│ foo       │ Deployment/MY_SERVICE_B          │        1        │                   │         │
│ foo       │ Deployment/MY_SERVICE_C          │        8        │                   │         │
└───────────┴──────────────────────────────────┴─────────────────┴───────────────────┴─────────┘
Severities: C=CRITICAL

Operating System

MacOs Sequioa 15.0

Version

Version: 0.49.1
Vulnerability DB:
  Version: 2
  UpdatedAt: 2024-04-16 12:12:35.993883383 +0000 UTC
  NextUpdate: 2024-04-16 18:12:35.993882871 +0000 UTC
  DownloadedAt: 2024-04-16 13:12:03.109661 +0000 UTC
Java DB:
  Version: 1
  UpdatedAt: 2024-04-02 00:43:48.241453498 +0000 UTC
  NextUpdate: 2024-04-05 00:43:48.241453297 +0000 UTC
  DownloadedAt: 2024-04-02 09:00:06.632043 +0000 UTC
Policy Bundle:
  Digest: sha256:cdff1bc8c97e4f5cd04782b057c00f5ea8cd81147a506ac4be76bef13710f2d3
  DownloadedAt: 2024-04-16 13:12:03.630765 +0000 UTC

Checklist

• Run trivy clean --all
• Read the troubleshooting

[afdesk]

thanks for the report. let me check it

[afdesk]

Version: 0.49.1

@ngraf could you retest your cluster with the latest Trivy version? thanks

[ngraf]

@afdesk ,
I updated trivy to 0.55.2 and did a retest.
Still the bug is present.

[afdesk]

thanks for recheck! I'll take a look

[afdesk]

@ngraf I've crteated #7573 for more investigation.
I can track it.
thanks!