Gitlab unable to parse json report (gitlab template) when url is an empty string

[DmitriyLewen]

Discussed in #6345

^{Originally posted by behemoth-il March 19, 2024}

Description

I've originally opened a report to Gitlab (suggested fix?), but they've passed the ball to Trivy maintainers, to handle avoiding empty strings as url value.

So if the report has an empty string at .vulnerabilities[].identifiers[].url, Gitlab's validation process will fail the parsing, and a report won't be shown in Gitlab's security dashboard.

Faulty sections in the templated json:

$ jq '.vulnerabilities[7]' gl-container-scanning-report.json
{
  "id": "DLA-3482-1",
  "name": "debian-archive-keyring - security update",
  "description": "",
  "severity": "Unknown",
  "solution": "Upgrade debian-archive-keyring to 2019.1+deb10u2",
  "location": {
    "dependency": {
      "package": {
        "name": "debian-archive-keyring"
      },
      "version": "2019.1+deb10u1"
    },
    "operating_system": "Unknown",
    "image": "python:3.9-slim-buster"
  },
  "identifiers": [
    {
      "type": "cve",
      "name": "DLA-3482-1",
      "value": "DLA-3482-1",
      "url": ""
    }
  ],
  "links": []
}

$ jq '.vulnerabilities[184]' gl-container-scanning-report.json
{
  "id": "DLA-3684-1",
  "name": "tzdata - new timezone database",
  "description": "",
  "severity": "Unknown",
  "solution": "Upgrade tzdata to 2021a-0+deb10u12",
  "location": {
    "dependency": {
      "package": {
        "name": "tzdata"
      },
      "version": "2021a-0+deb10u11"
    },
    "operating_system": "Unknown",
    "image": "python:3.9-slim-buster"
  },
  "identifiers": [
    {
      "type": "cve",
      "name": "DLA-3684-1",
      "value": "DLA-3684-1",
      "url": ""
    }
  ],
  "links": []
}

Gitlab Security Dashboard:

JSONs:
gl-container-scanning-report.json
generic-result.json

Desired Behavior

Gitlab's json template should not have urls with empty strings in them.

Actual Behavior

When the original report (generic json) doesn't have Referances, the output from the template will have empty sting in the url value of .vulnerabilities[].identifiers[].url, cause Gitlab parse to fail and not show a report.

Reproduction Steps

1. In order to view the security dashboard, you will need Gitlab Ultimate (free trial for 30 days).
2. Create a new repository.
3. My example requires Gitlab Runner executed as a Shell (you probably can do it with DIND, but my example won't work for you).
4. Create a new .gitlab-ci.yml, content:

stages:
  - build
  - security

build-job:
  stage: build
  script:
    - docker pull python:3.9-slim-buster

trivy-setup:
  stage: security
  script:
  - export TRIVY_VERSION=v0.50.0
  - mkdir -p ~/bin
  - curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/${TRIVY_VERSION}/contrib/install.sh | sh -s -- -b ~/bin ${TRIVY_VERSION}
  - curl -sSL -o /tmp/gitlab.tpl https://raw.githubusercontent.com/aquasecurity/trivy/${TRIVY_VERSION}/contrib/gitlab.tpl

trivy-scan:
  stage: security
  script:
  - ~/bin/trivy image --timeout 20m --no-progress --format template --template "@/tmp/gitlab.tpl" --output gl-container-scanning-report.json python:3.9-slim-buster
  artifacts:
    reports:
      container_scanning: gl-container-scanning-report.json
    paths:
      - gl-container-scanning-report.html
  dependencies:
    - "trivy-setup"
    - "build-job"

### Target

Container Image

### Scanner

Vulnerability

### Output Format

Template

### Mode

Standalone

### Debug Output

```bash
2024-03-19T03:28:32.598-0700	INFO	Vulnerability scanning is enabled
2024-03-19T03:28:32.598-0700	INFO	Secret scanning is enabled
2024-03-19T03:28:32.598-0700	INFO	If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2024-03-19T03:28:32.598-0700	INFO	Please see also https://aquasecurity.github.io/trivy/v0.50/docs/scanner/secret/#recommendation for faster secret detection
2024-03-19T03:28:32.614-0700	INFO	Detected OS: debian
2024-03-19T03:28:32.614-0700	INFO	Detecting Debian vulnerabilities...
2024-03-19T03:28:32.622-0700	INFO	Number of language-specific files: 1
2024-03-19T03:28:32.622-0700	INFO	Detecting python-pkg vulnerabilities...
gitlab-runner@gitlab-runner:~/builds/xWfgo9HKi/0/behemoth-labs/gitlab/simple-trivy-integration$ ~/bin/trivy image --timeout 20m --no-progress --format template --template "@/tmp/gitlab.tpl" --output gl-container-scanning-report.json python:3.9-slim-buster --debug
2024-03-19T03:28:46.272-0700	DEBUG	Severities: ["UNKNOWN" "LOW" "MEDIUM" "HIGH" "CRITICAL"]
2024-03-19T03:28:46.272-0700	DEBUG	Ignore statuses	{"statuses": null}
2024-03-19T03:28:46.306-0700	DEBUG	cache dir:  /home/gitlab-runner/.cache/trivy
2024-03-19T03:28:46.306-0700	DEBUG	DB update was skipped because the local DB is the latest
2024-03-19T03:28:46.307-0700	DEBUG	DB Schema: 2, UpdatedAt: 2024-03-19 06:11:26.197336549 +0000 UTC, NextUpdate: 2024-03-19 12:11:26.197336089 +0000 UTC, DownloadedAt: 2024-03-19 08:57:33.48259191 +0000 UTC
2024-03-19T03:28:46.307-0700	INFO	Vulnerability scanning is enabled
2024-03-19T03:28:46.307-0700	DEBUG	Vulnerability type:  [os library]
2024-03-19T03:28:46.307-0700	INFO	Secret scanning is enabled
2024-03-19T03:28:46.307-0700	INFO	If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2024-03-19T03:28:46.307-0700	INFO	Please see also https://aquasecurity.github.io/trivy/v0.50/docs/scanner/secret/#recommendation for faster secret detection
2024-03-19T03:28:46.307-0700	DEBUG	Enabling misconfiguration scanners: [azure-arm cloudformation dockerfile helm kubernetes terraform terraformplan-json terraformplan-snapshot]
2024-03-19T03:28:46.310-0700	DEBUG	No secret config detected: trivy-secret.yaml
2024-03-19T03:28:46.310-0700	DEBUG	The nuget packages directory couldn't be found. License search disabled
2024-03-19T03:28:46.311-0700	DEBUG	No secret config detected: trivy-secret.yaml
2024-03-19T03:28:46.311-0700	DEBUG	Image ID: sha256:c84dbfe3b8deeb39e17d121220107f8354a9083b468a320a77708cd128f11c87
2024-03-19T03:28:46.311-0700	DEBUG	Diff IDs: [sha256:e2ef8a51359d088511d34c725305c220294a1fcd5fe5e5dbe4d698c7239ce2c9 sha256:ae2d55769c5efcb6230d27c88eef033128fa1d238bdafe50812402f471152bb7 sha256:14cbeede8d6e46477338e9edd10f89c7a4f22ced60d4e6ae5ac1f0f886a19065 sha256:7fb1037e08b36a6ebd4b7f27021e7568a13dbc7cc72674a1b6361d0b53dc8f2f sha256:067ea27560c1b4abe60cab2335c24e824b74705d534e8d9e561660c116ceb1d0]
2024-03-19T03:28:46.311-0700	DEBUG	Base Layers: [sha256:e2ef8a51359d088511d34c725305c220294a1fcd5fe5e5dbe4d698c7239ce2c9]
2024-03-19T03:28:46.322-0700	INFO	Detected OS: debian
2024-03-19T03:28:46.322-0700	INFO	Detecting Debian vulnerabilities...
2024-03-19T03:28:46.322-0700	DEBUG	debian: os version: 10
2024-03-19T03:28:46.322-0700	DEBUG	debian: the number of packages: 93
2024-03-19T03:28:46.329-0700	INFO	Number of language-specific files: 1
2024-03-19T03:28:46.329-0700	INFO	Detecting python-pkg vulnerabilities...
2024-03-19T03:28:46.329-0700	DEBUG	Detecting library vulnerabilities, type: python-pkg, path:

Operating System

Linux Debian 12

Version

Version: 0.50.0
Vulnerability DB:
  Version: 2
  UpdatedAt: 2024-03-19 06:11:26.197336549 +0000 UTC
  NextUpdate: 2024-03-19 12:11:26.197336089 +0000 UTC
  DownloadedAt: 2024-03-19 08:57:33.48259191 +0000 UTC

Checklist

• Run trivy image --reset
• Read the troubleshooting

[dgrezza]

Hi @DmitriyLewen ,

I encountered a similar issue in a different report section. The problem lies in vulnerabilities[].links[].url as some URLs do not conform to the pattern: ^(https?|ftp)://.+

example:

a temporary solution to mitigate this issue by appending https:// in front of the value that does not match the pattern using jq

jq '.vulnerabilities[].links[].url |= if test("^(https?|ftp)://.*") then . else "https://" + . end' ${CI_PROJECT_DIR}/gl-container-scanning-report.json

I hope this issue can be fixed in the next version of gitlab.tpl , Thanks 😉

[DmitriyLewen]

Hello @dgrezza

I created #6348 with update of gitlab.tpl.
Can you download template from this PR and check it out?
It would be great if you could find any missing issues or confirm that these changes resolve the issue.

Regards, Dmitriy