feat: populate metadata.component.externalReferences VCS and build-system from common CI environment variables

[jeremylong]

Is your feature request related to a problem? Please describe.

To improve traceability of a given SBOM, it would be fantastic if the metadata.component.externalReferences were populated for the vcs and build-system URLs. These can be obtained via well-known environment variables in many CI systems.

Describe the solution you'd like

The _helper.ts could be extended with code to detect the build-system and VCS from common CI environment variables:

import { execSync } from 'child_process'

...

export function detectBuildUrl (): string | undefined {
  if (process.env.GITHUB_ACTIONS === 'true') {
    return `${process.env.GITHUB_SERVER_URL}/${process.env.GITHUB_REPOSITORY}/actions/runs/${process.env.GITHUB_RUN_ID}`
  }
  if (process.env.GITLAB_CI === 'true' && isNonNullable(process.env.CI_JOB_URL)) {
    return process.env.CI_JOB_URL
  }
  if (isNonNullable(process.env.CIRCLECI)) {
    return process.env.CIRCLE_BUILD_URL
  }
  if (isNonNullable(process.env.JENKINS_URL)) {
    return process.env.BUILD_URL
  }
  if (isNonNullable(process.env.TF_BUILD)) {
    return `${process.env.SYSTEM_TEAMFOUNDATIONCOLLECTIONURI}${process.env.SYSTEM_TEAMPROJECT}/_build/results?buildId=${process.env.BUILD_BUILDID}`
  }
  if (isNonNullable(process.env.TRAVIS)) {
    return process.env.TRAVIS_BUILD_WEB_URL
  }
  if (isNonNullable(process.env.BITBUCKET_BUILD_NUMBER)) {
    return process.env.BITBUCKET_GIT_HTTP_ORIGIN
  }
  if (isNonNullable(process.env.CODEBUILD_PUBLIC_BUILD_URL)) {
    return process.env.CODEBUILD_PUBLIC_BUILD_URL
  }
  if (isNonNullable(process.env.DRONE_BUILD_LINK)) {
    return process.env.DRONE_BUILD_LINK
  }
  return undefined
}

export function detectSourceUrl(): string | undefined {
  try {
    const hasGit = execSync('which git', { stdio: 'ignore' })
    if (hasGit !== null && hasGit.length > 0) {
      const gitUrl = execSync('git remote get-url origin 2>/dev/null', { encoding: 'utf8' }).trim()
      if (gitUrl !== null && gitUrl !== '') {
        if (gitUrl.startsWith('git@') && gitUrl.endsWith('.git')) {
          return gitUrl.replace(':', '/').replace('git@', 'https://')
        }
        return gitUrl
      }
    }
  } catch (error) {
    // Fall through to environment checks if git commands fail
  }

  if (isNonNullable(process.env.GITHUB_REPOSITORY)) {
    return `${process.env.GITHUB_SERVER_URL}/${process.env.GITHUB_REPOSITORY}`
  }
  if (isNonNullable(process.env.GITLAB_CI)) {
    return process.env.CI_REPOSITORY_URL
  }
  if (isNonNullable(process.env.CIRCLECI)) {
    return process.env.CIRCLE_REPOSITORY_URL
  }
  if (isNonNullable(process.env.JENKINS_URL)) {
    return process.env.GIT_URL
  }
  if (isNonNullable(process.env.TF_BUILD)) {
    return process.env.BUILD_REPOSITORY_URI
  }
  if (isNonNullable(process.env.BITBUCKET_GIT_HTTP_ORIGIN)) {
    return process.env.BITBUCKET_GIT_HTTP_ORIGIN
  }
  if (isNonNullable(process.env.BITBUCKET_GIT_SSH_ORIGIN)) {
    return process.env.BITBUCKET_GIT_SSH_ORIGIN
  }
  if (isNonNullable(process.env.CODEBUILD_BUILD_ID)) {
    return process.env.CODEBUILD_SOURCE_REPO_URL
  }
  if (isNonNullable(process.env.DRONE_REPO_LINK)) {
    return process.env.DRONE_REPO_LINK
  }
  return undefined
}

Then plugin.ts#makeRootComponent could be updated to end with the following instead of just return builder.makeComponent(thisPackageJson):

const component = builder.makeComponent(thisPackageJson)

    if (component !== undefined) {
      const sourceUrl = detectBuildUrl()
      if (sourceUrl !== undefined) {
        component.externalReferences.add({
          type: CDX.Enums.ExternalReferenceType.VCS,
          url: sourceUrl,
          hashes: new CDX.Models.HashDictionary(),
          comment: '',
          compare: () => 0
        })
      }

      const buildUrl = detectSourceUrl()
      if (buildUrl !== undefined) {
        component.externalReferences.add({
          type: CDX.Enums.ExternalReferenceType.BuildSystem,
          url: buildUrl,
          hashes: new CDX.Models.HashDictionary(),
          comment: '',
          compare: () => 0
        })
      }
    }

    return component

Describe alternatives you've considered

An alternative would be to allow these values to be specified as configuration options.

Additional context

• Cyclonedx-cocoapods implementation via parameters: feat: add vcs and build-system options to component cyclonedx-cocoapods#82
• Cyclonedx-gradle-plugin build-system implementation via environment variables: feat: retrieve build-system metadata from build environment cyclonedx-gradle-plugin#546
• Cyclonedx-gradle-plugin VCS implementation via a call to git: feat: add git external reference. cyclonedx-gradle-plugin#520

I'm more than happy to put in a PR with these changes - but I know the code is in flux with the pending 4.0.0 release and some of this may change when the project upgrades to the cyclonedx-library@7.0.0.

[jkowalleck]

Regarding the proposed implementation:
TBH i do not want to maintain a list of possible env vars for some CI or build engines.
Therefore, the proposed implementation that depends on them is not desirable.

Furthermore, I don't like the shell-out to git - it is error-prone.
Therefore, the proposed implementation for that is not desirable.

For both cases, I'd rather go with the alternative you've proposed:

An alternative would be to allow these values to be specified as configuration options.

for each, this could be a config setting (rootComponentVCS and rootComponentBuildSystem) that defaults to a value provided by well-known environment variables (CDX_WEBPACK_rootComponentVCS and CDX_WEBPACK_rootComponentBuildSystem).

PS: after thinking about this for a while, I actually do not see any reason to have these settings manipulable via an environment variable. The VCS and build-env should be known way before the actual build starts, and therefore it should be enough to configure them in a plugin. Would you agree?
If they still were dynamic, you could simply configure a well-known string, that you'd replace with sed -i 's#MY_BUILDENV_URL#https://be.example.com#' sbom.cdx.json later.

[jkowalleck]

I do have questions regarding the applied use case, still.

The current implementation is able to gather information from package manifests.
This also concerns $.metadata.component.

Currently, there is support for parsing npmjs/node's manifest (package.json).
This support might be extended to whatever package manager may apply (*.gemspec).

Do you maintain a package manifest for the thing you are building?

• If so, which one is it and does it have capabilities to store the relevant URLs?
• If not, could you explain what would hinder you to do so? (I mean, you need to get your dependencies managed somehow anyway, and this might involve - depending on the ecosystem - a package manifest anyway)

[jkowalleck]

this feature might be related to the propposal here: #675

[VinodAnandan]

@jeremylong thank you for this. What are your thoughts on including this information in the formulation section of the BOM? The formulation section could be utilised to describe how a component was manufactured. This could be achieved by leveraging formulas, workflows, tasks, and steps that outline the exact process for reproduction, alongside the observed formulas detailing the steps taken during the manufacturing process.

More information : https://cyclonedx.org/docs/1.6/json/#formulation

[jeremylong]

@VinodAnandan I don't see how something that would be a small part of the formulation (i.e., the webpack plugin itself) could reasonably produce the formulation SBOM. The formulation can cover the build server, all plugins installed on the build server, the build tool (e.g., maven, Gradle, npm), etc.

[jeremylong]

@jkowalleck regarding:

PS: after thinking about this for a while, I actually do not see any reason to have these settings manipulable via an environment variable. The VCS and build-env should be known way before the actual build starts, and therefore it should be enough to configure them in a plugin. Would you agree?

While the VCS would be known ahead of time, the build server would be known at build time. I have build servers for dev and uat (e.g. these are staging and testing for the build system itself and these should never produce a production artifact). As such, I want the actual build server that built the artifact listed to ensure something isn't messed up. Providing a configuration option to set the VCS and build-system would be a good approach for my use case.

I could work with your proposed rootComponentVCS and rootComponentBuildSystem. If that works - I'll create the PR.

[jkowalleck]

Providing a configuration option to set the VCS and build-system would be a good approach for my use case.

rootComponentVCS should not actually be needed as a config option, as it is an intended capability to to pull this information from package manifests. see #1344 (comment)

rootComponentBuildSystem as a config option:
Sure - go for it, please create a PR for this.
There should be no need for a default value pulled from any predefined environment variable.
If users wanted to populate the value dynamically, they could use replace a predetermined value (as described here), or thy could use a dynamic config right away, like so:

# file: webpack.config.js
const { CycloneDxWebpackPlugin } = require('@cyclonedx/webpack-plugin')

const cycloneDxWebpackPluginOptions = {
  rootComponentAutodetect: true,
  rootComponentType: 'application',
  rootComponentBuildSystem: process.env.MY_BUILDENV_URL
}

module.exports = {
  target: 'web',
  mode: 'production',
  entry: './src/index.js',
  plugins: [
    new CycloneDxWebpackPlugin(cycloneDxWebpackPluginOptions)
  ],
  stats: 'detailed'
}

[jeremylong]

rootComponentVCS should not actually be needed as a config option, as it is an intended capability to to pull this information from package manifests.

I disagree - I think a configurable option should still be available for rootComponentVCS. I would be concerned that the package manifest may not be updated correctly (think forks).

[jkowalleck]

rootComponentVCS should not actually be needed as a config option, as it is an intended capability to to pull this information from package manifests.

I disagree - I think a configurable option should still be available for rootComponentVCS. I would be concerned that the package manifest may not be updated correctly (think forks).

So you tell me there might be a fork out there, that has a unmaintained package manifest, but the build pipeline of that specific fork is properly maintained?

This whole SBOM generator is about the ability to produce accurate data. And the capability to do so already exists.
If people decide to not provide the needed information, then that is just out of scope.

No option for rootComponentVCS for now. (I do not see any reason to block the rootComponentBuildSystem feature by another disputable feature. let's call it a ticket spit by scope and clarity. the one part that is clear is going to be developer/merged, while the unclear part stars on the backlog.)

[jeremylong]

happens all the time when you fork public GH to GHE. If I want to build the package myself as part of a BYOB (Build your own Binary) program. I might not update the manifest files, but I would have a proper build that knows the internal GHE VCS.