Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨ Add experimental check for published SBOM #3903

Merged
merged 21 commits into from
May 17, 2024

Conversation

ashearin
Copy link
Contributor

What kind of change does this PR introduce?

Adds check for published sboms. This PR is still a draft as there is more discussion on applicability and implementation to be had. This PR is meant to spur those conversations.

What is the current behavior?

N/A

What is the new behavior (if this is a feature change)?**

*Tests were not added as the implementation may change, tests will be added prior to merge.

Adds a multi probe check for a Software Bill of Materials for a scanned repository.

More information regarding implementation and considerations for this check can be found here

Which issue(s) this PR fixes

Fixes #3574, #1476

Special notes for your reviewer

Does this PR introduce a user-facing change?

For user-facing changes, please add a concise, human-readable release note to
the release-note

(In particular, describe what changes users might need to make in their
application as a result of this pull request.)


Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>
@ashearin ashearin requested a review from a team as a code owner February 27, 2024 00:11
@ashearin ashearin requested review from justaugustus and laurentsimon and removed request for a team February 27, 2024 00:11
@ashearin ashearin marked this pull request as draft February 27, 2024 00:11
@ashearin ashearin added the kind/enhancement New feature or request label Feb 27, 2024
Copy link

codecov bot commented Feb 27, 2024

Codecov Report

Merging #3903 (037eb1a) into main (d58bfb0) will decrease coverage by 4.67%.
Report is 1 commits behind head on main.
The diff coverage is 67.73%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3903      +/-   ##
==========================================
- Coverage   74.94%   70.28%   -4.67%     
==========================================
  Files         223      232       +9     
  Lines       16046    16723     +677     
==========================================
- Hits        12026    11754     -272     
- Misses       3253     4230     +977     
+ Partials      767      739      -28     

@ashearin ashearin changed the title Draft: ⚠️ Add Check for published SBOM ⚠️ Add Check for published SBOM Feb 27, 2024
@ashearin ashearin changed the title ⚠️ Add Check for published SBOM ⚠️ Add Check for published SBOM Feb 27, 2024
@ashearin ashearin self-assigned this Feb 27, 2024
@ashearin ashearin changed the title ⚠️ Add Check for published SBOM ✨ Add Check for published SBOM Feb 27, 2024
Copy link
Contributor

@laurentsimon laurentsimon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this PR!

A few questions:

  1. Should the probe be aware of cases where there's no release assets? For example, a GitHub release may have no assets, if it's only source code release. Is an SBOM needed in this case? Note that the release could contain a container release as a GitHub package or something else. We could (maybe?) update the code to detect the former case, but the latter seems much harder.
  2. How do we determine if the project should create an SBOM or not, depending on the type of release (application, library, ?) - see https://blog.deps.dev/zillions-of-sboms/. Is there a document describing when an SBOM makes sense in each ecosystem?
  3. Remediation depends on point (2) above. We need a good story to tell folks how to remediate. Ideally SBOMs should come for free out of the package managers, which can build enough context to decide if an SBOM is needed or not (or always generate an empty SBOM?). @steiza @di @woodruffw @JoelMarcey is this something you intend to build for npm / pypi / homebrew / rust?
  4. Since GH has an API to generate SBOMs, why should maintainers generate an SBOM themselves?

Somewhat related but broader questions: Are registries interested in accepting SBOMs for applications. @steiza @di @woodruffw @JoelMarcey how are you thinking about this from npm / pypi / homebrew / rust side?

checker/raw_result.go Outdated Show resolved Hide resolved
clients/githubrepo/tarball.go Outdated Show resolved Hide resolved
probes/sbomCICDArtifactExists/impl.go Outdated Show resolved Hide resolved
clients/gitlabrepo/sbom.go Outdated Show resolved Hide resolved
probes/sbomCICDArtifactExists/impl.go Outdated Show resolved Hide resolved
@ashearin
Copy link
Contributor Author

ashearin commented Feb 27, 2024

@laurentsimon

  1. Should the probe be aware of cases where there's no release assets? For example, a GitHub release may have no assets, if it's only source code release. Is an SBOM needed in this case? Note that the release could contain a container release as a GitHub package or something else. We could (maybe?) update the code to detect the former case, but the latter seems much harder.

Both clients as written handle the possibility that there are no release assets, in which case it just continues on with the remainder of the check. For a source only release I'd expect an associated SBOM, as ideally each released version would have an associated SBOM for downstream users (assuming the project should be creating an SBOM).

For releases that include a container, we could potentially check for an SBOM layer in the container, but the effort involved in that may not be worth it. This would be a situation where the workflow that builds the container in preparation for release would ideally also generate an SBOM for it and save it as an artifact (at the very least) to be picked up by our checks, and to be available for downstream users.

  1. How do we determine if the project should create an SBOM or not, depending on the type of release (application, library, ?) - see https://blog.deps.dev/zillions-of-sboms/. Is there a document describing when an SBOM makes sense in each ecosystem?

I'm not sure about this. I know applicability is one of the sticking points for this check in general. There was talk of a Maintainers Annotation feature which could help here, but I'm not sure the status of that.

I'm unaware of any such document describing when an SBOM makes sense in each ecosystem. @idunbarh do you have any insight on this?

  1. Since GH has an API to generate SBOMs, why should maintainers generate an SBOM themselves?

This I actually forgot to mention in the notes doc I posted, I have since added it. I do hit the SBOM api endpoint for Github repos and, assuming a good response, count that generated SBOM as a release artifact and award appropriate points. As far as generating an SBOM themselves in GH repos, I would leave it up to the maintainers to decide if that is necessary for their project. i.e. if they need to generate one for a release container or in another format. I'm hesitant to award full points for the endpoint BOM, as it may not be comprehensive. I haven't seen anything from GL yet, but would expect to see a similar SBOM endpoint eventually.

@idunbarh
Copy link

I'm unaware of any such document describing when an SBOM makes sense in each ecosystem. @idunbarh do you have any insight on this?

The I don't believe the OpenSSF SBOM Everywhere SIG has provided any guidance on what ecosystems should or should not generate SBOMs. The SBOM Naming Convention Doc would be the ideal place for clarification.

@idunbarh
Copy link

ossf/sbom-everywhere#44

Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>
Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>
@laurentsimon
Copy link
Contributor

I'm unaware of any such document describing when an SBOM makes sense in each ecosystem. @idunbarh do you have any insight on this?

The I don't believe the OpenSSF SBOM Everywhere SIG has provided any guidance on what ecosystems should or should not generate SBOMs. The SBOM Naming Convention Doc would be the ideal place for clarification.

Yeah we really need a doc on this, per ecosystem, per-use case. What we found in https://blog.deps.dev/zillions-of-sboms/, is that most open-source projects are libraries, so often times an SBOM is not needed (there are nuances, of course). Keep me in the loop if this works starts.

@laurentsimon
Copy link
Contributor

laurentsimon commented Feb 28, 2024

I'm hesitant to award full points for the endpoint BOM, as it may not be comprehensive

For a first integration, one possibility could be to not include these probes in a default check. Users who want to detect the presence of SBOMs in a repo could then run the probes / checks they care about, using the context and knowledge they have about the repo. For example, organizations (@UlisesGascon @netomi) could use their own policy to decide whether to run this probe or not on their projects.

For arbitrary projects, I think we need more investigation to reduce noise (false positives, score decrease)

@pnacht
Copy link
Contributor

pnacht commented Feb 28, 2024

Yeah we really need a doc on this, per ecosystem, per-use case. What we found in https://blog.deps.dev/zillions-of-sboms/, is that most open-source projects are libraries, so often times an SBOM is not needed (there are nuances, of course).

Yeah, I was going to bring this up. Most projects that'll ever be handled by Scorecard likely don't need an SBOM (simple libraries).

For a first integration, one possibility could be to not include these probes in a default check.

Alternatively, make this a "bonus points" check: 10/10 if the latest release has an SBOM, $INCONCLUSIVE otherwise.

@idunbarh
Copy link

Coming from the Security Tooling WG discussions, one of the desired outcomes is to measure the impact other parts of OpenSSF are having around SBOM adoption. In this case the Security Tooling WG's SBOM Naming Conventions and future SBOM Strike Force that will be working directly with large projects to implement SBOMs. I see Scorecard as an mechanism to help adoption and also measure adoption.

I'm all for making it a bonus points check over a check that would be disabled by default.

Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>
@ashearin ashearin marked this pull request as ready for review May 14, 2024 22:05
Copy link
Member

@spencerschrock spencerschrock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR is in a generally good spot. Please address the latest review comments, and then we can merge

checker/raw_result.go Outdated Show resolved Hide resolved
checks/evaluation/sbom.go Outdated Show resolved Hide resolved
checks/evaluation/sbom.go Show resolved Hide resolved
checks/raw/sbom.go Outdated Show resolved Hide resolved
checks/raw/sbom.go Outdated Show resolved Hide resolved
checks/sbom_test.go Outdated Show resolved Hide resolved
checks/sbom_test.go Outdated Show resolved Hide resolved
checks/sbom_test.go Outdated Show resolved Hide resolved
docs/checks/internal/checks.yaml Outdated Show resolved Hide resolved
probes/hasReleaseSBOM/impl.go Outdated Show resolved Hide resolved
@spencerschrock
Copy link
Member

After trying on github.com/bomctl/bomctl, the output can be pretty verbose. Maybe we should limit release lookback to 5 most recent similar to signed releases?

Click to expand
SCORECARD_EXPERIMENTAL=1 go run main.go --repo github.com/bomctl/bomctl --format json --show-details --checks sbom 
| jq
{
  "date": "2024-05-15T12:22:25-07:00",
  "repo": {
    "name": "github.com/bomctl/bomctl",
    "commit": "4ef4b33ce10500a853403066e335a1d48e838b11"
  },
  "scorecard": {
    "version": "devel",
    "commit": "unknown"
  },
  "score": 10.0,
  "checks": [
    {
      "details": [
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175727:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175726:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175730:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175729:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175723:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175722:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175721:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175720:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175725:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175724:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175719:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175718:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162321:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162322:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162317:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162316:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162323:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162324:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162325:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162327:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162320:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162319:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162315:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162314:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865640:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865638:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865631:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865635:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865637:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865636:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865629:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865630:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865644:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865643:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865642:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865641:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457982:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457986:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457981:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457983:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457989:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457990:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457978:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457979:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457987:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457988:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457975:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457977:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454378:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454379:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454371:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454372:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454381:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454380:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454375:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454376:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454387:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454384:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454383:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454382:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163283690:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163283686:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163283691:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163283688:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163283689:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163283687:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163123197:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163123201:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163123198:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163123200:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163123196:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163123195:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163122640:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163122637:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163122638:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163122636:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163122641:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163122635:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/162076784:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/162076780:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/162076782:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/162076783:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/162076781:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/162076778:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/153684271:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/153684275:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/153684270:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/153684272:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/153684269:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/153684273:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175727:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175726:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175730:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175729:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175723:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175722:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175721:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175720:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175725:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175724:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175719:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175718:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162321:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162322:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162317:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162316:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162323:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162324:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162325:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162327:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162320:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162319:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162315:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162314:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865640:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865638:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865631:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865635:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865637:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865636:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865629:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865630:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865644:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865643:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865642:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865641:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457982:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457986:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457981:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457983:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457989:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457990:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457978:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457979:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457987:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457988:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457975:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457977:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454378:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454379:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454371:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454372:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454381:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454380:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454375:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454376:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454387:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454384:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454383:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454382:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163283690:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163283686:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163283691:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163283688:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163283689:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163283687:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163123197:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163123201:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163123198:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163123200:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163123196:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163123195:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163122640:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163122637:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163122638:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163122636:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163122641:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163122635:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/162076784:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/162076780:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/162076782:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/162076783:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/162076781:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/162076778:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/153684271:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/153684275:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/153684270:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/153684272:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/153684269:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/153684273:0"
      ],
      "score": 10,
      "reason": "SBOM file found in release artifacts",
      "name": "SBOM",
      "documentation": {
        "url": "https://github.com/ossf/scorecard/blob/main/docs/checks.md#sbom",
        "short": "Determines if the project maintains a Software Bill of Materials."
      }
    }
  ],
  "metadata": null
}

Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>
Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>
@ashearin
Copy link
Contributor Author

ashearin commented May 16, 2024

Same cmd with latest update. I changed it to check the 5 latest releases and only iterate over release assets until a match was found. So puts a cap on release sboms returned to 5

$ SCORECARD_EXPERIMENTAL=1 go run main.go --repo github.com/bomctl/bomctl --format json --show-details --checks sbom | jq
{
  "date": "2024-05-15T18:32:00-06:00",
  "repo": {
    "name": "github.com/bomctl/bomctl",
    "commit": "4ef4b33ce10500a853403066e335a1d48e838b11"
  },
  "scorecard": {
    "version": "devel",
    "commit": "unknown"
  },
  "score": 10.0,
  "checks": [
    {
      "details": [
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175727:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162321:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865640:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457982:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454378:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175727:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162321:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865640:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457982:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454378:0"
      ],
      "score": 10,
      "reason": "SBOM file found in release artifacts",
      "name": "SBOM",
      "documentation": {
        "url": "https://github.com/ossf/scorecard/blob/main/docs/checks.md#sbom",
        "short": "Determines if the project maintains a Software Bill of Materials."
      }
    }
  ],
  "metadata": null
}

@evverx
Copy link
Contributor

evverx commented May 16, 2024

gating the check behind experimental flag

Can I ask where this is going to be used? I suspect half-baked SBOMs created to make scorecard happy combined with bogus vulnerabilities like google/osv.dev#2176 are going to waste even more maintainers' time

This PR is still a draft as there is more discussion on applicability and implementation to be had

Given that it's no longer a draft I assume it was discussed and okayed somewhere. Is it possible to take a look at those discussions anywhere?

@spencerschrock
Copy link
Member

Can I ask where this is going to be used? I suspect half-baked SBOMs created to make scorecard happy combined with bogus vulnerabilities like google/osv.dev#2176 are going to waste even more maintainers' time

by people who run the scorecard binary on a repo and explicitly mention they want the SBOM check/probe.

This PR is still a draft as there is more discussion on applicability and implementation to be had

Given that it's no longer a draft I assume it was discussed and okayed somewhere. Is it possible to take a look at those discussions anywhere?

This was discussed during OSS NA 2024, there was some talk about it in the OpenSSF Slack as well. I've included some of the relevant Slack discussion below since slack history will hide it soon:

slack conversation

Dana Wang:
I met with Spencer Schrock, Laurent Simon, Ian Dunbar-Hall, Allan Friedman, Ryan Ware, justaugustus today on adding a non-opinionated check to Scorecard to report whether a project produces SBOM as the first "gold star" (Ian Dunbar-Hall owns the copyright) to report the existence of SBOM, it helps to gain visibility into the SBOM state, and put zero burden on maintainers, I feel it's a good balance on pushing the boundary on SBOM which is coming with acceleration, and protecting our maintainers. Ryan Ware has a great idea on how the Scorecard score could be tiered/applied to a project to incrementally to enhance the security posture of an open source project while protecting our maintainers from being overwhelmed with security burdens.

Brandon Mitchell:
I find myself triggered by the phrase "zero burden on maintainers". I really don't see how that's possible. Is there an example of a public project where this has been done without any maintainer involvement?

Spencer Schrock
For context, we're currently envisioning this as something that lives as a probe, not in a check. So it would not be a check in the sense of a Scorecard check, but rather as a binary "is there an SBOM".
So there would be no score attached in the normal output / score, just a "is there an SBOM?" "yes/no" when using the --format probe option.

Spencer Schrock
We're hoping this allows individual consumers to find out what their dependencies are doing, and if it matches their policy. And it may not, so the end result is "company X shouldn't use this dependency.", not that project Y should start doing Z for the benefit of company X

The consensus wasn't around including it as a default check, but rather getting some code in there for the purpose of:

  1. Individuals being able to run the analysis if they want it (either through --checks SBOM or the corresponding --probes hasSBOM,hasReleaseSBOM)
  2. Possibly running it as part of the weekly cron for the purpose of seeing how widely adopted SBOMs are currently. Not for the purpose of assigning a score.

As you can guess, the "default" aspect of Scorecard is the controversial part. This applies to other checks/probes as well. For example some people care about permissive license (#3838), but there's nothing inherently right or wrong with permissive vs copyleft vs other.

I've written more on this topic (which I'll be sharing "soon"), but ultimately Scorecard is a linter and should support any relevant analyses and move away from all of them being in the default output, but rather used through the enabling/disabling with individual configuration. Which lets different audiences use the tool in the manner they want without anything forced upon them.

@evverx
Copy link
Contributor

evverx commented May 16, 2024

Got it. Thank you!

people who run the scorecard binary on a repo and explicitly mention they want the SBOM check/probe

I get that part but in #3574 (and in some other issues related to SBOMs) I tried to figure out what the use cases were and I haven't seen any actual use cases anywhere.

Individuals being able to run the analysis if they want it

I wonder why? I assume this data is supposed to be acted on.

Not for the purpose of assigning a score

That's reassuring but I'm not talking about scores. I'm kind of concerned that things are seemingly getting adopted with no clear purpose and combined with OSV it offloads a lot of things onto upstream projects without any scores involved.

ultimately Scorecard is a linter and should support any relevant analyses and move away from all of them being in the default output

I agree with that.

@spencerschrock spencerschrock changed the title ✨ Add Check for published SBOM ✨ Add experimental check for published SBOM May 17, 2024
@spencerschrock spencerschrock enabled auto-merge (squash) May 17, 2024 18:08
@spencerschrock spencerschrock merged commit 8de9020 into ossf:main May 17, 2024
36 checks passed
@idunbarh
Copy link

🥳 Thanks @spencerschrock for all of the guidance and feedback through this PR!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/new-check New check for scorecard needs discussion
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

Add check for published sboms
9 participants