Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: Runtime error on Pinned-Dependencies check causes a -1 on its score #3316

Closed
diogoteles08 opened this issue Jul 26, 2023 · 1 comment · Fixed by #3515
Closed

BUG: Runtime error on Pinned-Dependencies check causes a -1 on its score #3316

diogoteles08 opened this issue Jul 26, 2023 · 1 comment · Fixed by #3515
Labels

Comments

@diogoteles08
Copy link
Contributor

diogoteles08 commented Jul 26, 2023

Describe the bug
When running Pinned-Dependencies check on apache/arrow or apache/beam, it's returning a -1 and reporting an Internal Error. It's not the same error on both repository, but both errors are breaking the execution and disabling users to view any actual result that could have been achieved.

See above the errors in details:

diogoteles:~$ scorecard --repo=https://github.com/apache/beam --show-details --format=json --checks=Pinned-Dependencies | jq .
Error: check runtime error: Pinned-Dependencies: internal error: internal error: unable to determine OS for job: Build python wheels on ${{matrix.arch}} for ${{ matrix.os_python.os }}
2023/07/26 10:47:12 error during command execution: check runtime error: Pinned-Dependencies: internal error: internal error: unable to determine OS for job: Build python wheels on ${{matrix.arch}} for ${{ matrix.os_python.os }}
{
  "date": "2023-07-26T10:47:05-03:00",
  "repo": {
    "name": "github.com/apache/beam",
    "commit": "907c5110163b0efe52e9e12127fd013c7fc455d7"
  },
  "scorecard": {
    "version": "4.11.0",
    "commit": "4edb07802fdad892fa8d10f8fd47666b6ccc27c9"
  },
  "score": -1,
  "checks": [
    {
      "details": null,
      "score": -1,
      "reason": "internal error: internal error: unable to determine OS for job: Build python wheels on ${{matrix.arch}} for ${{ matrix.os_python.os }}",
      "name": "Pinned-Dependencies",
      "documentation": {
        "url": "https://github.com/ossf/scorecard/blob/4edb07802fdad892fa8d10f8fd47666b6ccc27c9/docs/checks.md#pinned-dependencies",
        "short": "Determines if the project has declared and pinned the dependencies of its build process."
      }
    }
  ],
  "metadata": null
}
diogoteles:~$ scorecard --repo=https://github.com/apache/arrow --show-details --format=json --checks=Pinned-Dependencies | jq .
Error: check runtime error: Pinned-Dependencies: internal error: error parsing shell code: ci/docker/python-wheel-windows-test-vs2017.dockerfile:1:2: "if <cond>" must be followed by "then"
2023/07/26 10:51:40 error during command execution: check runtime error: Pinned-Dependencies: internal error: error parsing shell code: ci/docker/python-wheel-windows-test-vs2017.dockerfile:1:2: "if <cond>" must be followed by "then"
{
  "date": "2023-07-26T10:51:35-03:00",
  "repo": {
    "name": "github.com/apache/arrow",
    "commit": "8503c869fa80090bc849bae0b4e68a84605ea82e"
  },
  "scorecard": {
    "version": "4.11.0",
    "commit": "4edb07802fdad892fa8d10f8fd47666b6ccc27c9"
  },
  "score": -1,
  "checks": [
    {
      "details": null,
      "score": -1,
      "reason": "internal error: error parsing shell code: ci/docker/python-wheel-windows-test-vs2017.dockerfile:1:2: \"if <cond>\" must be followed by \"then\"",
      "name": "Pinned-Dependencies",
      "documentation": {
        "url": "https://github.com/ossf/scorecard/blob/4edb07802fdad892fa8d10f8fd47666b6ccc27c9/docs/checks.md#pinned-dependencies",
        "short": "Determines if the project has declared and pinned the dependencies of its build process."
      }
    }
  ],
  "metadata": null
}

Reproduction steps
Steps to reproduce the behavior:

  1. Run Scorecard (I've reproduced it on version 4.11.0 and 4.10.2) with Pinned-Dependencies check for apache/arrow or apache/beam

Expected behavior

At first, we should consider enhancing the error handling to avoid not showing any results to the user other than the internal error. I'd say that ideally the internal error should not break the whole execution of the check; the output should report the error(and probably keep reporting a -1), but should also contain any results that could be achieved before and after the internal error.

Additionally, we should work to avoid those specific internal errors, of course.

@agilgur5
Copy link

We also get a runtime error when Scorecard runs this check on argoproj/argo-workflows. Mentioned in CNCF Slack for the current Security Slam.

It throws an error while trying to parse our Dockerfile.windows: internal error: error parsing shell code: Dockerfile.windows:1:5: (( can only be used to open an arithmetic cmd.
As far as I can tell, that seems to be because the shell parser is assuming Linux shell, but that code is actually Windows PowerShell.

#3515 successfully warns on that error and continues, outputting relevant information, even in the same file.

(and probably keep reporting a -1)

I think it should normalize an overall score, not just report a -1 if a single line gets a runtime error. Especially since the error is within Scorecard, and not within the project it's checking. Running #3515 on Argo Workflows gets a 4, which makes a lot more sense to me than a -1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants