✨ Automatic dependency update checks #322

naveensrinivasan · 2021-04-02T18:45:05Z

Please check if the PR fulfills these requirements

Tests for the changes have been added (for bug fixes / features)
PR title follows the guidelines defined in https://github.com/ossf/scorecard/blob/main/CONTRIBUTING.md#pr-process

What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)
Feature
What is the current behavior? (You can also link to an open issue here)
New check - Use of tools that update deps like dependabot #179
What is the new behavior (if this is a feature change)?
- Checks if the dependencies are automatically updated.
Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)
None
Other information:
None

github-actions · 2021-04-02T19:00:23Z

Integration tests cancelled for e5bc9e51134e8d0a4f5599645b4d1f9ddf6bdb68

github-actions · 2021-04-02T19:04:26Z

Integration tests success for 13145736ba4fae9911e1d455739679ac6b6a70ec

checks.md

checks/automatic_dependency_update.go

inferno-chromium · 2021-04-03T08:27:46Z

checks.md

+
+**Remediation steps**:
+- Enable dependabot for the project and have a dependabot.yml file in the repository. 
+


This is an amazing check, but there are two problems:

Dependabot can be enabled on org level

Dependabot can be enabled via UI
and this is substantial portion of things, and we would show incorrect result for these. maybe we should bring some of these problems with github api in next openssf sync, e.g. branch protection, this one with knowing security settings like dependabot, etc. Landing this as-is will be problematic since we cannot get right status in many cases and scorecard needs to be accurate. thoughts?

This is an amazing check, but there are two problems:

Dependabot can be enabled on org level

Dependabot can be enabled via UI
and this is substantial portion of things, and we would show incorrect result for these. maybe we should bring some of these problems with github api in next openssf sync, e.g. branch protection, this one with knowing security settings like dependabot, etc. Landing this as-is will be problematic since we cannot get right status in many cases and scorecard needs to be accurate. thoughts?

I agree this might be an issue. We can have in-depth discussion.

Also

But why not provide an results with lesser confidence, we can never be 100% in some of the cases.

Another way to solve this issues is to look for commits and check for Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> and check for signed commits

yes can be then

look for that file .github/dependabot.yml, if exists, pass with high confidence.

if failed, check for commits [we have to see how far we can go without burning quota, or instead see if we can get user dependabot events by api]. if that exists, pass with medium-high confidence, depending on last update?

otherwise fail with low confidence (like 3?)

thoughts, can also brainstorm in next meeting with @azeemshaikh38 . but even this is a good start i think.

Sounds good.

Some projects may have dependabot for tooling (let's say a primary C++ project with no dependabot, but has python deps that utilize dependabot).

I wonder if we should lower confidence for projects with unsupported languages

re: dependabot commits. If people have dependabot installed, we might still want to verify that they accept dependabot's PRs once in a while.

yes right, this might be a better or definitely needed part here, which is did any dependendabot cls land in a while.

naveensrinivasan · 2021-06-03T22:50:07Z

Going to start working on this.

laurentsimon · 2021-06-03T22:51:45Z

re: @asraa I agree confidence should be lowered for other languages in the repo that are not defined in the dependabot config file. We could list files and their extension: if any of these files are not tracked by the dependabot config, they should reduce the score. It'd be great to weight the scoring based on how much code is tracked or not - maybe by looking at number of lines for each lanaguages.

There's the language API that I initially thought could help, but it unfortunately only works at the repo level, ie we cannot specify folders of interest.

laurentsimon · 2021-06-03T22:52:58Z

@nathannaveen what's left for this PR to land?

naveensrinivasan · 2021-06-03T23:19:14Z

re: @asraa I agree confidence should be lowered for other languages in the repo that are not defined in the dependabot config file. We could list files and their extension: if any of these files are not tracked by the dependabot config, they should reduce the score. It'd be great to weight the scoring based on how much code is tracked or not - maybe by looking at number of lines for each lanaguages.

There's the language API that I initially thought could help, but it unfortunately only works at the repo level, ie we cannot specify folders of interest.

I think for the initial release I am not going to target that

naveensrinivasan · 2021-06-03T23:35:16Z

@nathannaveen what's left for this PR to land?

if failed, check for commits [we have to see how far we can go without burning quota, or instead see if we can get user dependabot events by api]. if that exists, pass with medium-high confidence, depending on last update?

☝️ This will burn the API quota's . AFAIK there aren't dependabot events by API.

otherwise fail with low confidence (like 3?)

For now, we can stay with fail with low confidence.

Thoughts?

inferno-chromium · 2021-06-03T23:46:22Z

@nathannaveen what's left for this PR to land?

if failed, check for commits [we have to see how far we can go without burning quota, or instead see if we can get user dependabot events by api]. if that exists, pass with medium-high confidence, depending on last update?

☝️ This will burn the API quota's . AFAIK there aren't dependabot events by API.

otherwise fail with low confidence (like 3?)

For now, we can stay with fail with low confidence.

Thoughts?

Ok this is fine, but please add renovatebot file check in this as well. renovatebot is extremely common, and either one of the two is fine. in documentation, we should say this check only looks if it is enabled and does not ensure that it is run and prs are merged (maybe tbd for future ?) ?? @azeemshaikh38 @laurentsimon thoughts?

laurentsimon · 2021-06-03T23:58:08Z

@nathannaveen what's left for this PR to land?

if failed, check for commits [we have to see how far we can go without burning quota, or instead see if we can get user dependabot events by api]. if that exists, pass with medium-high confidence, depending on last update?

☝️ This will burn the API quota's . AFAIK there aren't dependabot events by API.

otherwise fail with low confidence (like 3?)

For now, we can stay with fail with low confidence.

Thoughts?

@nathannaveen what's left for this PR to land?

if failed, check for commits [we have to see how far we can go without burning quota, or instead see if we can get user dependabot events by api]. if that exists, pass with medium-high confidence, depending on last update?

☝️ This will burn the API quota's . AFAIK there aren't dependabot events by API.

Please add a TODO in the code that we should also list commits even when we find the dependabot's config file (to ensure users actually update their dependencies).

re: quota: An alternative approach could be to run the cron jobs with some checks turned off. Within a few days we could run different checks so that, every week, we have all the checks. Do we actually need the cron job to run every single day? (It's ideal, but is it really necessary?)

otherwise fail with low confidence (like 3?)

For now, we can stay with fail with low confidence.

Thoughts?

I suppose that's the best we can do now. Note that "fail with low confidence" achieves little from a user's point of view. If I use scorecard and I get this result, it's almost like a noop: I have no idea how to interpret it. @asraa who tests scorecard for envoy, so she may comment.

naveensrinivasan · 2021-06-04T00:32:10Z

@nathannaveen what's left for this PR to land?

if failed, check for commits [we have to see how far we can go without burning quota, or instead see if we can get user dependabot events by api]. if that exists, pass with medium-high confidence, depending on last update?

☝️ This will burn the API quota's . AFAIK there aren't dependabot events by API.

otherwise fail with low confidence (like 3?)

For now, we can stay with fail with low confidence.
Thoughts?

@nathannaveen what's left for this PR to land?

if failed, check for commits [we have to see how far we can go without burning quota, or instead see if we can get user dependabot events by api]. if that exists, pass with medium-high confidence, depending on last update?

☝️ This will burn the API quota's . AFAIK there aren't dependabot events by API.

Please add a TODO in the code that we should also list commits even when we find the dependabot's config file (to ensure users actually update their dependencies).

re: quota: An alternative approach could be to run the cron jobs with some checks turned off. Within a few days we could run different checks so that, every week, we have all the checks. Do we actually need the cron job to run every single day? (It's ideal, but is it really necessary?)

☝️ This is would be a separate issue

otherwise fail with low confidence (like 3?)

For now, we can stay with fail with low confidence.
Thoughts?

I suppose that's the best we can do now. Note that "fail with low confidence" achieves little from a user's point of view. If I use scorecard and I get this result, it's almost like a noop: I have no idea how to interpret it. @asraa who tests scorecard for envoy, so she may comment.

github-actions · 2021-06-04T03:06:19Z

Integration tests success for 63a9a8f116fc370f064f921a0f9e952d18527c08

github-actions · 2021-06-04T03:06:29Z

Integration tests success for f6f2601fb228d22b8bb547751d26ac87b1f23555

github-actions · 2021-06-04T03:12:01Z

Integration tests success for da88985558645d4952cbfdea518c1eadff91fc04

github-actions · 2021-06-04T03:18:14Z

Integration tests success for ad070b619aed860c27bf9e3b7a6161fcc00a2758

github-actions · 2021-06-04T03:37:23Z

Integration tests success for ea5cb0b8e49f299daf1f93cbfc8ab43ee1e39caf

inferno-chromium

Thanks a lot!

checks/automatic_dependency_update.go

inferno-chromium · 2021-06-04T04:18:34Z

checks/automatic_dependency_update.go

+
+// AutomaticDependencyUpdate will check the repository if it contains Automatic dependency update.
+func AutomaticDependencyUpdate(c *checker.CheckRequest) checker.CheckResult {
+	result := CheckIfFileExists(check, c, fileExists)


We need to later improve CheckIfFileExists so that it caches content and does not download again if it already exists. We have too many checks now that depend on files. @azeemshaikh38 fyi

Commented on the issue. This is definitely an improvement we should work on, but a low priority IMO.

checks/checks.md

github-actions · 2021-06-04T14:12:11Z

Integration tests success for 0eabcdcbd113d18b41d75a94c289f639b3658eb2

github-actions · 2021-06-04T14:13:42Z

Integration tests success for a3db8a4f80c8a76c02b8e89f2747f219bce21903

* Checks if the dependencies are automatically updated.

github-actions · 2021-06-04T14:20:44Z

Integration tests success for 71724aded09154d1ed49849996519a62be12142b

github-actions · 2021-06-04T14:34:39Z

Integration tests success for da27f3b562eaa992e88a3283c4b0934e6eb1e3b5

naveensrinivasan added core feature kind/enhancement New feature or request labels Apr 2, 2021

naveensrinivasan self-assigned this Apr 2, 2021

naveensrinivasan requested a review from inferno-chromium April 2, 2021 18:45

inferno-chromium reviewed Apr 3, 2021

View reviewed changes

naveensrinivasan mentioned this pull request May 11, 2021

New check: check the repo uses dependabot #431

Closed

laurentsimon added this to the milestone-q2 milestone May 11, 2021

naveensrinivasan force-pushed the feat/dependabot branch from e5bc9e5 to f6f2601 Compare June 4, 2021 02:58

naveensrinivasan requested review from inferno-chromium and azeemshaikh38 June 4, 2021 02:59

naveensrinivasan force-pushed the feat/dependabot branch from da88985 to ad070b6 Compare June 4, 2021 03:11

inferno-chromium approved these changes Jun 4, 2021

View reviewed changes

naveensrinivasan mentioned this pull request Jun 4, 2021

Feature - Handle Renovate bot package.json #542

Closed

naveensrinivasan force-pushed the feat/dependabot branch from ea5cb0b to 0eabcdc Compare June 4, 2021 14:04

✨ Automatic dependency update checks

71724ad

* Checks if the dependencies are automatically updated.

naveensrinivasan force-pushed the feat/dependabot branch from a3db8a4 to 71724ad Compare June 4, 2021 14:14

Merge branch 'main' into feat/dependabot

da27f3b

naveensrinivasan enabled auto-merge (squash) June 4, 2021 14:29

naveensrinivasan merged commit d00dd9c into main Jun 4, 2021

naveensrinivasan deleted the feat/dependabot branch June 4, 2021 14:35

naveensrinivasan mentioned this pull request Jun 4, 2021

New check - Use of tools that update deps like dependabot #179

Closed

naveensrinivasan mentioned this pull request Jul 1, 2021

BUG: ZeroConfidenceError and RetryError in cron job #629

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ Automatic dependency update checks #322

✨ Automatic dependency update checks #322

naveensrinivasan commented Apr 2, 2021

github-actions bot commented Apr 2, 2021

github-actions bot commented Apr 2, 2021

inferno-chromium Apr 3, 2021

naveensrinivasan Apr 3, 2021

inferno-chromium Apr 3, 2021

naveensrinivasan Apr 3, 2021

asraa Apr 5, 2021

laurentsimon Jun 3, 2021

inferno-chromium Jun 3, 2021

naveensrinivasan commented Jun 3, 2021

laurentsimon commented Jun 3, 2021

laurentsimon commented Jun 3, 2021

naveensrinivasan commented Jun 3, 2021

naveensrinivasan commented Jun 3, 2021 •

edited

Loading

inferno-chromium commented Jun 3, 2021 •

edited

Loading

laurentsimon commented Jun 3, 2021

naveensrinivasan commented Jun 4, 2021

github-actions bot commented Jun 4, 2021

github-actions bot commented Jun 4, 2021

github-actions bot commented Jun 4, 2021

github-actions bot commented Jun 4, 2021

github-actions bot commented Jun 4, 2021

inferno-chromium left a comment

inferno-chromium Jun 4, 2021

naveensrinivasan Jun 4, 2021

azeemshaikh38 Jun 4, 2021

github-actions bot commented Jun 4, 2021

github-actions bot commented Jun 4, 2021

github-actions bot commented Jun 4, 2021

github-actions bot commented Jun 4, 2021


		Remediation steps:
		- Enable dependabot for the project and have a dependabot.yml file in the repository.

✨ Automatic dependency update checks #322

✨ Automatic dependency update checks #322

Conversation

naveensrinivasan commented Apr 2, 2021

github-actions bot commented Apr 2, 2021

github-actions bot commented Apr 2, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

naveensrinivasan commented Jun 3, 2021

laurentsimon commented Jun 3, 2021

laurentsimon commented Jun 3, 2021

naveensrinivasan commented Jun 3, 2021

naveensrinivasan commented Jun 3, 2021 • edited Loading

inferno-chromium commented Jun 3, 2021 • edited Loading

laurentsimon commented Jun 3, 2021

naveensrinivasan commented Jun 4, 2021

github-actions bot commented Jun 4, 2021

github-actions bot commented Jun 4, 2021

github-actions bot commented Jun 4, 2021

github-actions bot commented Jun 4, 2021

github-actions bot commented Jun 4, 2021

inferno-chromium left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jun 4, 2021

github-actions bot commented Jun 4, 2021

github-actions bot commented Jun 4, 2021

github-actions bot commented Jun 4, 2021

naveensrinivasan commented Jun 3, 2021 •

edited

Loading

inferno-chromium commented Jun 3, 2021 •

edited

Loading