Reducing GitHub API calls to scale scanning repositories

[naveensrinivasan]

The GitHub API calls are throttled which makes it hard to scale the number of repositories to scan and provide results.

The code would have to wait for tens of minutes before continuing
{"level":"warn","ts":1613869247.8747272,"caller":"roundtripper/roundtripper.go:139","msg":"Rate limit exceeded. Waiting 44m34.125286853s to retry..."}

Scorecard checks for these don't need GitHub API, it requires a Git API

1. Active
2. Frozen-Deps
3. CodeQLInCheckDefinitions
4. Security-Policy
5. Packaging

Potential solution

1. Clone the Git Repo
2. Git pull on these repo's on a cron - to get the updates
3. Use an API to query these repositories directly instead of the GitHub

The https://github.com/go-git/go-git project provides an API on Git which could be used for avoiding the GitHub API limitations.

With httpcache #80 (comment) and reducing the number of GitHub API calls, we should be able to scale the scanning number of repositoreis.

related to #80

[naveensrinivasan]

Also, this approach could be used to Scan not GitHub repositories like Gitlab which helps one API irrespective of the provider.

[inferno-chromium]

Cloning might be too heavy weight for some big repos, and slow too.

Maybe lets start with httpcache first. We can also scale the number of github tokens. right now, we have 2, we can easily go into 4-5 (seperated with comma).

[naveensrinivasan]

Cloning might be too heavy weight for some big repos, and slow too.

Maybe lets start with httpcache first. We can also scale the number of github tokens. right now, we have 2, we can easily go into 4-5 (seperated with comma).

Cloning can be async as another cron job and it is a one-time effort. Cloning should not be run as part of scorecard , probably give an additional option to look at a location for cached git repo's if not fetch them from github.com.

[inferno-chromium]

Cloning might be too heavy weight for some big repos, and slow too.
Maybe lets start with httpcache first. We can also scale the number of github tokens. right now, we have 2, we can easily go into 4-5 (seperated with comma).

Cloning can be async as another cron job and it is a one-time effort. Cloning should not be run as part of scorecard , probably give an additional option to look at a location for cached git repo's if not fetch them from github.com.

Makes sense.

[azeemshaikh38]

Closing this since we are already tracking this here: #318