Scorecard scalability limitation: Reduce GitHub API calls

[htuch]

I've been running scorecard over ~40 Envoy dependencies a few times in the past hour and even using a personal access token for GitHub, I've hit the rate limit. I'm wondering what can be done to make this work better in scorecard. Suggestions:

• Profile GitHub API calls, determine the most costly, optimize to API alternatives. Share details on which are most expensive in public documentation.
• Add support for local caching of results for some time bound period.
• Provide a scorecard network service that collects results for projects periodically.
• Have scorecard JSON examples generalized to some contributor editable file that will collect projects nightly.

[dlorenc]

• Provide a scorecard network service that collects results for projects periodically.

This is the approach we've taken so far. There's a cron that runs every day on ~100 projects and publishes the results in GCS/BigQuery.

I'd be happy to add the envoy deps into that list, it's here: https://github.com/ossf/scorecard/blob/main/cron/projects.txt

[dlorenc]

Adding the envoy deps here: #84

[naveensrinivasan]

GitHub API supports conditional requests https://docs.github.com/en/rest/overview/resources-in-the-rest-api#conditional-requests

Most responses return an ETag header. Many responses also return a Last-Modified header. You can use the values of these headers to make subsequent requests to those resources using the If-None-Match and If-Modified-Since headers, respectively. If the resource has not changed, the server will return a 304 Not Modified.
Making a conditional request and receiving a 304 response does not count against your Rate Limit, so we encourage you to use it whenever possible.

Making a conditional request and receiving a 304 response does not count against your Rate Limit, so we encourage you to use it whenever possible.

https://github.com/google/go-github supports Conditional requests https://github.com/google/go-github#conditional-requests

As we are scaling more and more projects this would add a lot of value.

https://github.com/gregjones/httpcache

cc @inferno-chromium

[naveensrinivasan]

A visual representation of the proposed solution.

1. k8s cron job runs on a schedule.
2. Initial run fetches information using httpcache as a middleware, which caches the HTTP response initially in a large disk (PVC), probably move to Redis later as a cache instead of disk.
3. Subsequent cron runs will utilize the httpcache for checking content modification and load it from the cache if it isn't modified, which reduces the hitting the Rate Limit of the GitHub API.

[dlorenc]

Awesome! What would the cache keys be? URLs?

[naveensrinivasan]

Awesome! What would the cache keys be? URLs?

Here it is.
https://github.com/gregjones/httpcache/blob/901d90724c7919163f472a9812253fb26761123d/httpcache.go#L42

[naveensrinivasan]

This should reduce the GitHub API usage #227

[azeemshaikh38]

Close this, since the solution is being tracked/implemented in #318