"Signed-Releases" check scores above 10 leading to invalid scores

[sethmlarson]

See urllib3's current score for "Signed-Releases" of 40/10: https://deps.dev/pypi/urllib3, this also affects our total score which is above 10.

[spencerschrock]

Likely introduced by #3610, specifically #3610 (comment)

[pnacht]

FWIW, running the Scorecard CLI on the project returns a normal 10/10 for Signed-Releases. In fact, using the Scorecard Viewer, the score is normalized as well: https://securityscorecards.dev/viewer/?uri=github.com/urllib3/urllib3

The difference is that deps.dev uses the result from the weekly cronjob, which runs closer to HEAD and ran on Dec 25th, while the Scorecard viewer also accepts the results from your Scorecard Action, which runs an official release of Scorecard (not sure which version, precisely) and ran today.

[spencerschrock]

Note: the 2023.12.18 and 2023.12.25 runs are both affected, and so would the 2024.01.01 run if we let it finish.

[spencerschrock]

Among the weekly data, 79 / 1,244,612 repos (0.006%) have a Signed-Releases score greater than 10. Although the number of affected repos may be slightly higher, as this doesn't count the ones who have an inflated yet still valid score.

[spencerschrock]

Leaving this open while discussing options around deps.dev and BQ data. Although given the small number of repos affected, the answer may be to just wait.

[spencerschrock]

Closing since new data has propagated to deps.dev at this point.