-
Notifications
You must be signed in to change notification settings - Fork 474
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
✨ Feature: probe: unique code-reviewers #3265
Comments
Hi, I think having something like these three probes makes sense:
The logic for checking author login != reviewer login could look something like this:
Currently, for platforms outside of GitHub, we give full credit/points for Code Review, even though we just check the git commit message (which can be tampered with). In the future, we could invoke the Phabricator API to check whether the differential revision 1) exists and 2) was reviewed by the author listed in the commit message. We would need to rethink the logic here:
Scorecard has always trusted GitHub and the commit log, and doesn't go to huge lengths to validate the integrity of, say, committer identities. It makes sense to add structured results though, since changing Code-Review evaluation for existing repos that live on GitHub but don't use GitHub for code review would regress scoring somewhat confusingly. |
Thank you for your thorough input! I have the last probe you mentioned working, but I will work on the first and second as well. |
same question. I potentially see:
|
Stale issue message - this issue will be closed in 7 days |
This issue is stale because it has been open for 60 days with no activity. |
Is your feature request related to a problem? Please describe.
The Code-review check currently checks if there are unreviewed changesets and only counts reviewers whose logins != author login. If the logins can't be retrieved (ex. they were reviewed outside of GitHub) the check gets full points. This is an understandable approach until the scorecard can handle other review platforms, but not the safest.
Describe the solution you'd like
I am working on a probe that checks that:
A. the logins of both author and reviewers are retrieved(otherwise the results are inconclusive)
B. that there are at least two unique reviewers per changeset.
The amount of minimum unique reviewers wanted for a positive outcome can easily be changed in the impl.go file.
Describe alternatives you've considered
To update the scorecard's retrieval of raw data to support multiple review platforms, this would also be needed to utilize the probe's full potential. This, however, is another issue for another day. Not all users of the scorecard is interested in an aggregate score, but want more binary conditions that either pass or fail, hence the probe.
The text was updated successfully, but these errors were encountered: