BUG: Failures in Branch-Protection during release testing

[azeemshaikh38]

Release tests found some issues with Branch-Protection check. Looks like related to recent scoring PR. @laurentsimon assigning to you for further debugging.

./scorecard --repo=github.com/kubermatic/dashboard-v2 --checks=Branhc-Protection
./scorecard --repo=github.com/xwp/stream --checks=Branch-Protection
./scorecard --repo=github.com/americanexpress/one-app-cli --checks=Branch-Protection

[laurentsimon]

Release tests found some issues with Branch-Protection check. Looks like related to recent scoring PR. @laurentsimon assigning to you for further debugging.

./scorecard --repo=github.com/kubermatic/dashboard-v2 --checks=Branhc-Protection

"reason": "could not find branch name release/v2.18: branch not found",
@asraa can you take care of this one?

./scorecard --repo=github.com/xwp/stream --checks=Branch-Protection
./scorecard --repo=github.com/americanexpress/one-app-cli --checks=Branch-Protection

"reason": "internal error: invalid score 2 != 0",
I will look into these two.

[laurentsimon]

Assigning to @asraa the part on version not being found. not urgent

[asraa]

It seems like this is because there's a large number of branches (>30) and the GitHub API making the call to ListBranches only returns 30 at a time. I'll see if we can pagify/get all.

[asraa]

Ah, this was fixed by increasing the refsToAnalyze.

Is this a hard-limit? I could also increase this so that it makes another API call to fetch branches if it wasn't found in the first 30.

[laurentsimon]

I see. @azeemshaikh38 can you chime in? Will this trigger issues with API rate limiting in the weekly cron?

@asraa how do we calculate the "right" number of refsToAnalyze? Heuristics?

[azeemshaikh38]

Nice catch! I'm a bit hesitant about increasing refsToAnalyze since that'll increase the API cost and we don't know how many branches we'll have to iterate through to get the right ones.

My suggestion would be to update ListBranches API to take a ListBranchesOptions{} struct that contains an optional list of branch names to retrieve. I haven't found a way to batch retrieve these branches, so we'll then have to make multiple calls to get the BranchProtectionRules for these names. . This still means more API calls but at least since the number of branches we are interested in is probably a handful, it's better than iterating through a bucket load of them.

One FYI, GitHub allows you to list all BranchProtectionRules for a repository, which is great but there is no similar option for RefUpdateRules which is what we rely on mostly given a read-only token. So another long-term solution here is to work with GitHub to get this exposed.

[asraa]

My suggestion would be to update ListBranches API to take a ListBranchesOptions{} struct that contains an optional list of branch names to retrieve.

I think I can work with that! Given that we only want the release and default branches for branch protection, I can plumb that in through setup to create the query for those branch names.

[naveensrinivasan]

I am going to work on this issue.