Proposal: improving review process with bot-assisted checkboxes

[moul]

We've been mulling over boosting our review process (#998, #985, #1005, and more).

Here's what I'm thinking:

1. We kick-off with a comment sporting a checkbox and easy-peasy instructions. A bot gets to tick this when the contributor's nailed all tasks, or a reviewer steps in to give it a nod.
2. We keep a CI/CD check yellow - a signal we're on standby, not in error mode - until the checkbox gets a tick.
3. Bonus: this special comment can be updated by the bot to give straightforward specific instructions.

This ensures that either the contributor's done their bit, or a reviewer's given it a thumbs up.

There's good stuff happening over at https://github.com/snyk/release-notes-preview - they're using a similar system for making commit messages chime with automatic changelog and releases.

Later on, our bot could step up - like ensuring devs have signed off on a DCO, or handling specific directives for crucial repo folders. For this, I'd go with a CI/CD reacting to PR changes, and invoking a homemade Go script we could nestle in misc/gh-action.

What do you think? Let's chew the fat over this.

[moul]

cc @ajnavarro @harry-hov @zivkovicmilos

[moul]

Update:

Changed settings to Title and description from Title and commit messages.

Realized the extension "refined github" already handles improvements: Read more.

We can maintain the current setup and explore configuring "refined github" to omit checkboxes while preserving other elements.

[ajnavarro]

@moul a super simple required checked checker: https://github.com/marketplace/actions/require-checklist

[moul]

Hey, I'm recycling this old PR because we now have an R&D engineer on the team (@aeddi). I'm also re-inviting external people who want to contribute to provide their help.

I think we should start having a bot that will perform basic, regular actions in the context of GitHub, but linked with our tools (the chains, the CI, some specific realms, etc.). This bot should be hosted in a standalone repository because it will be global to the organization and not only to the gno repository (even if we start with a limited scope).
It should be built in the open, but keep it simple with hardcoded values instead of making it overly generic / reusable.

See also (potentially related):

• META GitHub oracle & r/gh #1133
• Create a .github/CODEFACILITATORS file and its system #1466
• [infra] Set up a load testing workflow for PRs #2620
• Improve r/boards to make it the official (daily) communication tool #238

[aeddi]

That sounds really interesting to work on. 👍 I could maybe bootstrap a base on which to iterate over time to add features.

[thehowl]

cc/ @aeddi, other references relating to this issue

• miro board: https://miro.com/app/board/uXjVLe7myxc=/?moveToWidget=3458764600879563700&cot=14
• Rough text outline of what we want:

Bot makes a comment, with "requirements" that can be automatic or manual, and be restricted on who can check the boxes

ie.

AUTOMATED:
- [ ] IF (touches folder tm2) THEN (1 (review|author) from @gnolang/eu && 1 (review|author) from @gnolang/us)
- [ ] is PR up-to-date with master?
- [ ] are all CI checks green EXCEPT known flaky ones?
- [ ] is the PR allowing edits from maintainers?

MANUAL:
- [ ] IF (touches dir1) THEN ("Infra changes needed?") CHANGEABLE BY: (@gnolang/tech-staff)
- [ ] Is the code style good? CHANGEABLE BY: (@gnolang/tech-staff)

MANUAL, by author:
- [ ] have you reviewed CONTRIBUTING?
- [ ] IF (touches dir2) THEN ("...")

All checkboxes must be ticked for merge to work
(CI is red until everything is ticked; this is the only CI that is required for merge)

---

One thing that I was considering together with the review team is that we should probably look to disable CODEOWNERS in the long term, possibly assisted by this bot; so we can have external PRs be reviewed by the review team first (possibly using the label), and then they can assign a reviewer (or team) to take a look at specific PRs after the first round.

[aeddi]

Thanks! 👍
Indeed, as the functionalities of the bot will overlap with those of CODEOWNERS, it seems logical to disable it (or replace it with the bot's config file).

[Kouteki]

@moul a proposal that we can discuss

Short term solution

• Remove CODEOWNERS, PR reviewers aren't assigned automatically
• EXTERNAL PR: Review team assign themselves for the 1st review, assign the appropriate core team member after the 1st review
• INTERNAL PR: Assign directly by teammates (1st and 2nd review)

Long term solution

• Remove CODEOWNERS
• Bot differentiates between internal and external owners, assigns either Review team or individuals from core team
• If the PR affects a critical component, bot assigns a specific individual. The bot can block a merge if that specific individual hasn't approved it.