Skip to content
This repository has been archived by the owner on Oct 11, 2023. It is now read-only.

paritytech/parity-processbot

Repository files navigation

Introduction

GitHub Issue Sync

parity-processbot is a GitHub App which drives the Companion Build System's merge process.

Note that parity-processbot works independently of the Companion Build System's cross-repository integration check, which is done on CI (see check_dependent_project for that).

Before starting to work on this project we recommend reading the Implementation section.

TOC

How it works

processbot receives commands from pull request comments.

The merge commands will either merge the pull request right away, if possible, or merge it automatically once all of its requirements are passing; a guided description of how that works internally is provided in the Implementation section.

Commands

The following commands should be posted as pull request comments. Your whole comment should only have the command.

  • bot merge: merge once checks pass
  • bot merge force: merge immediately while disregarding checks (not all of them can be disregarded)
  • bot merge cancel: cancel a pending bot merge; does not affect anything outside of processbot, only stops the bot from following through with the merge
  • bot rebase: create a merge commit from the target branch into the PR

Note: The commands will only work if you are a member of the organization where the GitHub App is installed. Organization membership is fetched from the GitHub API at the time a comment arrives.

Relation to CI

processbot categorizes CI statuses as following, ranked in descending order of importance:

1. Required

Required through GitHub branch protection rules

They are meant to be blockers so can't be skipped anyhow.

2. Important

Derived from Gitlab Jobs which do not have allow_failure: true

They are relevant but not blockers, thus can be skipped with bot merge force but will not pass bot merge. Note that the merge of companions follows the logic of bot merge, thus a brittle job in this category might get in the way of a companion merge.

3. Fallible

Derived from Gitlab Jobs which have allow_failure: true

Unstable statuses will have allow_failure: true encoded in their descriptions (delivered from vanity-service) which will allow processbot to detect and disregard them.

Criteria for merge

Checks and statuses

All Important and above checks should be green when using bot merge.

Non-Required statuses can bypassed by using bot merge force.

GitHub App

The GitHub App is necessary for the application to receive webhook events and access the GitHub API properly.

Follow the instructions of https://gitlab.parity.io/groups/parity/opstooling/-/wikis/Bots/Development/Create-a-new-GitHub-App for creating a new GitHub App.

After creating the app, you should configure and install it (make sure the environment is properly set up before using it).

Configuration

Repository permissions

  • Contents: Read & write
    • Enables pushing commits for updating companions after their dependencies have been merged
  • Issues: Read & write
    • Enables comment on pull requests
  • Pull requests: Read & write
    • Enables merging pull requests
  • Commit statuses: Read-only
    • Enables fetching the CI statuses before merge
  • Checks: Read-only
    • Enables fetching the checks' statuses before merge
  • Workflows: Read & write

Organization permissions

  • Members: Read-only
    • Enables fetching the command requester's organization membership even if their membership is private

Events

  • Issue comment
    • Enables reacting to commands from GitHub comments
  • Check run, Status, Workflow job
    • Used to trigger the processing of pending pull requests

Installation

Having created and configured the GitHub App, install it in a repository through https://github.com/settings/apps/${APP}/installations.

If processbot has to merge PRs into protected branches which have the "Restrict who can push to matching branches" rule enabled, it should be added to the allowlist for that rule, otherwise merging will not work (example). In such cases it's necessary to add the app to the allowlist, as demonstrated below:

image

Setup

Requirements

  • Rust for running the application
    • rustup is the recommended way of setting up a Rust toolchain
  • libssl for the HTTPS requests library
  • libclang for building the database (RocksDB)
  • git for cloning companions and updating them

Environment variables

All relevant environment variables are documented in the .env.example file. For development you're welcome to copy that file to .env so that all values will be loaded automatically once the application starts.

Development

Run the application

  1. Set up the GitHub App

  2. Set up the application

    During development it's handy to use a smee.io proxy, through the WEBHOOK_PROXY_URL environment variable, for receiving GitHub Webhook Events in your local server instance.

  3. Run the project with cargo run

  4. Optionally try out the example workflows in the repositories where you have installed the app or the test repositories after a deployment

Example workflows

Single merge use-case

Example: paritytech-stg/main-for-processbot-staging#55

Steps:

  1. Create a pull request in the repositories where the app is installed
  2. Comment bot merge

Companion use-case

Example:

Steps:

  1. Install the app in Repository A
  2. Install the app in Repository B
  • Repository B needs to be a dependency of Repository A (example)
  1. Create a pull request on Repository B and copy its link
  2. Create a pull request on Repository A and put companion: [link from step 3] in its description
  3. Comment bot merge on the pull request in Repository A
  4. Observe that the the pull request in Repository A will be merged first and the pull request on Repository B will be merged after

Test repositories

The staging instance is installed in the following repositories:

The GitHub App for staging is managed by paritytech's Organizational GitHub Admins.

Integration tests

The integration tests are executed as follows:

./scripts/run_integration_tests.sh

We use insta for integration tests' snapshots. After creating or modifying a snapshot, use cargo insta review to manage the results.

Deployment

All of the relevant configuration for deployment lives in the ./helm folder. The values for each specific environment are in helm/values-${ENVIRONMENT}.yml. If you add a value, it needs to be used in helm/templates/processbot.yaml.

Logs

See https://gitlab.parity.io/groups/parity/opstooling/-/wikis

Environments

When you push a deployment tag to GitHub, it will be mirrored to GitLab and then its CI pipeline will be run for deploying the app.

The application can be deployed to the following environments:

  • Production: push a tag with the pattern /^v[0-9]+\.[0-9]+.*$/, e.g. v1.1

    The production instance is installed in Substrate, Polkadot and Cumulus.

  • Staging: push a tag with the pattern /^pre-v[0-9]+\.[0-9]+.*$/, e.g. pre-v0.6

    The staging instance is installed in the test repositories.

Implementation

Before reading any of this, we strongly recommend to have a good understanding of the Companion Build System by consulting its explanation.

A web server (set up from main) receives GitHub Webhook events as HTTP POST requests.

When someone comments in a pull request, the issue comment event is parsed and from it a command is extracted and handled.

The merge chain is started from a merge command. If the pull request at the root of the chain is ready to be merged, it will be merged immediately, otherwise it will be saved to the database and merged later once its requirements are ready; by "requirements" we mean its statuses, checks and dependencies (the root of the chain is started without dependencies, hence why it can be merged first).

After a pull request is merged, its dependents are checked and possibly merged if all of their requirements are ready (note that a pull request my might depend on more than one pull request, as explained in the presentation at 4:48 or slide number 6). This process is repeated for each item that is merged throughout the merge chain (referred as "Phase 1 and Phase 2" in the presentation at 25:48 or slide number 21).