parity-processbot is a GitHub App which drives the Companion Build System's merge process.
Note that parity-processbot works independently of the Companion Build System's cross-repository integration check, which is done on CI (see check_dependent_project for that).
Before starting to work on this project we recommend reading the Implementation section.
processbot receives commands from pull request comments.
The merge commands will either merge the pull request right away, if possible, or merge it automatically once all of its requirements are passing; a guided description of how that works internally is provided in the Implementation section.
The following commands should be posted as pull request comments. Your whole comment should only have the command.
bot merge: merge once checks passbot merge force: merge immediately while disregarding checks (not all of them can be disregarded)bot merge cancel: cancel a pendingbot merge; does not affect anything outside of processbot, only stops the bot from following through with the mergebot rebase: create a merge commit from the target branch into the PR
Note: The commands will only work if you are a member of the organization where the GitHub App is installed. Organization membership is fetched from the GitHub API at the time a comment arrives.
processbot categorizes CI statuses as following, ranked in descending order of importance:
Required through GitHub branch protection rules
They are meant to be blockers so can't be skipped anyhow.
Derived from Gitlab Jobs which do not have allow_failure: true
They are relevant but not blockers, thus can be skipped with bot merge force
but will not pass bot merge. Note that the merge of companions follows the
logic of bot merge, thus a brittle job in this category might get in the way
of a companion merge.
Derived from Gitlab Jobs which have allow_failure: true
Unstable statuses will have allow_failure: true encoded in their descriptions
(delivered from vanity-service)
which will allow processbot to detect and disregard them.
All Important and above checks should be green when
using bot merge.
Non-Required statuses can bypassed by using bot merge force.
The GitHub App is necessary for the application to receive webhook events and access the GitHub API properly.
Follow the instructions of https://gitlab.parity.io/groups/parity/opstooling/-/wikis/Bots/Development/Create-a-new-GitHub-App for creating a new GitHub App.
After creating the app, you should configure and install it (make sure the environment is properly set up before using it).
- Contents: Read & write
- Enables pushing commits for updating companions after their dependencies have been merged
- Issues: Read & write
- Enables comment on pull requests
- Pull requests: Read & write
- Enables merging pull requests
- Commit statuses: Read-only
- Enables fetching the CI statuses before merge
- Checks: Read-only
- Enables fetching the checks' statuses before merge
- Workflows: Read & write
- Allows the bot to push commits to workflow files (see paritytech/cumulus#1436 (comment))
- Members: Read-only
- Enables fetching the command requester's organization membership even if their membership is private
- Issue comment
- Enables reacting to commands from GitHub comments
- Check run, Status, Workflow job
- Used to trigger the processing of pending pull requests
Having created and configured the
GitHub App, install it in a repository through
https://github.com/settings/apps/${APP}/installations.
If processbot has to merge PRs into protected branches which have the "Restrict who can push to matching branches" rule enabled, it should be added to the allowlist for that rule, otherwise merging will not work (example). In such cases it's necessary to add the app to the allowlist, as demonstrated below:
- Rust for running the application
- rustup is the recommended way of setting up a Rust toolchain
- libssl for the HTTPS requests library
- libclang for building the database (RocksDB)
- git for cloning companions and updating them
All relevant environment variables are documented in the
.env.example file. For development you're welcome to copy that
file to .env so that all values will be loaded automatically once the
application starts.
-
During development it's handy to use a smee.io proxy, through the
WEBHOOK_PROXY_URLenvironment variable, for receiving GitHub Webhook Events in your local server instance. -
Run the project with
cargo run -
Optionally try out the example workflows in the repositories where you have installed the app or the test repositories after a deployment
Example: paritytech-stg/main-for-processbot-staging#55
Steps:
- Create a pull request in the repositories where the app is installed
- Comment
bot merge
Example:
- Repository A: paritytech-stg/main-for-processbot-staging#53
- Repository B: paritytech-stg/companion-for-processbot-staging#31
Steps:
- Install the app in Repository A
- Install the app in Repository B
- Repository B needs to be a dependency of Repository A (example)
- Create a pull request on Repository B and copy its link
- Create a pull request on Repository A and put
companion: [link from step 3]in its description - Comment
bot mergeon the pull request in Repository A - Observe that the the pull request in Repository A will be merged first and the pull request on Repository B will be merged after
The staging instance is installed in the following repositories:
- https://github.com/paritytech/main-for-processbot-staging
- https://github.com/paritytech/companion-for-processbot-staging
The GitHub App for staging is managed by paritytech's Organizational GitHub Admins.
The integration tests are executed as follows:
./scripts/run_integration_tests.shWe use insta for integration
tests' snapshots. After creating or modifying a snapshot, use cargo insta review to manage the results.
All of the relevant configuration for deployment lives in the ./helm
folder. The values for each specific environment are in
helm/values-${ENVIRONMENT}.yml. If you add a value, it needs to be used in
helm/templates/processbot.yaml.
See https://gitlab.parity.io/groups/parity/opstooling/-/wikis
When you push a deployment tag to GitHub, it will be mirrored to GitLab and then its CI pipeline will be run for deploying the app.
The application can be deployed to the following environments:
-
Production: push a tag with the pattern
/^v[0-9]+\.[0-9]+.*$/, e.g.v1.1The production instance is installed in Substrate, Polkadot and Cumulus.
-
Staging: push a tag with the pattern
/^pre-v[0-9]+\.[0-9]+.*$/, e.g.pre-v0.6The staging instance is installed in the test repositories.
Before reading any of this, we strongly recommend to have a good understanding of the Companion Build System by consulting its explanation.
A
web server
(set up from
main)
receives
GitHub Webhook events
as HTTP POST requests.
When someone comments in a pull request, the issue comment event is parsed and from it a command is extracted and handled.
The merge chain is started from a merge command. If the pull request at the root of the chain is ready to be merged, it will be merged immediately, otherwise it will be saved to the database and merged later once its requirements are ready; by "requirements" we mean its statuses, checks and dependencies (the root of the chain is started without dependencies, hence why it can be merged first).
After a pull request is merged, its dependents are checked and possibly merged if all of their requirements are ready (note that a pull request my might depend on more than one pull request, as explained in the presentation at 4:48 or slide number 6). This process is repeated for each item that is merged throughout the merge chain (referred as "Phase 1 and Phase 2" in the presentation at 25:48 or slide number 21).