renovate-bot

• Overview
• Renovate
• Token
• Renovate bot configuration
• Scan new repositories
• Preset config
• Signed commits
  • Generating a GPG key
  • Add the public GPG to GitHub
  • Add the private GPG key to eana-bot
• Using CircleCI
  • The pipeline
  • Automerge
  • Reviewers
  • Scheduled triggers
• Caching
• Final words

Overview

A lot of developers either spend a lot of time on dependency management or spend no time on it at all. Often the developers don't even know that a new version of a dependency has been released.

Especially in large projects it frequently happens to skip 2-4 Major Versions before updating the dependencies again, simply because no one spends time to regularly check for updates. Bots like renovate or dependabot try to solve this by creating a new MR as soon as a new version of a dependency has been released. Unfortunately, these bots are not that well known yet and could be tricky to configure.

This repository tries to mitigate this very problem.

Renovate

Renovate is an open source project that you can be self-hosted. The hosted service works fine if your project is on GitHub or GitLab, but I've decided to run the service myself. This sounds a lot more complicated than it actually is, but it is not and the rest of this document will show you how to do it.

Renovate will use the eana-bot user to open merge requests. It will also have a dedicated repository for the main configuration. The CircleCI config on this repository will take care of regularly running the Renovate bot.

Renovate is a JavaScript-based project, but it supports updating dependencies of a lot of other languages too. Because it is based on JavaScript we will use npm (or yarn if you prefer) to install it and for that the first file we will create is the package.json file.

{
  "name": "renovate-bot",
  "private": true,
  "scripts": {
    "renovate": "renovate"
  },
  "dependencies": {
    "npm": "9.1.2",
    "renovate": "34.29.2"
  }
}

Afterwards, run npm install in the repository to install the renovate package into the node_modules folder. This step should also create a package-lock.json file which needs to be committed to the repository together with the new package.json file.

$ docker run -it --rm -v $(pwd):/data cimg/node:19.1.0 bash
$ cd /data
$ npm install --verbose
$ exit

Note: The node_modules should not be checked in the git repository and hence should be added to .gitignore.

Token

Note: If you plan to use the Renovate bot in a GitHub organization it's highly recommended to have a dedicated user for this, because using personal users is not a good practice and the Renovate bot will stop working when the user will be deleted.

Generate a personal access token with the repo:public_repo scope for only public repositories or the repo scope for public and private repositories, and add write it down for later use.

Renovate bot configuration

By default, renovate expects this to be called config.js. Inside this file we will configure how Renovate will behave by default. Please note that most of this can be overridden on a per-repository basis.

module.exports = {
  platform: "github",
  gitAuthor: "Eana Bot <eana-bot@eana.ro>",
  token: process.env.GITHUB_COM_TOKEN,
  gitPrivateKey: process.env.GPG_KEY,

  hostRules: [
    {
      hostType: "ecr",
      matchHost: "/d+.dkr.ecr.[-a-z0-9]+.amazonaws.com/",
      awsAccessKeyID: process.env.AWS_ACCESS_KEY_ID,
      awsSecretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
    },
  ],

  requireConfig: true,
  onboarding: true,

  onboardingConfig: {
    $schema: "https://docs.renovatebot.com/renovate-schema.json",
    extends: ["github>bolawell/renovate-config"],
  },
};

const fs = require("fs");
if (fs.existsSync("renovate-repos.json")) {
  if (
    !"CIRCLE_NODE_INDEX" in process.env ||
    !"CIRCLE_NODE_TOTAL" in process.env
  ) {
    console.log(
      "renovate-repos.json exists, but CIRCLE_NODE_INDEX and CIRCLE_NODE_TOTAL are not set. See https://circleci.com/docs/parallelism-faster-jobs",
    );
    process.exit(1);
  }

  segmentNumber = Number(process.env.CIRCLE_NODE_INDEX); // CIRCLE_NODE_INDEX is 1 indexed
  segmentTotal = Number(process.env.CIRCLE_NODE_TOTAL);
  allRepositories = JSON.parse(fs.readFileSync("renovate-repos.json"));
  allSize = allRepositories.length;
  chunkSize = parseInt(allSize / segmentTotal);
  chunkStartIndex = chunkSize * (segmentNumber - 1);
  chunkEndIndex = chunkSize * segmentNumber;

  if (chunkEndIndex > allSize) {
    chunkEndIndex = allSize;
  }

  segmentNumber = Number(process.env.CIRCLE_NODE_INDEX); // CIRCLE_NODE_INDEX is 1 indexed
  segmentTotal = Number(process.env.CIRCLE_NODE_TOTAL);
  allRepositories = JSON.parse(fs.readFileSync("renovate-repos.json"));
  repositories = allRepositories.filter(
    (_, i) => segmentNumber - 1 === i % segmentTotal,
  );
  module.exports.repositories = repositories;
  module.exports.autodiscover = false;
  console.log(
    `renovate-repos.json contains ${allRepositories.length} repositories. This is chunk number ${segmentNumber} of ${segmentTotal} total chunks. Processing ${repositories.length} repositories.`,
  );

  console.log(`Repositories to be scanned:`);
  console.log(repositories);
}

The first section of the file above tells our Renovate bot how to talk to our self-hosted GitLab instance and the private key to use to sign the git commits. Afterwards on which repositories it should run and what the renovate.json config file should look like that it will add to each repository in the first merge request.

To optimize the pipeline and reduce the running time, we need to scan multiple repositories in parallel. We do this by defining parallelism: x in config.yml. In the second part of the renovate config file, we split the renovate-repos.json file in x chunks. When the pipeline is triggered x jobs will be created, each job scanning their particular chunk of repositories.

Please note that at the moment of writing this document x = 3.

Scan new repositories

To scan new repositories with renovate we need to add the desired repositories to the renovate-repos.json file. There should be only one repository (including the full path) per line, between double-quotes and the line should end with a comma.

[
  "acme/repo1",
  "acme/group/repo",
  "acme/another-group/another-repo"
]

Make sure the repositories are sorted alphabetically (this is how you sort the lines in IntelliJ, vscode, vim) and there are no duplicates (this is how you remove duplicates in IntelliJ, vscode, vim).

Preset config

We want to manage multiple repositories using Renovate and want the same custom config across all or most of them, hence we have a preset config so that we can "extend" it in every applicable repository. This way when we want to change the Renovate configuration we can make the change in one location rather than having to copy/paste it to every repository individually. The preset config is configured in the renovate-config repository.

Signed commits

Why sign git commits? As useful as signing packages and ISOs is, an even more important use of GPG signing is in signing Git commits. When you sign a Git commit, you can prove that the code you submitted came from you and wasn't altered while you were transferring it. You also can prove that you submitted the code and not someone else.

For a commit to be verified by GitHub the following things are required:

• The committer must have a GPG public/private key pair.
• The committer's public key must have been uploaded to their GitHub account.
• One of the emails in the GPG key must match a verified email address used by the committer in GitHub.
• The committer's email address must match the verified email address from the GPG key.

Generating a GPG key

If you don't already have the GPG keys this document will help you get started. Use Eana bot for Real name and eana-bot@eana.ro for Email address. Also the key should not expire.

A very important note is that you need set an empty passphrase.

Add the public GPG to GitHub

After generating the GPG keys, the public key must be added to GitHub and this guide could be followed to add it.

Add the private GPG key to eana-bot

We need to export, base64 encode the key and add it as an environment variable.

On Linux:

gpg --armor --export-secret-key eana-bot@eana.ro | base64 | sed ':a;N;$!ba;s/\n//g' | xclip -sel clip

On Mac:

gpg --armor --export-secret-key eana-bot@eana.ro | base64 | sed ':a;N;$!ba;s/\n//g' | pbcopy

Using CircleCI

Create a CircleCI context called renovate-bot and add three environment variables as follows:

Environment variable	Value
GITHUB_COM_TOKEN	The generated token
GPG_KEY_BASE64	The base64 encoded private key you have in the clipboard
LOG_LEVEL	debug

This token is only used by Renovate, see the token configuration, and gives it access to the repositories.

The environment variable GITHUB_COM_TOKEN is used when fetching release notes for repositories in order to increase the hourly API limit.

When using Renovate in a project where dependencies are loaded from github.com (such as Go modules hosted on GitHub) it is highly recommended to add a token as the rate limit from the github.com API will be reached, which will lead to Renovate closing and reopening PRs because it could not get reliable info on updated dependencies.

The pipeline

The pipeline is triggered when a change is pushed/merged to master. Having the pipeline triggered only on pushing or merging pull requests to master is not enough, we want the renovate bot to run regularly. This pipeline uses restricted contexts (aws_svc_renovate and renovate-bot), hence scheduled workflows can not be used because they can't access these contexts. To make this work we will use scheduled pipelines.

Automerge

Automerging is a Renovate feature that you can use to automate upgrading dependencies. When enabled, Renovate tries to merge the proposed update once the tests pass. Renovate tries platform-native automerge only when it initially creates the PR. Any PR that is being updated will be automerged with the Renovate-based automerge.

Reviewers

When automerge is enabled on a PR, Renovate will not add assignees or reviewers at PR creation time, in order to decrease notifications noise a little. If tests subsequently fail, making automerge not possible, then Renovate will add the configured assignees and/or reviewers.

Note: Renovate won't add assignees and reviewers to a PR with failing checks if the PR already has assignees or reviewers present.

Scheduled triggers

The pipeline can be scheduled by using scheduled triggers. At the moment of writing this document there are two ways to create the triggers.

• the scheduled triggers can now also be managed via the UI, in the project settings under the "Triggers" section
• using the CircleCI API

To create a new scheduled trigger using the API a API token is required.

curl --location --request POST 'https://circleci.com/api/v2/project/gh/bolawell/renovate-bot/schedule' \
    --header 'circle-token: <your_API_token>' \
    --header 'Content-Type: application/json' \
    --data-raw '{
        "name": "run-renovate-on-schedule",
        "description": "Monday to Friday, at 9:00, 12:00, 15:00",
        "attribution-actor": "current",
        "parameters": {
            "branch": "master"
        },
        "timetable": {
            "per-hour": 1,
            "hours-of-day": [9, 12, 15],
            "days-of-week": ["MON", "TUE", "WED", "THU", "FRI"]
        }
    }'

The API call above create a scheduled trigger named run-renovate-on-schedule, owned by the current user (eana), against the master branch on Monday to Friday, at 9:00, 12:00, 15:00

Caching

By default, renovate caches lookup results, including dependency versions and release notes between repositories and runs on the local file system. However, since multiple jobs run in parallel, caching on the local file system is suboptimal and underutilized. Therefore, I have decided to use a small redis instance as the global cache instead.

To start, head over to https://redislabs.com/ and sign up and once you create and verify your account, click "Create your subscription". Scroll down to "Fixed size" and pick the free option. Click "Create". We now need to create the database under the subscription. User renovate-bot as the database name. Be sure to copy "Redis Password" and save it to the password manager. Once you are ready, click "Activate". Navigate to "Data access control" and then create a new Role and User.

To use the redis instance, add the RENOVATE_REDIS_URL environment variable following the same steps as described above. In the 'Environment Variable Name' field, enter RENOVATE_REDIS_URL and redis://username:password@redis-endpoint as the 'Value'.

Replace:

• username and password with user and password created above
• redis-endpoint with the public endpoint available in the database details

Final words

The next hour CircleCI should trigger the run job, that will run our Renovate bot, and that will create merge requests on your configured repositories.

Aaaand that's it!