Skip to content

Latest commit

 

History

History
266 lines (213 loc) · 10.6 KB

PROCESSES.md

File metadata and controls

266 lines (213 loc) · 10.6 KB
Raw

Processes, release procedures etc

This repository includes multiple projects at multiple versions, some of which depend on each other. That makes things tricky, and leads to rather more complicated procedures than a simple "single package" repository. Please read this guide carefully to understand how releases happen, and why they happen that way.

The API catalog

Only the source within the apis directory is published, although not every project within that directory is published. (There are test projects and tools alongside the production code.) Nothing in the tools directory is published as a package.

Most project files under apis are at least partially generated. Metadata used for generation is in apis.json - the API catalog file. There's an entry for each API, containing:

  • The kind of API (grpc, rest, other)
  • The version number
  • A product title and home page for documentation
  • Dependencies
  • Additional test project dependencies
  • Target framework versions
  • Package description for NuGet
  • Tags for NuGet (in addition to default ones)
  • Proto path within the googleapis repository
  • Service YAML file

The catalog is used to generate project files and also during the release process described below. Running the project generator is very simple: from the root directory, in a bash shell, run

./generateprojects.sh

The CI systems run this before building, to ensure that the project files are in a stable state.

Generating the project files allows for broad changes (such as adding Source Link support) to be made very simply, just by changing the generator. Modifying every project file by hand simply doesn't scale.

However, sometimes manual editing of project files is required. The project generator supports this by assuming it "owns":

  • The first PropertyGroup element (for general properties)
  • The first ItemGroup element (for dependencies)
  • The ItemGroup element for Source Link (with a label of "dotnet pack instructions")
  • The common import to only attempt to build desktop assemblies on Windows

Any other elements are left as they are - so if you wish to add an ItemGroup such as for file grouping, just add it anywhere after the generated one, and it should do the right thing.

Additionally, for each API, the project generator adds all projects under the API directory (and project references) to the solution for that API. It will create project files from scratch as well - so when adding a new package from autogenerated API sources, the simplest approach is to copy the source files, modify the API catalog, and run the project generator. The project files and solution file will be generated and should immediately be usable.

Versioning

All releasable packages follow Semantic Versioning. Non-releasable code (tests, tools, analyzers) are not versioned. The precise meaning of a breaking change (dictating version number increments) is out of the scope of this document. (Jon is writing a blog post about this topic and will link to it when it is published.)

The version number in the API catalog (and therefore in project files) is one of:

  • The mostly recently released package version
  • The version that's about to be released (see below)
  • A prerelease with a suffix of 00 to indicate that the next release should be the first prerelease for that major/minor version.

The last of these cases can happen if either an existing version has gone to GA and new features have been merged since (to ensure that we don't add features within a minor GA version) or for brand-new APIs that haven't been released at all. The tooling for tagging is aware of this convention, and doesn't create tags for alpha00 or beta00 suffixes.

Typically version numbers should be changed in a commit which does nothing else, for clarity. Include both the apis.json change and the project file changes that occur when the project generator has been run, in the same commit.

Dependencies

Dependencies in the API catalog are specified as properties where the property name is the package name and the value is the version number.

If the version number is set to "project", then a project reference is used. If project X has a production dependency on project Y, then both X and Y must be released together, to avoid misaligned versions. The tagging tool enforces this.

If the version number is set to "default", then the version number is determined by the project generator, to keep these dependencies in sync for all appropriate packages.

Two project types gain dependencies automatically if they're not specified:

  • "grpc" projects always have dependencies on Google.Api.Gax.Grpc and Grpc.Core
  • "rest" projects always have a dependency on Google.Api.Gax.Rest

The best dependency version to specify depends on the version of the project itself:

  • For GA versions, every version must be explicitly specified. This prevents us from accidentally upgrading a dependency minor version when creating a patch version of the project itself.
  • For alpha/beta versions, specify as little as possible: allow the "grpc"/"rest" dependencies above to be added automatically, and use the "default" version number where possible. Explicit versions should only be used either for dependencies without default versions, or when the desired version is ahead of the default version, for example to use a GAX prerelease.

Releasing

Releasing consists of these steps:

  • Updating the version number in GitHub (via standard pull requests), as well as the version history.
  • Creating a release tag and GitHub release
  • Building and testing
  • Pushing the package to nuget.org
  • Updating the documentation in GitHub (in the gh-pages branch)

The last three of these are typically performed by a Kokoro release job.

Detailed steps

Much of this work is automated by the prepare-release.sh script, but allows for intervention at any point. This documentation assumes the use of prepare-release.sh, but that's just for simplicity. Everything can be done manually.

Note that only the set-version command for prepare-release needs the package ID to be specified. All other commands work on the basis of "whatever versions have been changed since the latest commit on GitHub are the ones we want to release".

Create the release PR

  1. Make sure the master branch is up-to-date (as that's what prepare-release.sh uses to determine the current versions) and create a new branch from that. If you want to use the compare option later, run git fetch --all --tags -f as well to make sure you have all the latest tags.

  2. Run ./prepare-release.sh set-version <api> <new-version>, e.g. prepare-release.sh set-version Google.Cloud.Speech.V1 2.0.0-beta03. This updates apis.json and regenerates the project and metadata files. The output includes the old version and the new version, so you can check this is really what you meant to do.

    As an alternative to set-version, you can run ./prepare-release.sh increment-version <api> which will increment the minor version for GA libraries (e.g. 3.1.0 to 3.2.0) or the prerelease version of non-GA libraries (e.g. 3.0.0-beta01 to 3.0.0-beta02). This just avoids you having to look up the current version number if you don't already know it. If you want to look up the version number anyway, use ./prepare-release.sh show-version <api>.

  3. If you want to perform an API surface comparison, run ./prepare-release.sh compare. This will build the code locally, and compare the previously released NuGet packages with the current source code, showing you what's changed. Check there's nothing unexpected. Note that this will only show changes to the API surface; internal changes (even if they change behavior) will not be shown.

  4. Run ./prepare-release.sh update-history. This will update the version history files of all changed APIs, based on the commits that touched the relevant directories. The command displays any additions that it makes to the version history, along with the name of the history file for each API.

  5. If you're only releasing a single package, run ./prepare-release.sh commit. This will commit all the current changes, with a message taken from the version history for the package. Use git commit --amend to change the commit message if you need to.

    If you're releasing more than one package, use ./prepare-release.sh commit-multiple <commit title>, e.g. ./prepare-release.sh commit-multiple "Release Spanner libraries version 3.3.0". Alternatively, create the commit manually, including one line per package of the form - Release XYZ version ABC.

    Note: If you're using the Dockerfile described in ENVIRONMENT.md, please read the notes there around commiting to git "inside" vs "outside" the container.

  6. Run ./prepare-release.sh push to push the current branch and create a pull request with the autorelease: pending tag. Note that this checks that there are no project references from APIs being released now to APIs that aren't being released. Without this check, it's possible for a released version to depend on unreleased changes. This uses the GITHUB_ACCESS_TOKEN environment variable to authenticate with the API, so make this is set beforehand. The access token should include the repo scope.

Sample session when releasing Google.Cloud.Speech.V1:

$ git checkout master
$ git pull upstream master
$ git fetch --all --tags -f
$ git checkout -b release-speech
$ ./prepare-release.sh set-version Google.Cloud.Speech.V1 2.0.0-beta03
$ ./prepare-release.sh compare
$ ./prepare-release.sh update-history
$ ./prepare-release.sh commit
$ export GITHUB_ACCESS_TOKEN=...
$ ./prepare-release.sh push

Equivalent process using increment-version, assuming the current version is 2.0.0-beta02:

$ git checkout master
$ git pull upstream master
$ git fetch --all --tags -f
$ git checkout -b release-speech
$ ./prepare-release.sh increment-version Google.Cloud.Speech.V1
$ ./prepare-release.sh compare
$ ./prepare-release.sh update-history
$ ./prepare-release.sh commit
$ export GITHUB_ACCESS_TOKEN=...
$ ./prepare-release.sh push

Building and publishing the release

Once the pull request is merged, the commit will be tagged automatically, and a release will be created. A Kokoro release build will then execute automatically.

The Kokoro build will:

  • Fetch the commit
  • See which tags are present for that commit (to work out which packages need releasing)
  • For all those packages:
    • Build
    • Run unit tests
    • Run integration/snippet tests
    • Build documentation
    • Push packages to nuget.org
    • Push documentation to GitHub packages
    • Push documentation to googelapis.dev