docs: add info around next version, testing other projects (#19157)

Co-authored-by: Matt Henkes <mjhenkes@gmail.com> Co-authored-by: Emily Rohrbough <emilyrohrbough@users.noreply.github.com> Co-authored-by: Tyler Biethman <tbiethman@users.noreply.github.com>
cypress-io · Dec 2, 2021 · 42dfb8a · 42dfb8a · cypress-bot · Dec 2, 2021
1 parent 2c9cb68
commit 42dfb8a
Show file tree

Hide file tree

Showing 9 changed files with 164 additions and 47 deletions.
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -650,7 +650,7 @@ Below are some guidelines Cypress uses when reviewing dependency updates.
 
 ## Deployment
 
-We will try to review and merge pull requests quickly. If you want to know our build process or build your own Cypress binary, read [DEPLOY.md](./DEPLOY.md).
+We will try to review and merge pull requests quickly. If you want to know our build process or build your own Cypress binary, read [the "Release Process" guide](./guides/release-process.md).
 
 Independent packages are deployed immediately upon being merged into master. You can read more [above](#independent-packages-ci-workflow).
 

diff --git a/guides/README.md b/guides/README.md
@@ -0,0 +1,17 @@
+# Guides
+
+The `guides` directory contains information about specific things encountered *when contributing to the Cypress project* that fall outside the scope of `CONTRIBUTING.md`, which is more geared towards everyday development in this repo.
+
+For information about using Cypress for testing web applications, visit [`docs.cypress.io`](https://docs.cypress.io).
+
+For information related to a specific package in the monorepo, see that package's respective `README.md`.
+
+For general contributor information, check out [`CONTRIBUTING.md`](../CONTRIBUTING.md).
+
+## Table of Contents
+
+* [Building release artifacts](./building-release-artifacts.md)
+* [Code signing](./code-signing.md)
+* [Determining the next version of Cypress to be released](./next-version.md)
+* [Release process](./release-process.md)
+* [Testing other projects](./testing-other-projects.md)
diff --git a/guides/building-release-artifacts.md b/guides/building-release-artifacts.md
@@ -0,0 +1,38 @@
+# Building Release Artifacts
+
+The `cypress` NPM package consists of two main parts:
+
+1. The `cypress` NPM package `.tgz` (built from [`cli`](../cli))
+    * Contains the command line tool `cypress`, type definitions, and the [Module API](https://on.cypress.io/module-api).
+    * End users install this via NPM to the project's `node_modules`.
+2. The "binary" `.zip` (built from [`packages/server`](../packages/server))
+    * Contains the Electron app, `ffmpeg`, and built versions of the [`server`](../packages/server), [`desktop-gui`](../packages/desktop-gui), [`runner`](../packages/runner), [`example` project](../packages/example), and [`extension`](../packages/extension)
+        * Also contains all the production dependencies of the above.
+    * This is installed when the `cli` is installed or when `cypress install` is run, to a system cache.
+
+This guide has instructions for building both.
+
+## Building the npm package
+
+> :warning: Note: The steps in this section are automated in CI, and you should not need to do them yourself when going through the [release process](./release-process.md).
+
+Building a new npm package is two commands:
+
+1. Increment the version in the root `package.json`
+2. `yarn build --scope cypress`
+
+The steps above:
+
+- Build the `cypress` npm package
+- Transpile the code into ES5 to be compatible with the common Node versions
+- Put the result into the [`cli/build`](../cli/build) folder.
+
+## Building the binary
+
+> :warning: Note: The steps in this section are automated in CI, and you should not need to do them yourself when going through the [release process](./release-process.md).
+
+The npm package requires a corresponding binary of the same version. In production, it will try to retrieve the binary from the Cypress CDN if it is not cached locally.
+
+You can build the Cypress binary locally by running `yarn binary-build`. You can use Linux to build the Cypress binary (just like it is in CI) by running `yarn binary-build` inside of `yarn docker`.
+
+`yarn binary-zip` can be used to zip the built binary together.
diff --git a/guides/code-signing.md b/guides/code-signing.md
@@ -0,0 +1,23 @@
+# Code Signing
+
+Code signing is done for the Windows and Mac distributions of Cypress when they are built in CI.
+
+`electron-builder` handles code signing during the `create-build-artifacts` jobs. This guide assumes that the reader is already familiar with [`electron-builder`'s Code Signing documentation](https://www.electron.build/code-signing).
+
+## Installing a new Mac code signing key
+
+Follow the directions supplied by `electron-builder`: https://www.electron.build/code-signing#travis-appveyor-and-other-ci-servers
+
+Set the environment variables `CSC_LINK` and `CSC_KEY_PASSWORD` in the `test-runner:sign-mac-binary` CircleCI context.
+
+## Installing a new Windows code signing key
+
+1. Obtain the private key and full certificate chain in ASCII-armored PEM format and store each in a file (`-----BEGIN PRIVATE KEY-----`, `-----BEGIN CERTIFICATE-----`)
+2. Using `openssl`, convert the plaintext PEM public and private key to binary PKCS#12/PFX format and encrypt it with a real strong password.
+    ```shell
+    ➜ openssl pkcs12 -export -inkey key.pem -in cert.pem -out encrypted.pfx
+    Enter Export Password: <password>
+    Verifying - Enter Export Password: <password>
+    ```
+3. Upload the `encrypted.pfx` file to the Cypress App Google Drive and obtain a [direct download link](http://www.syncwithtech.org/p/direct-download-link-generator.html).
+4. Within the `test-runner:sign-windows-binary` CircleCI context, set `CSC_LINK` to that URL and `CSC_KEY_PASSWORD` to the password.
diff --git a/guides/next-version.md b/guides/next-version.md
@@ -0,0 +1,18 @@
+# Next Version
+
+When the Cypress CLI and binary are built in CI, the version number they share is determined by the script in [`/scripts/get-next-version.js`](../scripts/get-next-version.js).
+
+In most cases, the script will correctly determine the next version. If it needs to be overridden, the environment variable `NEXT_VERSION` can be set to the desired version (`NEXT_VERSION=1.2.3`). This is commonly desired on a release branch or when trying to force a specific major version.
+
+The `get-next-version.js` script follows these steps:
+
+1. If the environment variable `NEXT_VERSION` exists, print `NEXT_VERSION` and exit.
+2. Otherwise, analyze the commits to the current branch since the last release, using semantic commit messages. Print out the calculated version.
+    * Only commits that touch files in `packages/*` or `cli/*` are considered.
+        * This is done so that commits to the `npm/` packages do not necessarily bump the CLI and binary versions.
+    * This project uses the [`angular` commit message style](https://gist.github.com/brianclements/841ea7bffdb01346392c/8e1f9b44d3fc7a4f2b448581071f9805f759c912).
+        * The prefix `fix:` will trigger a patch version bump.
+        * The prefix `feat:` will trigger a minor version bump.
+        * A commit with the footer containing `BREAKING CHANGE:` should trigger a major version bump.
+
+You can debug the script locally by running it using `node ./scripts/get-next-version.js`. It will analyze the commits on your current branch and print out the calculated version number.
diff --git a/DEPLOY.md → guides/release-process.md b/DEPLOY.md → guides/release-process.md
@@ -1,40 +1,10 @@
-# Deployment
+# Release Process
 
-These deployment procedures mainly concern the Cypress binary and `cypress` npm module.
+These procedures concern the release process for the Cypress binary and `cypress` npm module.
 
-Independent `@cypress/` packages that live inside the [`npm`](./npm) directory are automatically published to npm (with [`semantic-release`](https://semantic-release.gitbook.io/semantic-release/)) upon being merged into master. You can read more about this in [CONTRIBUTING.md](./CONTRIBUTING.md#committing-code)
+The `@cypress/`-namespaced NPM packages that live inside the [`/npm`](../npm) directory are automatically published to npm (with [`semantic-release`](https://semantic-release.gitbook.io/semantic-release/)) upon being merged into `master`. You can read more about this in [CONTRIBUTING.md](../CONTRIBUTING.md#independent-packages-ci-workflow).
 
-Anyone can build the binary and npm package, but you can only deploy the Cypress application and publish the npm module `cypress` if you are a member of the `cypress` npm organization.
-
-> :information_source: See the [publishing](#publishing) section for how to build, test and publish a
-new official version of the binary and `cypress` npm package.
-
-## Building Locally
-
-### Building the npm package
-
-> :warning: Note: The steps in this section are automated in CI, and you should not need to do them yourself when publishing.
-
-Building a new npm package is very quick.
-
-- Increment the version in the root `package.json`
-- `yarn build --scope cypress`
-
-The steps above:
-
-- Build the `cypress` npm package
-- Transpile the code into ES5 to be compatible with the common Node versions
-- Put the result into the [`cli/build`](./cli/build) folder.
-
-### Building the binary
-
-> :warning: Note: The steps in this section are automated in CI, and you should not need to do them yourself when publishing.
-
-The npm package requires a corresponding binary of the same version. In production, it will try to retrieve the binary from the Cypress CDN if it is not cached locally.
-
-You can build the Cypress binary locally by running `yarn binary-build`. You can use Linux to build the Cypress binary (just like it is in CI) by running `yarn binary-build` inside of  `yarn docker`.
-
-`yarn binary-zip` can be used to zip the built binary together.
+[Anyone can build the binary and npm package locally](./building-release-artifacts.md), but you can only deploy the Cypress application and publish the npm module `cypress` if you are a member of the `cypress` npm organization.
 
 ## Publishing
 
@@ -61,6 +31,11 @@ You can build the Cypress binary locally by running `yarn binary-build`. You can
         ZENHUB_API_TOKEN="..."
         ```
     - The `cypress-bot` GitHub app credentials are also needed. Ask another team member who has done a deploy for those.
+        ```text
+        GITHUB_APP_CYPRESS_INSTALLATION_ID=
+        GITHUB_APP_ID=
+        GITHUB_PRIVATE_KEY=
+        ```
     - For purging the Cloudflare cache (part of the `move-binaries` step), you'll need `CF_ZONEID` and `CF_TOKEN` set. These can be found in 1Password. If you don't have access, ask a team member who has done a deploy.
        ```text
         CF_ZONEID="..."
@@ -70,16 +45,12 @@ You can build the Cypress binary locally by running `yarn binary-build`. You can
 
 ### Before Publishing a New Version
 
-In order to publish a new `cypress` package to the npm registry, we must build and test it across multiple platforms and test projects. This makes publishing *directly* into the npm registry impossible. Instead, we have CI set up to do the following on every commit to `develop`:
+In order to publish a new version of the `cypress` package to the npm registry, CI must build and test it across multiple platforms and test projects. CI is set up to do the following on every commit to `develop`:
 
-1. Build the npm package with the new target version baked in.
+1. Build the npm package with the [next target version](./next-version.md) baked in.
 2. Build the Linux/Mac binaries on CircleCI and build Windows on AppVeyor.
-3. Upload the binaries and the new npm package to `cdn.cypress.io` under the "beta" folder.
-4. Launch the test projects like [cypress-test-node-versions](https://github.com/cypress-io/cypress-test-node-versions) and [cypress-test-example-repos](https://github.com/cypress-io/cypress-test-example-repos) using the newly-uploaded package & binary instead of installing from the npm registry. That installation looks like this:
-    ```shell
-    export CYPRESS_INSTALL_BINARY=https://cdn.../binary/<new version>/<commit hash>/cypress.zip
-    npm i https://cdn.../npm/<new version>/<commit hash>/cypress.tgz
-    ```
+3. Upload the binaries and the new npm package to the AWS S3 Bucket `cdn.cypress.io` under the "beta" folder.
+4. [Launch test projects](./testing-other-projects.md) using the newly-uploaded package & binary instead of installing from the npm registry.
 
 Multiple test projects are launched for each target operating system and the results are reported
 back to GitHub using status checks so that you can see if a change has broken real-world usage
@@ -91,11 +62,11 @@ Once the `develop` branch for all test projects are reliably passing with the ne
 
 ### Steps to Publish a New Version
 
-In the following instructions, "X.Y.Z" is used to denote the version of Cypress being published.
+In the following instructions, "X.Y.Z" is used to denote the [next version of Cypress being published](./next-version.md).
 
 1. `develop` should contain all of the changes made in `master`. However, this occasionally may not be the case. Ensure that `master` does not have any additional commits that are not on `develop` and all auto-generated pull requests designed to merge master into develop have been successfully merged.
 
-2. If there is a new [`cypress-example-kitchensink`](https://github.com/cypress-io/cypress-example-kitchensink/releases) version, update the corresponding dependency in [`packages/example`](./packages/example) to that new version.
+2. If there is a new [`cypress-example-kitchensink`](https://github.com/cypress-io/cypress-example-kitchensink/releases) version, update the corresponding dependency in [`packages/example`](../packages/example) to that new version.
 
 3. Use the `move-binaries` script to move the binaries for `<commit sha>` from `beta` to the `desktop` folder for `<new target version>`. This also purges the cloudflare cache for this version.
     ```shell
@@ -110,7 +81,7 @@ In the following instructions, "X.Y.Z" is used to denote the version of Cypress
             ![cdn-tgz-link](https://user-images.githubusercontent.com/1157043/80608736-3791e800-8a05-11ea-8d75-e4f80128e857.png)
     - Publish to the npm registry straight from the URL:
         ```shell
-        npm publish https://cdn.../npm/X.Y.Z/<long sha>/cypress.tgz --tag dev
+        npm publish https://cdn.cypress.io/beta/npm/X.Y.Z/<long sha>/cypress.tgz --tag dev
         ```
 
 5. Double-check that the new version has been published under the `dev` tag using `npm info cypress` or [available-versions](https://github.com/bahmutov/available-versions). `latest` should still point to the previous version. Example output:
@@ -157,7 +128,7 @@ In the following instructions, "X.Y.Z" is used to denote the version of Cypress
 
 11. If needed, push out any updated changes to the links manifest to [`on.cypress.io`](https://github.com/cypress-io/cypress-services/tree/develop/packages/on).
 
-12. If needed, deploy the updated [`cypress-example-kitchensink`][cypress-example-kitchensink] to `example.cypress.io` by following [these instructions under "Deployment"](./packages/example/README.md).
+12. If needed, deploy the updated [`cypress-example-kitchensink`][cypress-example-kitchensink] to `example.cypress.io` by following [these instructions under "Deployment"](../packages/example/README.md).
 
 13. Update the releases in [ZenHub](https://app.zenhub.com/workspaces/test-runner-5c3ea3baeb1e75374f7b0708/reports/release):
     - Close the current release in ZenHub.

diff --git a/guides/testing-other-projects.md b/guides/testing-other-projects.md
@@ -0,0 +1,46 @@
+# Testing other projects
+
+In `develop`, `master`, and any other branch configured in [`circle.yml`](../circle.yml), the Cypress binary and npm package are built and uploaded to `cdn.cypress.io`. Then, tests are run, using a variety of real-world example repositories.
+
+Two main strategies are used to spawn these test projects:
+
+1. Local CI
+2. Remote CI
+
+## Local CI
+
+A number of CI jobs in `circle.yml` clone test projects and run tests as part of `cypress-io/cypress`'s CI pipeline.
+
+You can find a list of test projects that do this by searching for usage of the `test-binary-against-repo` step.
+
+Similarly to "Remote CI" test projects, Local CI test projects will attempt to check out a branch that is named after the [next version](./next-version.md) (`X.Y.Z`) if one exists in the test project git repo.
+
+One advantage to local CI is that it does not require creating commits to another repo.
+
+## Remote CI
+
+After the production binary and NPM package are build and uploaded in CI, [`/scripts/test-other-projects.js`](../scripts/test-other-projects.js) is run as part of the `test-other-projects` `circle.yml` step.
+
+This script creates commits inside of several test projects (hence "Remote CI") in order to trigger a realistic, continous-integration test of Cypress.
+
+For a list of the projects, see the definition of `_PROVIDERS` in [`/scripts/binary/bump.js`](../scripts/binary/bump.js).
+
+For each project and operating system combo in `_PROVIDERS`, the script:
+
+1. Creates a commit to the test project's GitHub repo using the API. [An example of such a commit.](https://github.com/cypress-io/cypress-test-tiny/commit/5b39f3f43f6b7598f0d57cffcba71a7048d1d809)
+    * Note the commit is specifically for `linux`, and only the `linux-tests` job runs to completion.
+    * If a branch exists that is named after the [next version](./next-version.md) (`X.Y.Z`), the commit will be made to that branch.
+        * This is useful to test a release's breaking changes or new features against an example project without having to have the project's main branch in a broken state.
+    * Otherwise, the default branch is used for the commit.
+2. Creates a status check in this GitHub repo (`cypress-io/cypress`) and marks it `pending`.
+3. Waits for the test project's CI workflow to finish running.
+    * Each test project is configured to use [`@cypress/commit-message-install`](https://github.com/cypress-io/commit-message-install) to configure the exact test required via the information in the commit message.
+    * Each test project is configured to update the `pending` CI job in `cypress-io/cypress` to a `success` when the CI workflow successfully finishes.
+
+These tests add coverage to the Cypress code base by:
+
+* Providing a super-close-to-real-world usage of Cypress (i.e. installing fresh from an NPM package and running in a bare repo using the repo's CI setup)
+* Testing in a variety of environments
+  * Different Node.js versions
+  * Different operating systems
+  * A multitude of CI providers
diff --git a/scripts/get-next-version.js b/scripts/get-next-version.js
@@ -1,5 +1,7 @@
 /* eslint-disable no-console */
 
+// See ../guides/next-version.md for documentation.
+
 const semver = require('semver')
 const Bluebird = require('bluebird')
 const bumpCb = require('conventional-recommended-bump')

diff --git a/scripts/test-other-projects.js b/scripts/test-other-projects.js
@@ -9,6 +9,8 @@ const { getInstallJson } = require('@cypress/commit-message-install')
 
 /* eslint-disable no-console */
 
+// See ../guides/testing-other-projects.md for documentation.
+
 const { npm, binary } = getNameAndBinary(process.argv)
 
 la(is.unemptyString(npm), 'missing npm url')