release process improvements

[calebcartwright]

(n.b. while anyone should feel free to share perspectives, this thread is largely intended for discussion amongst the rustfmt team and recurring contributors)

Rustfmt's primary (and only supported) release model is through the official Rust distribution channels; i.e. users are expected to grab it via Rustup along with their respective toolchain. As such, Rustfmt changes are ultimately "released" by synchronizing those changes into the main rust-lang/rust repo.

However, the current process has some surprising vectors of complexity and time consuming activities, many historical in nature, that make this more painful than, IMO, it could be.

Relevant background context

• Rustfmt is developed/triaged/etc. in a dedicated repo at rust-lang/rustfmt (recognize this is an obvious one 😄), and we're not going to change that/move everything into rust-lang/rust (there's no benefit to doing so, and several drawbacks)
• Rustfmt exists as a subtree in the rust-lang/rust repo, which means rustfmt source is sometimes modified by r-l/rust contributors (e.g. when compiler changes modify the AST and rustfmt+other tools in turn need to be updated to handle those AST changes)
• Rustfmt is currently versioned independently of Rust itself
• Rustfmt depends on several internal rustc modules (e.g. rustc_parse, rustc_ast, etc.) which are consumed from the sysroot with dynamic linking on rustc_driver. This adds some additional steps when building rustfmt from source, and as such if Rustfmt was also published on crates.io it would in turn would create some challenges/complexity for consumers of the crate(s)
• Rustfmt's changelog info is not currently included in the main Rust release notes/blog posts
• We don't currently capture the Rustfmt changes that occurred between Rust releases

Current release process

Note that these steps are sequenced in the order I typically follow, though there's not always a strict pre/post dependency between these list items (e.g. the changelog could be updated before the incoming subtree sync)

• Merge any open PRs in r-l/rustfmt that should be included in the next rustfmt "release" (note this is done first so that we can deal with any subsequent merge conflicts ourselves vs. potentially getting blocked waiting on the respective PR authors to resolve conflicts)
• Ensure latest rustc changes have been copied over to this repo, landing a PR that performs a subtree push from rust-lang/rust (e.g. Subtree sync  #5803). Notes:
  • this typically results in merge conflicts that have to be resolved
  • this sometimes require other changes be made to the rustfmt source and/or dependency tree
  • ensure the rustfmt toolchain file is bumped to an appropriate nightly toolchain
• Update the rustfmt changelog (e.g. prep v1.6.0 release #5804)
• Bump the rustfmt version number (e.g. prep v1.6.0 release #5804)
• Create a signed git tag off the rust-lang/rustfmt master branch and push it to this repo
• Pushing the tag will automatically kick off a GitHub Actions Workflow to validate the tag, so wait to ensure the upload workflow passes (this is to make sure the tag builds as expected, both so that it can be reused as a point in time snapshot and so that you know one of the latter steps in this process will work)
• Open a PR in r-l/rust to sync the rustfmt changes back into the compiler repo (via a subtree pull). Typically this should be done by setting the reviewer to @ghost or yourself to avoid spamming Mark/the infra team (though on occasion it may be beneficial or necessary to consult with them around timing)
• Wait for the all initial CI runs to pass in r-l/rust
• Give bors the approval for the sync PR with an explicit priority set and rollup disabled. The subtree sync PRs should generally have a priority between 1-5. I usually use a p=1, but before providing approval to bors I'll typically look at the queue to see what else is in flight and will increase the priority if necessary to jump in front of any other PRs in r-l/rust that are modifying rustfmt source
• Once the r-l/rust PR is merged, create a GitHub Release here in this repo using the tag created earlier, and add the corresponding changelog entries into the description
• The creation of the GitHub Release will automatically start another run of the Upload job that has some extra steps to create rustfmt binaries for some of the Tier 1 platforms, and then publishes those binaries on the GitHub release, for example: https://github.com/rust-lang/rustfmt/releases/tag/v1.6.0. Wait to make sure this Workflow passes and that the associated binaries are tacked on to the release.

---

As mentioned earlier, this can be a rather time consuming process from start to finish.

Yes it's true some of the steps are just "waiting" and there's some steps with variable duration (e.g. how long it takes for bors to start building). However, each and every step has to be performed, and there's a lot of varying factors that can complicate things at various steps (e.g. system modifications necessitating re-patching of git's subtree and having to rebuild the r-l/rust index), or even derail the process and require starting over (a different PR touching rustfmt source in r-l/rust landing)

I typically plan on needing to be able to be fully available for a minimum of 2-3 consecutive hours before I start this because any delays around any step in the process can greatly increase the odds of one of those external circumstances requiring the whole process be restarted. I've been burned several times trying to do a release in a piecemeal, spread out process, and as such strongly advise against it.

This is obviously problematic for many reasons, not the least of which being my bandwidth. Fortunately, I typically have multiple, shorter duration (e.g. 30 minute) windows of availability throughout the day which are great for triaging issues, helping users, and even working through parts of code reviews. However, those short windows scattered about are insufficient for going through the current release process, and it's increasingly uncommon for me to have a full 3 hours of consecutive availability.

We can certainly improve things a bit by incorporating more people, either another maintainer driving the entire process or a "pair release" type of exercise where we can split tasks when we have adjacent/overlapping availability. I'd like to pursue that in parallel (cc @ytmimi), though I'd also really like to focus on how we can improve the process itself.

I've been noodling on this for a while, and I think some of the lowest hanging fruit would be a couple new procedural items that could lessen the pain on existing steps:

• Grabbing changes from r-l/rust (subtree pushes) more frequently, including doing so independently of releases
  • The longer the delta between these, the greater the chances of a larger diff and resultant merge conflicts, so doing these more frequently should, in theory, lower the odds of having nasty merge conflicts during this part of the release process
• Tracking changelog/release notes items more proactively during development
  • I don't think we should go to the extent of requiring every contributor to add a changelog entry (e.g. under the "unreleased" section), but I also don't think we should continue punting this to release time. We try to add the release-notes label to PRs/issues that we believe warrant a changelog entry, and I think we should try to get in the habit of more frequently doing this and also updating the changelog in between and independent of releases
  • We could also look into attempting to automate this, as IIRC the cargo team has done some rather interesting work on this front for theirs

Beyond those procedural items, I think we could also look into whether certain steps could and should be removed. As an example, if we were to stop versioning rustfmt independently of Rust, then we'd be able to dump several steps and PRs (e.g. the versioning and potential tagging too since the main Rust release process could handle that).

I think we also need to seriously talk about dropping our GitHub Releases, at least the associated binaries (which aren't really viable anymore anyway (refs #5761, #5675 (comment), #5675 (comment))

cc @rust-lang/rustfmt @CosmicHorrorDev @fee1-dead @HarrisonHemstreet