Automate management of broken tools

[nrc]

Rather than using the tools manifest for opting-in to breaking a tool.

The rough plan: on master (but not beta or stable) the CI would ignore failing to build or test some tools (RLS, Clippy, etc). However, some bot would note when a tool started to fail and email interested parties (and/or display the info in a dashboard).

This should lower the friction for contributors working in the Rust repo when they make tool-breaking changes. However, we might miss some 'true positive' failure in the tools (e.g., #45820).

[kennytm]

We could do it like this on the Travis CI side:

1. Change the purpose of the cargotest job and aux job.

   • The old cargotest job becomes the new aux job. It will run:
     • cargotest
     • pretty tests
   • The old aux job becomes the tools job. It will run test against external tools:
     • cargo, rls, rustfmt, miri, clippy

2. Don't run the tools job in auto branch. (Promote something to tier-2 to take its place)

   • Alternatively, still run the job but mark it as allow_failure. That's rather pointless though.

3. Run the tools job whenever the branch is master/beta/stable and not a PR.

   • We could also run this in a PR, so reviewer can inspect the ❌ and determine whether it is legit or spurious, to reduce the chance a "true positive" is missed. This will double the resource spent on testing PRs, and won't reflect the actual behavior when merged to master.

4. Monitor the Travis badge.

Rough sketch of the change:

diff --git a/.travis.yml b/.travis.yml
index db34f14044..a28954761b 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -161,16 +161,16 @@ matrix:
       if: branch = auto
     - env: IMAGE=i686-gnu-nopt
       if: branch = auto
-    # - env: IMAGE=wasm32 issue 42646
-    #   if: branch = auto
+    - env: IMAGE=wasm32
+      if: branch = auto
     - env: IMAGE=x86_64-gnu
       if: branch = auto
     - env: IMAGE=x86_64-gnu-full-bootstrap
       if: branch = auto
+    - env: IMAGE=x86_64-gnu-tools
+      if: branch IN (master, beta, stable) # (<- uncomment to disable PR test) AND type != pull_request
     - env: IMAGE=x86_64-gnu-aux
       if: branch = auto
-    - env: IMAGE=x86_64-gnu-cargotest
-      if: branch = auto
     - env: IMAGE=x86_64-gnu-debug
       if: branch = auto
     - env: IMAGE=x86_64-gnu-nopt

Not sure what will be the corresponding change on the AppVeyor side.

[kennytm]

Ideas after discussion with @nrc,

5. We will create a repository to store the current test status. The Travis+AppVeyor job will push a commit whenever the test result changes. The dev-tools team will create a dashboard to read from this repository.

6. The commit message will @-mention any party responsible for the tool.

[nrc]

We discussed this at the dev-tools meeting today. We mostly discussed the implementation details. One interesting issue that came up: we want to avoid notification except on changing from testing to not testing. That requires some state in the 'bot' doing the monitoring. One way to avoid security concerns is to use a GitHub repo (I've done this for rustaceans.org and findwork). We might then be able to rely on GitHub notifications for doing the email notification when a tool breaks.

[kennytm]

Tracker.

Changes to the rust-lang/rust CI system

• Ensure dist jobs can still function correctly even when a tool is broken (ignoring toolstate.toml) ([auto-toolstate][1/8] Always ignore build failure of failable tools (rls, rustfmt, clippy) #46102)

• Perform the cargotest → aux and aux → tools rotation as outlined in point 1 in Automate management of broken tools #45861 (comment). ([auto-toolstate][2+3/8] Move external tools tests into its own job with --no-fail-fast #46393)

• Ensure the tools jobs is not fail-fast; collect test result into a local machine-readable file (Probably just a CSV file so that it can be append-only). ([auto-toolstate][2+3/8] Move external tools tests into its own job with --no-fail-fast #46393)

• Create a GitHub repository to store the test result, and a secret bot token to push to this repository.

• Let tools job upload the test result:

  • If the current channel is not nightly (e.g. beta or stable), exit the CI job so bors can prevent it merging.
  • If the current channel is nightly, append the test result and push to the toolstate-records repository. Since both AppVeyor and Travis will push to the same repository, the compare-and-push action must be --ff-only and needs several retries with jitter.

• Create a new lightweight CI job on master that promotes a result from the toolstate-records repo. If the result differ, create a commit with @-mentions.

• Update rustbuild to remove traces of toolstate.toml.

• Update documentation.

(Note: we need to disable tools jobs from auto before enabling @-mentions to prevent spurious mentions due to failing builds.)

Other changes

• Build a dashboard using the content of the new repository?

[nrc]

There doesn't seem to be anywhere else to track it, so #47146 is another example of a PR breaking rustfmt, and causing a real bug, which should have been noticed by the author if tool bustage was not handled automatically.

[kennytm]

Reopened as tracking issue.

(The breakage from #47146 is noted as soon as the PR is merged: rust-lang-nursery/rust-toolstate@1b316cc)

[kennytm]

Follow-up changes:

• Use a proper link for the issue number rust-lang-nursery/rust-toolstate#1: Use a proper link for the issue number (i.e. https://github.com/rust-lang/rust/pulls/12345 instead of #12345)
• Comment on the breaking PR rust-lang-nursery/rust-toolstate#2: Leave a comment on the PR which regressed a tool (may need to regenerate an API token for commenting?)
• Track the 6-week cycle. If a tool is regressed at the last week of the 6-week cycle, do not merge.
  • Set up a cron job to email the stakeholders if a tool is still not test-pass before entering the last week of this 6-week cycle.
• (?) Migrate cargo tests back to tools job (but mandatory pass is required). This is to rebalance the time taken for the check-aux job.
• Support docs in rust-toolstate.

[steveklabnik]

Triage: only one or two boxes unchecked, but I'm not 100% sure what the current status is.

[Mark-Simulacrum]

Pretty sure this is basically all done or not needed today, though there is ongoing work in this area (subtrees making toolstate entirely unnecessary, as well as #65000