No easy way to analyze the results of a crater run

[nikomatsakis]

When a crater run occurs, I am pointed to a nice web page. This web page has a lot of nice links. Now I have to go through those links to decide which of the errors are spurious, which ones really matter, etc. This is kind of tedious though, for a few reasons.

First off, there is no easy way to get a "template" in which I have all the crate names listed. I'd like to be able to have a kind of "markdown table" prepopulated with the crate names and links to the output, so that I can update this table with a summary of what I found.

In the "old school crater" days, the output was a markdown file, so I found it relatively easy to copy-and-paste the list of regressions into an issue and then edit the GH comment repeatedly with the results.

(Even better than having to click on links, though, would be if crater could try collect the errors from failed runs via grep, and perhaps even sort the regressions into bins for me based on what compiler error messages they got.)

[Mark-Simulacrum]

https://github.com/Mark-Simulacrum/crater-generate-report is at least somewhat helpful here, hopefully

[Mark-Simulacrum]

A few pain points I've come across (some are solved-ish by the generate-report tool, but not perfectly):

Analyzing regressed logs means opening dozens of tabs

This is somewhat painful to keep track of. Particularly on runs with ~60-70 failed crates, going through them one by one is often pretty annoying since you have to keep track of the last one you opened and such. This is a relatively minor pain point, but one that affects almost every run and in a pretty painful way.

No good way to check for spurious failures

It would be huge to just rerun only the regressed crates automatically, say, 3 times, and filter out any that didn't get marked as regressed all N times.

Reproducing build failures locally

Building the docker container locally takes forever, so it'd be great if we could just publish it ourselves (as a tarball or something).

Generate regression report issue bits

It'd be great to have some copy/paste-able snippets that can be put into GitHub (e.g. description of rust-lang/rust#61477). crater-generate-report is pretty helpful here, but it'd be good to integrate that directly into the crater index page UI and have "copy" button for the https://crater-reports.s3.amazonaws.com/1.30-beta.15/index.html with crate name, link to crates.io or GH (for crate source, ideally with version etc); markdown log links, and most importantly cc for the authors.

[pietroalbini]

Building the docker container locally takes forever, so it'd be great if we could just publish it ourselves (as a tarball or something).

Wouldn't this work?

docker pull rustops/crates-build-env

[Mark-Simulacrum]

I believe right now to run the container properly (i.e., with security settings and such) we need to specify a bunch of flags to it. If there was e.g. a bash script that did that for triagers that would already be wonderful.

[Centril]

I made another program, https://github.com/Centril/crater-cat-errors/, which allows you to triage crater runs which emit several distinct errors (and you want to segregate them). See rust-lang/rust#63247 (comment) for an example report.