Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automate filing issues based on failing tests in CI runs #799

Open
RikkiGibson opened this issue Jul 29, 2020 · 1 comment
Open

Automate filing issues based on failing tests in CI runs #799

RikkiGibson opened this issue Jul 29, 2020 · 1 comment

Comments

@RikkiGibson
Copy link
Contributor

I have found that the process of opening an informative issue about a test failure in a CI run (particularly a merge PR where the test failures are often due to flakiness) is a little bit cumbersome and full of manual steps. I would like to come up with an automation that we can run on a build with failing tests whose output is one or more URLs, which when visited, opens up the new issue page pre-populated with a title and description. For example, https://github.com/dotnet/roslyn-tools/issues/new?title=My%20issue&body=Some%20description

The issue should contain the name of the test, a link to a failing build, and the Xunit output for the test. For example, dotnet/roslyn#46414.

Ideally we could link to the source code of the failing test as well, but generating a search link from the test name and putting it in the issue description might be sufficient.

/cc @JoeRobich @dibarbet

@sharwell
Copy link
Member

I've been wanting a feature like this for a while, but it's critical that it has a few additional things:

  1. The ability to link a build job failure to the cause of a non-test failure (e.g. network issue, compilation failure, etc.)
  2. Ability to link a test crash (e.g. stack overflow) to an issue
  3. Ability to assign the job failure to an issue that already exists
  4. Ability to list all job failures which have not yet been assigned to an issue
  5. Ability to keep track of the number of jobs impacted by the issue (otherwise the issue is treated as a one-off failure with no need to take action)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants