proposal: better go test outputs

[laurentsenta]

During my work on metadata in gateway-conformance, I found a proposal for golang to add metadata during tests, the author parses test2json outputs to make go tests nicer in CI.

Turns out github provides many interesting workflow commands for outputs like debug message, log lines groupping, etc.

I started prototyping updating uci with a go output parser, which generates groups and nicer outputs. Example:

Before	After

I tried this in kubo and go-libp2p.

I'd like to invest some more time on this, but would like input from other team members first.

Using gotestfmt

I am using gotestfmt. It has a few nice properties:

• A tokenizer/parser that should be easy to extend after a few refactors
• Templates specific to the execution environment, there is one for github CI that uses workflow commands for example
• You pipe the test outputs through it (go test -json .... | tee testoutput.json | gotestfmt) which means results can be rendered while the test is being processed (useful in CI and local dev). It's also well suited to chaining tools and transformations.

It's not a test runner like gotestsum, which is more unix-ish like. (we could replace gotestsum with https://github.com/jstemmer/go-junit-report I think).

(related #494)

Rough Plan

• get input from the team
• make a few improvements to the gotestfmt
• install this in uci
• install this in Kubo
• search for something similar for the JS and/or Rust stack.

[dnephin]

Hello! Author of gotestsum here. I'm also interested in adding an output format that uses the github actions folding, similar to gotestfmt. I am always looking for new ideas and ways to improve gotestsum! If you have anything you'd like to share, your feedback would be very welcome. If you were interested in contributing a github actions output format I would be very happy to help make that happen. It's very easy to add new formats, I have a proof of concept implementation of a github actions format in gotestyourself/gotestsum#315 (example output), and the implementation is ~20 lines of Go.

jstemmer/go-junit-report has been around for a long time and predates the go test -json output. It looks like it has now been updated to read that output too so also benefits from test2json parsing. Does it provide a custom format, our would you have to pipe the output to a second tool to handle the github actions code folding? It used to be that you had to output the full go test -v output which was sometimes too verbose to be useful in CI.

gotestfmt is very new. I tried it out once and found it still had a lot of rough edges. For example, if I look at the links you share, expanding the folds for the failed tests and some of the skipped tests does not show any output. From my understanding it also implements a custom parser, which seems to not be all that reliable yet. It's possible these bugs will be or can be fixed. In general I am weary of custom parsing of tests and like to rely on the stdlib test2json parsing of test output as much as possible.

Both of these tools use pipes of the output stream to collect output. While that approach may sound appealing, I've found in practice (even before writing gotestsum) that the pipe approach is not ideal for this use case. For one you either miss the build errors sent to stderr, or you have to merge the two streams using 2>&1 which makes parsing of the merged stream much more error prone. Reading the streams independently makes it much easier to identify build failures and allows you to use the stdlib test2json parsing.

The second common problem with the pipe approach is that it's easier to lose the correct error code. gotestsum always exits with the same exit code from go test -json. That's easy to ensure when you run the binary as a subcommand. With a pipe you have to be careful to use set -o pipefail (in the shell) or trust the output is always parsed correctly. The exit code from go test is more reliable than parsing hundreds or thousands of lines of a merged output stream.

If you really wanted to, you could use gotestsum in a pipe with

go test -json | gotestsum --junitfile junit.xml --format=standard-json --summary=none --raw-command -- cat | gotestfmt

But then you lose the reliability of the exit code. It is not something I would recommend, but it's essentially equivalent.

[laurentsenta]

Thanks for reaching out and the details @dnephin fwiw the team has been using gotestsum for a few years, really lovely to see you around, thanks for maintaining the project. I'll look into your PR, happy to contribute if we can get this released.

[galargh]

This issue was transferred to ipdxco/unified-github-workflows#50 in preparation for archiving of this repository.