cmd/test2json: Fails to mark a test as failed when using os.Exit

[dlsniper]

What version of Go are you using (go version)?

$ go version
go version go1.16beta1 windows/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
set GO111MODULE=on
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\Florin\AppData\Local\go-build
set GOENV=C:\Users\Florin\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=D:\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=D:\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=D:\go-sdks\go1.16beta1
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=D:\go-sdks\go1.16beta1\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.16beta1
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=D:\GoLandProjects\awesomeProject37\go.mod
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\Florin\AppData\Local\Temp\go-build898034260=/tmp/go-build -gno-record-g
cc-switches

What did you do?

I used the following test scenario

package main

import (
	"os"
	"testing"
)

func TestDemo(t *testing.T) {
	t.Log("This is a demo")
	os.Exit(0)
}

When I run the command below, the test fails, as documented in the release notes

go test -v ./...
=== RUN   TestDemo
    main_test.go:9: This is a demo
--- FAIL: TestDemo (0.00s)
panic: unexpected call to os.Exit(0) during test [recovered]
        panic: unexpected call to os.Exit(0) during test

goroutine 19 [running]:
testing.tRunner.func1.2(0x9801e0, 0x9d7090)
        D:/go-sdks/go1.16beta1/src/testing/testing.go:1144 +0x345
testing.tRunner.func1(0xc000158300)
        D:/go-sdks/go1.16beta1/src/testing/testing.go:1147 +0x4b6
panic(0x9801e0, 0x9d7090)
        D:/go-sdks/go1.16beta1/src/runtime/panic.go:965 +0x1c7
os.Exit(0x0)
        D:/go-sdks/go1.16beta1/src/os/proc.go:68 +0x74
awesomeProject37.TestDemo(0xc000158300)
        D:/GoLandProjects/awesomeProject37/main_test.go:10 +0x7d
testing.tRunner(0xc000158300, 0x9b4e90)
        D:/go-sdks/go1.16beta1/src/testing/testing.go:1194 +0xef
created by testing.(*T).Run
        D:/go-sdks/go1.16beta1/src/testing/testing.go:1239 +0x2b3
FAIL    awesomeProject37        0.270s
FAIL

However, when I run:

go test -c -o demo.exe ./...
go tool test2json -t demo.exe -test.v
{"Time":"2020-12-18T13:09:04.2031136+02:00","Action":"run","Test":"TestDemo"}
{"Time":"2020-12-18T13:09:04.2047196+02:00","Action":"output","Test":"TestDemo","Output":"=== RUN   TestDemo\n"}
{"Time":"2020-12-18T13:09:04.2052725+02:00","Action":"output","Test":"TestDemo","Output":"    main_test.go:9: This is a demo\n"}
{"Time":"2020-12-18T13:09:04.2166141+02:00","Action":"pass","Test":"TestDemo","Elapsed":0.207}

The test passes.

What did you expect to see?

I expect to see a test failure when using go test -c too.

What did you see instead?

I see the tests passing.

[dlsniper]

@ianlancetaylor can we please prioritize this to Go 1.16 instead? This is a new feature added to Go 1.16 and everyone using test2json will have different behavior than those that don't use it. For example, this is a problem for those using GoLand to run tests, as we rely on the test2json command to produce correct data to display to end-users.

[ianlancetaylor]

The fix is to also use -test.paniconexit0, as in

go tool test2json -t demo.exe -test.v -test.paniconexit0

I'll update the docs. Invoking go tool test2json directly is an unusual case.

[gopherbot]

Change https://golang.org/cl/283873 mentions this issue: cmd/test2json: document passing -test.paniconexit0

[bcmills]

This is for #29062.