Duplicated go test arguments on rerun, causing package is not in std

[matthiasr]

Given a failing test

test_test.go

package buildtag

import (
	"testing"
)

func TestTest(t *testing.T) {
	t.Error("failing")
}

when running gotestsum with multiple reruns and additional go test flags:

gotestsum --rerun-fails=4 --packages=./... -- -run TestTest -tags some_tag -json

we get the red-herring error

package some_tag is not in std (/opt/homebrew/Cellar/go/1.21.3/libexec/src/some_tag)

on the second(!) rerun.

full output

✖  . (109ms)

DONE 1 tests, 1 failure in 0.245s

✖  . (65ms)

DONE 2 runs, 2 tests, 2 failures in 0.444s

package some_tag is not in std (/opt/homebrew/Cellar/go/1.21.3/libexec/src/some_tag)

=== Failed
=== FAIL: . TestTest (0.00s)
    tag_test.go:8: failing

=== FAIL: . TestTest (re-run 1) (0.00s)
    tag_test.go:8: failing

=== Errors
package some_tag is not in std (/opt/homebrew/Cellar/go/1.21.3/libexec/src/some_tag)

DONE 3 runs, 2 tests, 2 failures, 1 error in 0.449s
ERROR rerun aborted because previous run had errors

Note that the reruns stop after 2, even though I specified 4.

I set up an interception of the go commands that gotestsum actually invokes, and these are interesting:

go test -run TestTest -tags some_tag -json ./...
go test -test.run=^TestTest$ -tags some_tag -json buildtag
go test -test.run=^TestTest$ -tags some_tag -json some_tag -json buildtag

how to intercept

Add a file `go` to some directory, say `./fake-go`, with the contents

#!/bin/sh
echo go "$@" >> commands.log
exec /opt/homebrew/bin/go "$@" 

If not using homebrew, insert the absolute path to go (command -v go).
Make it executable (chmod +x fake-go/go) and run gotestsum as

PATH="$(pwd)/fake-go:${PATH}" gotestsum …

Something goes wrong when massaging the test arguments for reruns, but only the second time.
I can work around this by using a one-argument form of the tags flag:

gotestsum --rerun-fails=4 --packages=./... -- -run TestTest -tags=some_tag -json

which correctly reruns.

full output

✖  . (231ms)

DONE 1 tests, 1 failure in 0.710s

✖  . (91ms)

DONE 2 runs, 2 tests, 2 failures in 0.947s

✖  . (84ms)

DONE 3 runs, 3 tests, 3 failures in 1.167s

✖  . (74ms)

DONE 4 runs, 4 tests, 4 failures in 1.372s

✖  . (87ms)

=== Failed
=== FAIL: . TestTest (0.00s)
    tag_test.go:8: failing

=== FAIL: . TestTest (re-run 1) (0.00s)
    tag_test.go:8: failing

=== FAIL: . TestTest (re-run 2) (0.00s)
    tag_test.go:8: failing

=== FAIL: . TestTest (re-run 3) (0.00s)
    tag_test.go:8: failing

=== FAIL: . TestTest (re-run 4) (0.00s)
    tag_test.go:8: failing

DONE 5 runs, 5 tests, 5 failures in 1.593s
0.84s user 1.34s system 131% cpu 1.646s total

The commands it invokes are still wrong but this no longer breaks go test:

go test -run TestTest -tags=some_tag -json ./...
go test -test.run=^TestTest$ -tags=some_tag -json buildtag
go test -test.run=^TestTest$ -tags=some_tag -json -tags=some_tag -json buildtag
go test -test.run=^TestTest$ -tags=some_tag -json -tags=some_tag -json buildtag
go test -test.run=^TestTest$ -tags=some_tag -json -tags=some_tag -json buildtag

It is worth noting that the argument list does not continue to grow after the second rerun.

[dnephin]

Thank you for the bug report! You can tell gotestsum to print the commands it is running with --debug as well. That may be easier than faking the go command.

It seems like there is probably an append bug here:

gotestsum/cmd/main.go

Line 358 in b8d5d8f

args = append(args[:runIndex], args[runIndexEnd+1:]...)

That modifies args, which also modifies the backing array of opts.args and opts.args is re-used again later.

This line:

gotestsum/cmd/main.go

Line 338 in b8d5d8f

args := opts.args

Should make a copy instead:

args := append([]string{}, opts.args...)