cmd/go: go list fails with submodules which have test-only dependencies

[smira]

What version of Go are you using (go version)?

$ go version
go1.20.5

Does this issue reproduce with the latest release?

yes, I believe it is related to #60001

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/smira/.cache/go-build"
GOENV="/home/smira/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/smira/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/smira/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/smira/sdk/go1.20.5"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/smira/sdk/go1.20.5/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.20.5"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/smira/Documents/go-test-package-failure/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build337276505=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I hit this on a big repository, but I reduced this down to https://github.com/smira/go-test-package-failure.

There's a submodule which has an import of a package from _test.go file.

What did you expect to see?

Switching to older version of Go "fixes" it:

$ ~/go/bin/go1.20.3 list -u -m -json all
{
	"Path": "github.com/smira/go-test-package-failure",
	"Main": true,
	"Dir": "/home/smira/Documents/go-test-package-failure",
	"GoMod": "/home/smira/Documents/go-test-package-failure/go.mod",
	"GoVersion": "1.20"
}
{
	"Path": "github.com/santhosh-tekuri/jsonschema/v5",
	"Version": "v5.3.0",
	"Time": "2023-04-04T18:54:33Z",
	"Indirect": true,
	"Dir": "/home/smira/go/pkg/mod/github.com/santhosh-tekuri/jsonschema/v5@v5.3.0"
}
{
	"Path": "github.com/smira/go-test-failure/subpackage",
	"Version": "v0.0.0-00010101000000-000000000000",
	"Replace": {
		"Path": "./subpackage",
		"Dir": "/home/smira/Documents/go-test-package-failure/subpackage",
		"GoMod": "/home/smira/Documents/go-test-package-failure/subpackage/go.mod",
		"GoVersion": "1.20"
	},
	"Dir": "/home/smira/Documents/go-test-package-failure/subpackage",
	"GoMod": "/home/smira/Documents/go-test-package-failure/subpackage/go.mod",
	"GoVersion": "1.20"
}

What did you see instead?

In the checkout, the following fails:

$ ~/go/bin/go1.20.5 list -u -m -json all
go: updates to go.sum needed, disabled by -mod=readonly

[smira]

Looking more into it, it might be related to github.com/santhosh-tekuri/jsonschema/v5 package, but I'm not sure what's exactly wrong with it

[smira]

Found a hot fix

I believe go mod tidy would remove this from go.mod, as it's unused.

[smira]

@Nasfame I can't reproduce it, go mod tidy removes the line from go.sum and things keep failing.

Basically the bug is that go mod tidy and go list are in disagreement on what should be in go.sum.

[dr2chase]

@bcmills @ianlancetaylor @rsc

[bcmills]

Duplicate of #56222

[smira]

I'm not sure why it's a duplicate, the problem comes only with #56222 fix backported to Go 1.20 in Go 1.20.5. It worked fine before that.

[bcmills]

Ah, I had missed that detail. Yes, it seems that the backported fix needs an extra condition for when to save the checksums.

[lespea]

I see this is marked for 1.21, when this is resolved will this be backported to 1.20.6? This is breaking a lot of our tooling and we won't be able to update to 1.21+ for the foreseeable future giving that go is dropping support for a lot of operating system versions.

[bcmills]

Yes, I intend for it to be. (That, or we will roll back part of all of #60001 and #60000.)

[gopherbot]

Change https://go.dev/cl/502015 mentions this issue: cmd/go: omit checksums for go.mod files needed for go version lines more often in pre-1.21 modules

[bcmills]

@gopherbot, please backport to Go 1.19 and 1.20. This is a regression introduced in a previous patch release.

[gopherbot]

Backport issue(s) opened: #60697 (for 1.19), #60698 (for 1.20).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

[bcmills]

In the meantime you can work around this with one of the following:

• build from source and patch in the proposed fix CLs (once I've resolved the merge conflicts for the backports)
• set -mod=mod in the go list command (to allow it to update the go.sum file)
• run go mod tidy -compat=1.16 (instead of just go mod tidy) to record all of the needed checksums upfront.

[gopherbot]

Change https://go.dev/cl/502016 mentions this issue: [release-branch.go1.20] cmd/go: omit checksums for go.mod files needed for go version lines more often in pre-1.21 modules

[gopherbot]

Change https://go.dev/cl/502017 mentions this issue: [release-branch.go1.19] cmd/go: omit checksums for go.mod files needed for go version lines more often in pre-1.21 modules