cmd/go: go.sum checksum different on Windows vs. Linux

[kjk]

What version of Go are you using (go version)?

$ go version
go version go1.12.7 windows/amd64

$go version
go version go1.12.4 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\kjk\AppData\Local\go-build
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=C:\Users\kjk\go
set GOPROXY=
set GORACE=
set GOROOT=c:\go
set GOTMPDIR=
set GOTOOLDIR=c:\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=C:\Users\kjk\src\apps\offdocs\go.mod
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\kjk\AppData\Local\Temp\go-build015649226=/tmp/go-build -gno-record-gcc-switches

What did you do?

I have a program that imports github.com/alecthomas/chroma. I've created go.sum on Windows (by building a program with go 1.2.7) and checked it in.

I checked it out on Linux (Ubuntu 18.04) and tried to build it with go 1.2.4.

What did you expect to see?

It should work.

What did you see instead?

It failed with:

verifying github.com/alecthomas/chroma@v0.6.5: checksum mismatch
        downloaded: h1:1gZ11dfxovHkbAM+xzyxjgxhG3Gr7U/cfOXI2eoEBkU=
        go.sum:     h1:4CN9fmMfN/k6Ppb+djxJqu3BG7WuLkct0dvScOc0w2w=

I fixed it by deleting go.sum but after a build it was indeed different on Linux than on Windows:

-github.com/alecthomas/chroma v0.6.5 h1:4CN9fmMfN/k6Ppb+djxJqu3BG7WuLkct0dvScOc0w2w=
+github.com/alecthomas/chroma v0.6.5 h1:1gZ11dfxovHkbAM+xzyxjgxhG3Gr7U/cfOXI2eoEBkU=

Please note I know little about how go.sum works but here's my theory:

• go build calculates checksum in go.sum as hash of file content
• go build downloads repositories using git installed on the system
• git can change newlines depending on core.autocrlf settings, changing the content and therefore changing the hash

On Windows I have:

PS C:\Users\kjk\src\apps\offdocs> git config --list
core.autocrlf=true

On Linux I have:

kjk@kjk-thinkpad25:~/src/onlinetools$ git config --list
core.autocrlf=false

Presumably, we would prefer go.sum not be affected by local value of git's core.autocrlf setting.

I imagine it would be possible for go tool to invoke git in such a way as to over-ride local setting and ensure that core.autocrlf=false when downloading dependent repos locally for module cache.

[kjk]

Or it could be a stale module cache / hash generated with earlier go version.
Apparently there was a change in how hash was calculated.
I failed to create a reproducible by trying to force this. I created:

main.go:

package main

import (
	"fmt"
	_ "github.com/alecthomas/chroma"
)

func main() {
	fmt.Printf("Hello\n")
}

go.mod:

module tst

go 1.12

require github.com/alecthomas/chroma v0.6.5

And a repro.bat:

del /s /q go.sum
git config --unset-all core.autocrlf

git config --add core.autocrlf  true
git config --get core.autocrlf

go clean -modcache
go build .
copy go.sum go.sum1

del /s /q go.sum

git config --unset-all core.autocrlf
git config --add core.autocrlf false
git config --get core.autocrlf

go clean -modcache
go build .
copy go.sum go.sum2

The theory was to re-create go.sum with core.autocrlf=true and core.autocrlf=false but it seems in both cases I get the same:

github.com/alecthomas/chroma v0.6.5 h1:1gZ11dfxovHkbAM+xzyxjgxhG3Gr7U/cfOXI2eoEBkU=

But maybe I'm just not reproducing whatever git settings that led to creating:

github.com/alecthomas/chroma v0.6.5 h1:4CN9fmMfN/k6Ppb+djxJqu3BG7WuLkct0dvScOc0w2w=

One way to confirm the "format change" theory would be to rerun this with a version of go before the go.sum format change and see what hash it produces for github.com/alecthomas/chroma.

I don't know when this change happened.

[jayconrod]

There was a change in the way sums were calculated between go1.11.3 and go1.11.4 (we will ensure that never happens again). Is it possible the h1:4CN9... sum or the module it was derived from was from before that? Is there any way to reproduce this after cleaning the cache (go clean -modcache)?

The checksum algorithm explicitly sets core.autocrlf to avoid problems with line endings, so that shouldn't be an issue.

For the record, h1:1gZ11dfxovHkbAM+xzyxjgxhG3Gr7U/cfOXI2eoEBkU= is the correct sum (per sum.golang.org).

[kjk]

It's possible. I imagine running go build with go <1.11.3 on the sample program in #33178 (comment) and checking the checksum would resolve this for sure.

I don't have a setup with go 1.11.3 thought.

[FiloSottile]

You can get Go 1.11.3 with go get golang.org/dl/go1.11.3 alongside your system Go.

[kjk]

It's still h1:1gZ11dfxovHkbAM+xzyxjgxhG3Gr7U/cfOXI2eoEBkU= with go 1.11 (as best as I can tell).

[bcmills]

There was an earlier extraction bug (#27153) fixed in Go 1.11.2, and an even earlier one way back in the vgo phase (#26229). Any idea how old the module cache on your Windows machine is?

[bcmills]

Also note that before Go 1.12, we had no locking or synchronization on the module cache. Is it possible that the cache entry for this module was corrupted due to concurrent go commands?

[kjk]

Yes, it's possible it comes from corrupted cache.

I've ran go clean -modcache since so no idea when that cache was corrupted.

It's fine to close this bug as I see no way to make progress determining the root cause.

[bcmills]

Thanks. Do let us know if you see this again going forward, especially if you discover a clue as to the underlying cause.