cmd/go: git shallow fetches broken at CL 556358 #66147

bcmills · 2024-03-06T22:31:23Z

Go version

go version devel go1.23-7986e26a39 Mon Mar 4 18:59:50 2024 +0000 linux/amd64

Output of `go env` in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/usr/local/google/home/bcmills/.cache/go-build'
GOENV='/usr/local/google/home/bcmills/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/usr/local/google/home/bcmills/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/usr/local/google/home/bcmills'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/google/home/bcmills/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='local'
GOTOOLDIR='/usr/local/google/home/bcmills/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='devel go1.23-7986e26a39 Mon Mar 4 18:59:50 2024 +0000'
GODEBUG=''
GCCGO='/usr/bin/gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='c++'
CGO_ENABLED='1'
GOMOD='/usr/local/google/home/bcmills/go/src/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build954033280=/tmp/go-build -gno-record-gcc-switches'

What did you do?

$ go clean -modcache
$ GOPROXY=direct go mod download -json gerrit.wikimedia.org/r/mediawiki@v0.0.0-20240202145822-67da0cbcfdf7

What did you see happen?

$ GOPROXY=direct go mod download -x -json gerrit.wikimedia.org/r/mediawiki@v0.0.0-20240202145822-67da0cbcfdf7
# get https://gerrit.wikimedia.org/r/mediawiki?go-get=1
# get https://gerrit.wikimedia.org/r/mediawiki?go-get=1: 200 OK (0.066s)
mkdir -p /tmp/tmp.eR42iY5yuP/.gopath/pkg/mod/cache/vcs # git3 https://gerrit.wikimedia.org/r/mediawiki
# lock /tmp/tmp.eR42iY5yuP/.gopath/pkg/mod/cache/vcs/2392d99bd78679c31c3fa1a4a878b61b45bbf0cdee531116b849df5fa3954d79.lock
mkdir -p /tmp/tmp.eR42iY5yuP/.gopath/pkg/mod/cache/vcs/2392d99bd78679c31c3fa1a4a878b61b45bbf0cdee531116b849df5fa3954d79 # git3 https://gerrit.wikimedia.org/r/mediawiki
cd /tmp/tmp.eR42iY5yuP/.gopath/pkg/mod/cache/vcs/2392d99bd78679c31c3fa1a4a878b61b45bbf0cdee531116b849df5fa3954d79; git init --bare
0.010s # cd /tmp/tmp.eR42iY5yuP/.gopath/pkg/mod/cache/vcs/2392d99bd78679c31c3fa1a4a878b61b45bbf0cdee531116b849df5fa3954d79; git init --bare
cd /tmp/tmp.eR42iY5yuP/.gopath/pkg/mod/cache/vcs/2392d99bd78679c31c3fa1a4a878b61b45bbf0cdee531116b849df5fa3954d79; git remote add origin -- https://gerrit.wikimedia.org/r/mediawiki
0.006s # cd /tmp/tmp.eR42iY5yuP/.gopath/pkg/mod/cache/vcs/2392d99bd78679c31c3fa1a4a878b61b45bbf0cdee531116b849df5fa3954d79; git remote add origin -- https://gerrit.wikimedia.org/r/mediawiki
cd /tmp/tmp.eR42iY5yuP/.gopath/pkg/mod/cache/vcs/2392d99bd78679c31c3fa1a4a878b61b45bbf0cdee531116b849df5fa3954d79; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' 67da0cbcfdf7 --
0.007s # cd /tmp/tmp.eR42iY5yuP/.gopath/pkg/mod/cache/vcs/2392d99bd78679c31c3fa1a4a878b61b45bbf0cdee531116b849df5fa3954d79; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' 67da0cbcfdf7 --
cd /tmp/tmp.eR42iY5yuP/.gopath/pkg/mod/cache/vcs/2392d99bd78679c31c3fa1a4a878b61b45bbf0cdee531116b849df5fa3954d79; git ls-remote -q origin
0.116s # cd /tmp/tmp.eR42iY5yuP/.gopath/pkg/mod/cache/vcs/2392d99bd78679c31c3fa1a4a878b61b45bbf0cdee531116b849df5fa3954d79; git ls-remote -q origin
cd /tmp/tmp.eR42iY5yuP/.gopath/pkg/mod/cache/vcs/2392d99bd78679c31c3fa1a4a878b61b45bbf0cdee531116b849df5fa3954d79; git tag -l
0.005s # cd /tmp/tmp.eR42iY5yuP/.gopath/pkg/mod/cache/vcs/2392d99bd78679c31c3fa1a4a878b61b45bbf0cdee531116b849df5fa3954d79; git tag -l
cd /tmp/tmp.eR42iY5yuP/.gopath/pkg/mod/cache/vcs/2392d99bd78679c31c3fa1a4a878b61b45bbf0cdee531116b849df5fa3954d79; git fetch -f -c protocol.version=2 --depth=1 origin 67da0cbcfdf7f6e1c40e9d145df77a088fb6231e:refs/dummy
0.006s # cd /tmp/tmp.eR42iY5yuP/.gopath/pkg/mod/cache/vcs/2392d99bd78679c31c3fa1a4a878b61b45bbf0cdee531116b849df5fa3954d79; git fetch -f -c protocol.version=2 --depth=1 origin 67da0cbcfdf7f6e1c40e9d145df77a088fb6231e:refs/dummy
cd /tmp/tmp.eR42iY5yuP/.gopath/pkg/mod/cache/vcs/2392d99bd78679c31c3fa1a4a878b61b45bbf0cdee531116b849df5fa3954d79; git fetch -f origin 'refs/heads/*:refs/heads/*' 'refs/tags/*:refs/tags/*'
0.122s # cd /tmp/tmp.eR42iY5yuP/.gopath/pkg/mod/cache/vcs/2392d99bd78679c31c3fa1a4a878b61b45bbf0cdee531116b849df5fa3954d79; git fetch -f origin 'refs/heads/*:refs/heads/*' 'refs/tags/*:refs/tags/*'
cd /tmp/tmp.eR42iY5yuP/.gopath/pkg/mod/cache/vcs/2392d99bd78679c31c3fa1a4a878b61b45bbf0cdee531116b849df5fa3954d79; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' 67da0cbcfdf7f6e1c40e9d145df77a088fb6231e --
0.006s # cd /tmp/tmp.eR42iY5yuP/.gopath/pkg/mod/cache/vcs/2392d99bd78679c31c3fa1a4a878b61b45bbf0cdee531116b849df5fa3954d79; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' 67da0cbcfdf7f6e1c40e9d145df77a088fb6231e --
{
        "Path": "gerrit.wikimedia.org/r/mediawiki",
        "Version": "v0.0.0-20240202145822-67da0cbcfdf7",
        "Error": "gerrit.wikimedia.org/r/mediawiki@v0.0.0-20240202145822-67da0cbcfdf7: invalid version: unknown revision 67da0cbcfdf7f6e1c40e9d145df77a088fb6231e",
        "Origin": {
                "VCS": "git",
                "URL": "https://gerrit.wikimedia.org/r/mediawiki",
                "Ref": "67da0cbcfdf7f6e1c40e9d145df77a088fb6231e",
                "RepoSum": "r1:sSchJC9s3IXQjz9XqT6meZ2hU1YmTimbSLTEdXEHdYQ="
        }
}

What did you expect to see?

Successful fetch, as with previous go versions.

It appears that I put the arguments in the wrong order in https://go.dev/cl/556358, inadventently breaking shallow git fetches. That went undiagnosed, because it fixed the symptom I was testing for (#56881) and we didn't have any tests specifically probing for shallowness of git repos in the module cache.

(CC @suzmue @matloob @samthanawalla)

The text was updated successfully, but these errors were encountered:

bcmills · 2024-03-06T22:32:32Z

Notably, following the commands in that -x trace:

/tmp/mediawiki$ git fetch -f -c protocol.version=2 --depth=1 origin 67da0cbcfdf7f6e1c40e9d145df77a088fb6231e:refs/dummy
error: unknown switch `c'

gopherbot · 2024-03-06T23:01:51Z

Change https://go.dev/cl/569422 mentions this issue: cmd/go/internal/modfetch: pass "-c" arguments to git before subcommands

bcmills · 2024-03-11T22:10:01Z

@gopherbot, please backport to Go 1.22. This was an accidental regression in https://go.dev/cl/556358, and may cause increased client and server load when fetching modules directly from git repositories.

gopherbot · 2024-03-11T22:10:49Z

Backport issue(s) opened: #66256 (for 1.22).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

gopherbot · 2024-03-11T22:12:34Z

Change https://go.dev/cl/570737 mentions this issue: cmd/go/internal/modfetch: pass "-c" arguments to git before subcommands

… to git before subcommands I accidentally transposed the arguments in CL 556358, causing the shallow 'git fetch' attempt to always fail. That didn't break any tests because we fall back to a full fetch, which works for nearly all real Git servers, and we didn't have a test that checked for shallow fetches. Tested manually using: GOPROXY=direct go mod download -x -json gerrit.wikimedia.org/r/mediawiki@v0.0.0-20240202145822-67da0cbcfdf7 (I'm still thinking about how to add a proper regression test.) Fixes #66256. Updates #66147. Change-Id: I0bb17283bae856f369fd24f29375e507d0999933 Reviewed-on: https://go-review.googlesource.com/c/go/+/569422 Auto-Submit: Bryan Mills <bcmills@google.com> Reviewed-by: Michael Matloob <matloob@golang.org> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Run-TryBot: Bryan Mills <bcmills@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> (cherry picked from commit 2ab9218) Reviewed-on: https://go-review.googlesource.com/c/go/+/570737 Reviewed-by: Sam Thanawalla <samthanawalla@google.com>

bcmills added NeedsFix The path to resolution is known, but the work has not been done. GoCommand cmd/go modules labels Mar 6, 2024

bcmills self-assigned this Mar 6, 2024

bcmills added this to the Go1.23 milestone Mar 6, 2024

bcmills changed the title ~~import/path: issue title~~ cmd/go: git shallow fetches broken at CL 556358 Mar 6, 2024

gopherbot mentioned this issue Mar 11, 2024

cmd/go: git shallow fetches broken at CL 556358 [1.22 backport] #66256

Closed

gopherbot closed this as completed in 2ab9218 Mar 11, 2024

gabyhelp mentioned this issue Jul 9, 2024

cmd/go: support fetching git repos with sha256 hashes #68359

Open

gabyhelp mentioned this issue Aug 17, 2024

cmd/go: support git/subcommand invocations for git9 on plan9 #68926

Closed

gabyhelp mentioned this issue Sep 28, 2024

import/path: issue title go get not work normal #69697

Closed

gabyhelp mentioned this issue Nov 9, 2024

cmd/go: go build -mod=readonly still updates a go.work.sum file #70272

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/go: git shallow fetches broken at CL 556358 #66147

cmd/go: git shallow fetches broken at CL 556358 #66147

bcmills commented Mar 6, 2024

bcmills commented Mar 6, 2024

gopherbot commented Mar 6, 2024

bcmills commented Mar 11, 2024

gopherbot commented Mar 11, 2024

gopherbot commented Mar 11, 2024

cmd/go: git shallow fetches broken at CL 556358 #66147

cmd/go: git shallow fetches broken at CL 556358 #66147

Comments

bcmills commented Mar 6, 2024

Go version

Output of go env in your module/workspace:

What did you do?

What did you see happen?

What did you expect to see?

bcmills commented Mar 6, 2024

gopherbot commented Mar 6, 2024

bcmills commented Mar 11, 2024

gopherbot commented Mar 11, 2024

gopherbot commented Mar 11, 2024

Output of `go env` in your module/workspace: