storage: genproto/googleapis/storage/v2 QueryWriteStatusResponse.GetCommittedSize undefined

[chrisntb]

Storage client does not compile. It uses "QueryWriteStatusResponse.GetCommittedSize" and "Object_CustomerEncryption.GetKeySha256" from "google.golang.org/genproto/googleapis/storage/v2" and they were removed "v0.0.0-20211015135405-485ec31e706e Oct 15, 2021", see https://pkg.go.dev/google.golang.org/genproto@v0.0.0-20211015135405-485ec31e706e/googleapis/storage/v2?tab=versions

Client

Storage

Environment

Linux

Go Environment

go version go1.17.1 linux/amd64

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/.../.cache/go-build"
GOENV="/home/.../.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/.../go/pkg/mod"
GONOPROXY="github.com/.../"
GONOSUMDB="github.com/.../"
GOOS="linux"
GOPATH="/home/.../go"
GOPRIVATE="github.com/.../*"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/.../tools/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/.../go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.17.1"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home.../go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2973016987=/tmp/go-build -gno-record-gcc-switches"

Code

e.g.

storagepb "google.golang.org/genproto/googleapis/storage/v2"

...

func (w *Writer) queryProgress() (int64, error) {
	q, err := w.o.c.gc.QueryWriteStatus(w.ctx, &storagepb.QueryWriteStatusRequest{UploadId: w.upid})

	// q.GetCommittedSize() will return 0 if q is nil.
	return q.GetCommittedSize(), err
}

Expected behavior

Code compiles

Actual behavior

go: module github.com/golang/protobuf is deprecated: Use the "google.golang.org/protobuf" module instead.
# cloud.google.com/go/storage
/home/.../go/pkg/mod/cloud.google.com/go/storage@v1.18.1/storage.go:1447:53: o.GetCustomerEncryption().GetKeySha256 undefined (type *"google.golang.org/genproto/googleapis/storage/v2".Object_CustomerEncryption has no field or method GetKeySha256)
/home/.../go/pkg/mod/cloud.google.com/go/storage@v1.18.1/writer.go:439:10: q.GetCommittedSize undefined (type *"google.golang.org/genproto/googleapis/storage/v2".QueryWriteStatusResponse has no field or method GetCommittedSize)

Additional context
https://pkg.go.dev/google.golang.org/genproto@v0.0.0-20211015135405-485ec31e706e/googleapis/storage/v2?tab=versions

Methods were removed "v0.0.0-20211015135405-485ec31e706e Oct 15, 2021"

They are available in "v0.0.0-20211013025323-ce878158c4d4 Oct 13, 2021"

[codyoss]

Related to googleapis/go-genproto#700.

[codyoss]

Hey, this was an intentional breaking change to a currently internal api. With the versions specified in the released go.mod I believe everything should work. The breaking change is fixed in #4987 with correct versions of genproto and the code being refactored. This code has not been released yet though. Are you using Go module to control versioning? If so could you share it?

[johanbrandhorst]

Don't we need a release of this repo to fix this? v1.18.1 is using the broken APIs: https://github.com/googleapis/google-cloud-go/releases/tag/storage%2Fv1.18.1, https://github.com/googleapis/google-cloud-go/blob/storage/v1.18.1/storage/storage.go#L1447

EDIT: I think it might require you to have any other dependency on go-genproto to include the bad versions. That's going to be pretty common if you use anything like gRPC or protobuf in your depedencies.

[TomislavHorvat1]

I've managed to work around it by downgrading go-genproto to google.golang.org/genproto@v0.0.0-20211013025323-ce878158c4d4, that one seems to work

[codyoss]

https://github.com/googleapis/google-cloud-go/releases/tag/storage/v1.18.2 has been released which should help manage dependencies and force a newer version of genproto to be pulled in. Thanks for reporting the issue and all of the input!

[johanbrandhorst]

I can confirm this fixed my issue, thanks for the quick response!

[chrisntb]

Yep fixed our issue. This was our go.mod file yesterday:

module github.com/foo/bar

go 1.17

require (
	cloud.google.com/go v0.97.0
	cloud.google.com/go/cloudtasks v1.0.0
	cloud.google.com/go/firestore v1.6.0
	cloud.google.com/go/logging v1.4.2
	cloud.google.com/go/profiler v0.1.0
	cloud.google.com/go/pubsub v1.17.0
	cloud.google.com/go/spanner v1.26.0
	cloud.google.com/go/storage v1.18.1
	contrib.go.opencensus.io/exporter/stackdriver v0.13.8
	firebase.google.com/go v3.13.0+incompatible
	github.com/aws/aws-sdk-go v1.41.3
	github.com/dgrijalva/jwt-go v3.2.0+incompatible
	github.com/dustin/go-humanize v1.0.0
	github.com/go-chi/chi/v5 v5.0.4
	github.com/go-chi/cors v1.2.0
	github.com/golang/mock v1.6.0
	github.com/golang/protobuf v1.5.2
	github.com/google/uuid v1.3.0
	github.com/googleapis/gax-go/v2 v2.1.1
	github.com/olekukonko/tablewriter v0.0.5
	github.com/pkg/errors v0.9.1
	github.com/sethgrid/pester v1.1.0
	github.com/ssor/bom v0.0.0-20170718123548-6386211fdfcf
	github.com/tidwall/gjson v1.9.4
	github.com/tidwall/sjson v1.2.2
	github.com/ttacon/libphonenumber v1.2.1
	github.com/xeipuuv/gojsonschema v1.2.0
	go.opencensus.io v0.23.0
	golang.org/x/net v0.0.0-20211014172544-2b766c08f1c0
	golang.org/x/oauth2 v0.0.0-20211005180243-6b3c2da341f1
	google.golang.org/api v0.58.0
	google.golang.org/genproto v0.0.0-20211013025323-ce878158c4d4
	google.golang.org/grpc v1.41.0
	google.golang.org/protobuf v1.27.1
	googlemaps.github.io/maps v1.3.2
)