Cannot use 'rules_go' + 'com_google_protobuf' to compile gogo/status

[psigen]

What version of rules_go are you using?

v0.25.1

What version of gazelle are you using?

v0.22.3

What version of Bazel are you using?

v4.0.0

Does this issue reproduce with the latest releases of all the above?

I think these are all the latest possible.

What operating system and processor architecture are you using?

Ubuntu 18.04, Linux, x86_64

Any other potentially useful information about your toolchain?

$ go version
go version go1.15.6 linux/amd64

What did you do?

I am trying to compile a minimal go binary that uses gogo/status as the only dependency. However, every combination of build_file_proto_mode = "disable|disable_global" and com_google_protobuf version seems to have compilation errors related to proto version mismatches.

A minimal example can be found in this repo:
https://github.com/psigen/rules-go-2805

Related information I have been using for reference:

• https://github.com/bazelbuild/rules_go/blob/master/go/dependencies.rst#overriding-dependencies
• https://github.com/bazelbuild/rules_go/blob/master/proto/core.rst#avoiding-conflicts
• https://github.com/bazelbuild/bazel-gazelle/blob/master/README.rst#directives
• overwritting gogo/protobuf dependency makes protobuf compilation fail #1405 (comment)

What did you expect to see?

I expect bazel build //... to complete without error.

What did you see instead?

$ bazel build //...
INFO: Invocation ID: b2095db6-98ae-41f3-b505-8597aa444779
INFO: Analyzed 2 targets (0 packages loaded, 0 targets configured).
INFO: Found 2 targets...
ERROR: /home/pras/.cache/bazel/_bazel_pras/7ff1d612b81fb750bcd6f1a5fd699714/external/com_github_gogo_status/BUILD.bazel:3:11: GoCompilePkg external/com_github_gogo_status/status.a failed: (Exit 1): builder failed: error executing command bazel-out/host/bin/external/go_sdk/builder compilepkg -sdk external/go_sdk -installsuffix linux_amd64 -src external/com_github_gogo_status/status.go -arc ... (remaining 27 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox builder failed: error executing command bazel-out/host/bin/external/go_sdk/builder compilepkg -sdk external/go_sdk -installsuffix linux_amd64 -src external/com_github_gogo_status/status.go -arc ... (remaining 27 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
external/com_github_gogo_status/status.go:158:22: cannot use &types.Any literal (type *types.Any) as type *any.Any in append
external/com_github_gogo_status/status.go:186:21: cannot use any (type *types.Any) as type *any.Any in append
external/com_github_gogo_status/status.go:200:31: cannot use any (type *any.Any) as type *types.Any in argument to types.UnmarshalAny
compilepkg: error running subcommand external/go_sdk/pkg/tool/linux_amd64/compile: exit status 2
INFO: Elapsed time: 0.394s, Critical Path: 0.27s
INFO: 2 processes: 2 internal.
FAILED: Build did NOT complete successfully

[jayconrod]

I was able to reproduce this, but I'm not sure exactly what's going wrong. It seems like a version mismatch. Things to try:

• Update to newer versions. Make sure they're consistent with go mod tidy when importing.
• Add build_file_proto_mode = "disable_global" on any go_repository that defines or imports proto packages.

[psigen]

I updated all the dependencies, the only change is that *any.Any becomes *anypb.Any in the error message.

Here is a branch of my test repository with those dependencies updated via:

go get -u all
go mod tidy
bazel run //:gazelle -- update-repos -from_file=./go.mod -prune=true -to_macro=repositories.bzl%go_repositories

https://github.com/psigen/rules-go-2805/tree/latest

[psigen]

Adding build_file_proto_mode = "disable_global" to

go_repository(
    name = "com_github_gogo_status",
    importpath = "github.com/gogo/status",
    sum = "h1:+eIkrewn5q6b30y+g/BJINVVdi2xH7je5MPJ3ZPK3JA=",
    version = "v1.1.0",
    build_file_proto_mode = "disable_global",
)

replaces the above error with:

external/com_github_gogo_status/status.go:40:2: internal compiler error: conflicting package heights 23 and 24 for path "google.golang.org/genproto/googleapis/rpc/status"

which appears to be the issue reported in #2758 (comment)

[psigen]

😢 I think I'm pretty stuck here. I'm not even using protobuf, but my codebase depends on a third-party go library that uses it internally.

Are there any other workarounds to try @jayconrod ?

[jayconrod]

I tried this sequence, and bazel build @com_github_gogo_status//:status worked for me:

git checkout 722700cbcf94abbc456c0d09dc42f219aee48a18
go get -d -u all
bazel run :gazelle -- update-repos -from_file=go.mod -build_file_proto_mode=disable_global -to_macro=repositories.bzl%go_repositories

That produced this repositories.bzl file. Not sure what the difference is?

"""
External repositories used in this workspace.
"""

load("@bazel_gazelle//:deps.bzl", "go_repository")

def go_repositories():
    """
    Autogenerated dependencies from Gazelle
    """
    go_repository(
        name = "co_honnef_go_tools",
        build_file_proto_mode = "disable_global",
        importpath = "honnef.co/go/tools",
        sum = "h1:/hemPrYIhOhy8zYrNj+069zDB68us2sMGsfkFJO0iZs=",
        version = "v0.0.0-20190523083050-ea95bdfd59fc",
    )
    go_repository(
        name = "com_github_burntsushi_toml",
        build_file_proto_mode = "disable_global",
        importpath = "github.com/BurntSushi/toml",
        sum = "h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=",
        version = "v0.3.1",
    )
    go_repository(
        name = "com_github_census_instrumentation_opencensus_proto",
        build_file_proto_mode = "disable_global",
        importpath = "github.com/census-instrumentation/opencensus-proto",
        sum = "h1:glEXhBS5PSLLv4IXzLA5yPRVX4bilULVyxxbrfOtDAk=",
        version = "v0.2.1",
    )
    go_repository(
        name = "com_github_client9_misspell",
        build_file_proto_mode = "disable_global",
        importpath = "github.com/client9/misspell",
        sum = "h1:ta993UF76GwbvJcIo3Y68y/M3WxlpEHPWIGDkJYwzJI=",
        version = "v0.3.4",
    )
    go_repository(
        name = "com_github_cncf_udpa_go",
        build_file_proto_mode = "disable_global",
        importpath = "github.com/cncf/udpa/go",
        sum = "h1:cqQfy1jclcSy/FwLjemeg3SR1yaINm74aQyupQ0Bl8M=",
        version = "v0.0.0-20201120205902-5459f2c99403",
    )
    go_repository(
        name = "com_github_davecgh_go_spew",
        build_file_proto_mode = "disable_global",
        importpath = "github.com/davecgh/go-spew",
        sum = "h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8=",
        version = "v1.1.0",
    )
    go_repository(
        name = "com_github_envoyproxy_go_control_plane",
        build_file_proto_mode = "disable_global",
        importpath = "github.com/envoyproxy/go-control-plane",
        sum = "h1:EmNYJhPYy0pOFjCx2PrgtaBXmee0iUX9hLlxE1xHOJE=",
        version = "v0.9.9-0.20201210154907-fd9021fe5dad",
    )
    go_repository(
        name = "com_github_envoyproxy_protoc_gen_validate",
        build_file_proto_mode = "disable_global",
        importpath = "github.com/envoyproxy/protoc-gen-validate",
        sum = "h1:EQciDnbrYxy13PgWoY8AqoxGiPrpgBZ1R8UNe3ddc+A=",
        version = "v0.1.0",
    )

    go_repository(
        name = "com_github_gogo_googleapis",
        build_file_proto_mode = "disable_global",
        importpath = "github.com/gogo/googleapis",
        sum = "h1:zgVt4UpGxcqVOw97aRGxT4svlcmdK35fynLNctY32zI=",
        version = "v1.4.0",
    )
    go_repository(
        name = "com_github_gogo_protobuf",
        build_file_proto_mode = "disable",
        importpath = "github.com/gogo/protobuf",
        sum = "h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=",
        version = "v1.3.2",
    )
    go_repository(
        name = "com_github_gogo_status",
        build_file_proto_mode = "disable_global",
        importpath = "github.com/gogo/status",
        sum = "h1:+eIkrewn5q6b30y+g/BJINVVdi2xH7je5MPJ3ZPK3JA=",
        version = "v1.1.0",
    )
    go_repository(
        name = "com_github_golang_glog",
        build_file_proto_mode = "disable_global",
        importpath = "github.com/golang/glog",
        sum = "h1:VKtxabqXZkF25pY9ekfRL6a582T4P37/31XEstQ5p58=",
        version = "v0.0.0-20160126235308-23def4e6c14b",
    )
    go_repository(
        name = "com_github_golang_mock",
        build_file_proto_mode = "disable_global",
        importpath = "github.com/golang/mock",
        sum = "h1:G5FRp8JnTd7RQH5kemVNlMeyXQAztQ3mOWV95KxsXH8=",
        version = "v1.1.1",
    )

    go_repository(
        name = "com_github_golang_protobuf",
        build_file_proto_mode = "disable_global",
        importpath = "github.com/golang/protobuf",
        sum = "h1:JjCZWpVbqXDqFVmTfYWEVTMIYrL/NPdPSCHPJ0T/raM=",
        version = "v1.4.3",
    )
    go_repository(
        name = "com_github_google_go_cmp",
        build_file_proto_mode = "disable_global",
        importpath = "github.com/google/go-cmp",
        sum = "h1:L8R9j+yAqZuZjsqh/z+F1NCffTKKLShY6zXTItVIZ8M=",
        version = "v0.5.4",
    )
    go_repository(
        name = "com_github_google_uuid",
        build_file_proto_mode = "disable_global",
        importpath = "github.com/google/uuid",
        sum = "h1:EVhdT+1Kseyi1/pUmXKaFxYsDNy9RQYkMWRH68J/W7Y=",
        version = "v1.1.2",
    )
    go_repository(
        name = "com_github_kisielk_errcheck",
        build_file_proto_mode = "disable_global",
        importpath = "github.com/kisielk/errcheck",
        sum = "h1:e8esj/e4R+SAOwFwN+n3zr0nYeCyeweozKfO23MvHzY=",
        version = "v1.5.0",
    )
    go_repository(
        name = "com_github_kisielk_gotool",
        build_file_proto_mode = "disable_global",
        importpath = "github.com/kisielk/gotool",
        sum = "h1:AV2c/EiW3KqPNT9ZKl07ehoAGi4C5/01Cfbblndcapg=",
        version = "v1.0.0",
    )
    go_repository(
        name = "com_github_pmezard_go_difflib",
        build_file_proto_mode = "disable_global",
        importpath = "github.com/pmezard/go-difflib",
        sum = "h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=",
        version = "v1.0.0",
    )
    go_repository(
        name = "com_github_prometheus_client_model",
        build_file_proto_mode = "disable_global",
        importpath = "github.com/prometheus/client_model",
        sum = "h1:gQz4mCbXsO+nc9n1hCxHcGA3Zx3Eo+UHZoInFGUIXNM=",
        version = "v0.0.0-20190812154241-14fe0d1b01d4",
    )
    go_repository(
        name = "com_github_stretchr_objx",
        build_file_proto_mode = "disable_global",
        importpath = "github.com/stretchr/objx",
        sum = "h1:4G4v2dO3VZwixGIRoQ5Lfboy6nUhCyYzaqnIAPPhYs4=",
        version = "v0.1.0",
    )
    go_repository(
        name = "com_github_stretchr_testify",
        build_file_proto_mode = "disable_global",
        importpath = "github.com/stretchr/testify",
        sum = "h1:nOGnQDM7FYENwehXlg/kFVnos3rEvtKTjRvOWSzb6H4=",
        version = "v1.5.1",
    )
    go_repository(
        name = "com_github_yuin_goldmark",
        build_file_proto_mode = "disable_global",
        importpath = "github.com/yuin/goldmark",
        sum = "h1:ruQGxdhGHe7FWOJPT0mKs5+pD2Xs1Bm/kdGlHO04FmM=",
        version = "v1.2.1",
    )
    go_repository(
        name = "com_google_cloud_go",
        build_file_proto_mode = "disable_global",
        importpath = "cloud.google.com/go",
        sum = "h1:e0WKqKTd5BnrG8aKH3J3h+QvEIQtSUcf2n5UZ5ZgLtQ=",
        version = "v0.26.0",
    )
    go_repository(
        name = "in_gopkg_check_v1",
        build_file_proto_mode = "disable_global",
        importpath = "gopkg.in/check.v1",
        sum = "h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=",
        version = "v0.0.0-20161208181325-20d25e280405",
    )
    go_repository(
        name = "in_gopkg_yaml_v2",
        build_file_proto_mode = "disable_global",
        importpath = "gopkg.in/yaml.v2",
        sum = "h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw=",
        version = "v2.2.2",
    )
    go_repository(
        name = "org_golang_google_appengine",
        build_file_proto_mode = "disable_global",
        importpath = "google.golang.org/appengine",
        sum = "h1:/wp5JvzpHIxhs/dumFmF7BXTf3Z+dd4uXta4kVyO508=",
        version = "v1.4.0",
    )

    go_repository(
        name = "org_golang_google_genproto",
        build_file_proto_mode = "disable_global",
        importpath = "google.golang.org/genproto",
        sum = "h1:N98SvVh7Hdle2lgUVFuIkf0B3u29CUakMUQa7Hwz8Wc=",
        version = "v0.0.0-20210207032614-bba0dbe2a9ea",
    )
    go_repository(
        name = "org_golang_google_grpc",
        build_file_proto_mode = "disable_global",
        importpath = "google.golang.org/grpc",
        sum = "h1:TwIQcH3es+MojMVojxxfQ3l3OF2KzlRxML2xZq0kRo8=",
        version = "v1.35.0",
    )
    go_repository(
        name = "org_golang_google_protobuf",
        build_file_proto_mode = "disable_global",
        importpath = "google.golang.org/protobuf",
        sum = "h1:Ejskq+SyPohKW+1uil0JJMtmHCgJPJ/qWTxr8qp+R4c=",
        version = "v1.25.0",
    )
    go_repository(
        name = "org_golang_x_crypto",
        build_file_proto_mode = "disable_global",
        importpath = "golang.org/x/crypto",
        sum = "h1:psW17arqaxU48Z5kZ0CQnkZWQJsqcURM6tKiBApRjXI=",
        version = "v0.0.0-20200622213623-75b288015ac9",
    )
    go_repository(
        name = "org_golang_x_exp",
        build_file_proto_mode = "disable_global",
        importpath = "golang.org/x/exp",
        sum = "h1:c2HOrn5iMezYjSlGPncknSEr/8x5LELb/ilJbXi9DEA=",
        version = "v0.0.0-20190121172915-509febef88a4",
    )
    go_repository(
        name = "org_golang_x_lint",
        build_file_proto_mode = "disable_global",
        importpath = "golang.org/x/lint",
        sum = "h1:XQyxROzUlZH+WIQwySDgnISgOivlhjIEwaQaJEJrrN0=",
        version = "v0.0.0-20190313153728-d0100b6bd8b3",
    )
    go_repository(
        name = "org_golang_x_mod",
        build_file_proto_mode = "disable_global",
        importpath = "golang.org/x/mod",
        sum = "h1:RM4zey1++hCTbCVQfnWeKs9/IEsaBLA8vTkd0WVtmH4=",
        version = "v0.3.0",
    )
    go_repository(
        name = "org_golang_x_net",
        build_file_proto_mode = "disable_global",
        importpath = "golang.org/x/net",
        sum = "h1:003p0dJM77cxMSyCPFphvZf/Y5/NXf5fzg6ufd1/Oew=",
        version = "v0.0.0-20210119194325-5f4716e94777",
    )
    go_repository(
        name = "org_golang_x_oauth2",
        build_file_proto_mode = "disable_global",
        importpath = "golang.org/x/oauth2",
        sum = "h1:vEDujvNQGv4jgYKudGeI/+DAX4Jffq6hpD55MmoEvKs=",
        version = "v0.0.0-20180821212333-d2e6202438be",
    )

    go_repository(
        name = "org_golang_x_sync",
        build_file_proto_mode = "disable_global",
        importpath = "golang.org/x/sync",
        sum = "h1:DcqTD9SDLc+1P/r1EmRBwnVsrOwW+kk2vWf9n+1sGhs=",
        version = "v0.0.0-20201207232520-09787c993a3a",
    )
    go_repository(
        name = "org_golang_x_sys",
        build_file_proto_mode = "disable_global",
        importpath = "golang.org/x/sys",
        sum = "h1:VwygUrnw9jn88c4u8GD3rZQbqrP/tgas88tPUbBxQrk=",
        version = "v0.0.0-20210124154548-22da62e12c0c",
    )
    go_repository(
        name = "org_golang_x_term",
        build_file_proto_mode = "disable_global",
        importpath = "golang.org/x/term",
        sum = "h1:v+OssWQX+hTHEmOBgwxdZxK4zHq3yOs8F9J7mk0PY8E=",
        version = "v0.0.0-20201126162022-7de9c90e9dd1",
    )
    go_repository(
        name = "org_golang_x_text",
        build_file_proto_mode = "disable_global",
        importpath = "golang.org/x/text",
        sum = "h1:i6eZZ+zk0SOf0xgBpEpPD18qWcJda6q1sxt3S0kzyUQ=",
        version = "v0.3.5",
    )
    go_repository(
        name = "org_golang_x_tools",
        build_file_proto_mode = "disable_global",
        importpath = "golang.org/x/tools",
        sum = "h1:CB3a9Nez8M13wwlr/E2YtwoU+qYHKfC+JrDa45RXXoQ=",
        version = "v0.0.0-20210106214847-113979e3529a",
    )
    go_repository(
        name = "org_golang_x_xerrors",
        build_file_proto_mode = "disable_global",
        importpath = "golang.org/x/xerrors",
        sum = "h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=",
        version = "v0.0.0-20200804184101-5ec99f83aff1",
    )

[jayconrod]

The error conflicting package heights 23 and 24 for path "google.golang.org/genproto/googleapis/rpc/status" is likely caused by some targets depending on @go_googleapis and other depending on @org_golang_google_genproto. The two repositories conceptually contain the same packages, but they're generated differently, and they may be at different versions.

Still, if you can post a commit and a bazel build command where that error reproduces, it would be helpful for improving that diagnostic. The conflict should be reported earlier.

[psigen]

Alright, I am able to replicate your steps and confirm that:
bazel build @com_github_gogo_status//:status
is successful.

Then I was running into the following issue:

bazel build //:bazel-go-issue
[...]
compilepkg: missing strict dependencies:
	/home/pras/.cache/bazel/_bazel_pras/7ff1d612b81fb750bcd6f1a5fd699714/sandbox/linux-sandbox/179/execroot/com_github_psigen_rules_go_2805/main.go: import of "google.golang.org/grpc/codes"
No dependencies were provided.

This turned out to be because I used @org_golang_google_grpc//:grpc, in my deps, but I really needed to use @org_golang_google_grpc//codes, which I found by manually inspecting the generated BUILD.bazel files for the right naming.

At this point I was able to run 🎉, so I tried to figure out where I went wrong before. I think the issue was that I was confused by the above error message and removed the disable_global unnecessarily from the go_repository(name="org_golang_google_grpc", [...]) declaration, which flipped me to the legitimate package mismatch error. I didn't realize that this error message was actually further from working than the other one.

• I committed a working branch based on your instructions to 85953bd9094c7f4f19ec77bd9282aed5a41f6bc3.
• I removed disable_global from "org_golang_google_grpc" to get the package mismatch errror in 91eb8b48bd6f83d30e7e20855795bae520ad0079.

I will try to backport this fix into my actual repository with much more going on and see if it still holds. Hopefully this post will at least show up on google for other people trying to figure out what to do when they see these error messages.

[psigen]

Ok, this worked for me! To summarize the necessary steps:

• If you see the original error message cannot use &types.Any literal (type *types.Any) as type *any.Any in append, update to latest version of all go libraries using

  go get -d -u all
  bazel run :gazelle -- update-repos -from_file=go.mod -build_file_proto_mode=disable_global -to_macro=repositories.bzl%go_repositories
  

• Remember to use -build_file_proto_mode=disable_global in the gazelle command above on all libraries ☝️, or else you will get internal compiler error: conflicting package heights
• If you see compilepkg: missing strict dependencies: import of "[...]" No dependencies were provided., this means you need to add a corresponding entry to deps=[...] in the go_library/go_binary target that is having the problem. You can grep the BUILD.bazel files in your bazel-*/external/... paths to find the target you are missing. But this is not a library versioning problem, it is just a bazel declaration issue.

I sure did not know all of those things when I first made this ticket 😅 Thanks for all the help @jayconrod !

[nchepanov]

If someone finds this in the future... I spent 2 days figuring this out too, as I was attempting to migrate a large repository to Bazel.

The key to this was -build_file_proto_mode=disable_global that was missing in @psigen original message. I'm not sure why # gazelle:proto disable_global directive didn't work.