Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bazel query doesn't return the .bzl files that load external repositories #14280

Closed
linzhp opened this issue Nov 15, 2021 · 5 comments
Closed
Assignees
Labels
P2 We'll consider working on this in future. (Assignee optional) team-ExternalDeps External dependency handling, remote repositiories, WORKSPACE file. team-OSS Issues for the Bazel OSS team: installation, release processBazel packaging, website type: feature request

Comments

@linzhp
Copy link
Contributor

linzhp commented Nov 15, 2021

Description of the problem / feature request:

bazel query doesn't return the .bzl files that load an external repository

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

  1. Create such Bazel workspace:
-- BUILD.bazel --
-- WORKSPACE --
load("//:http_archives.bzl", "http_archives")

http_archives()

load("@io_bazel_rules_go//go:deps.bzl", "go_register_toolchains", "go_rules_dependencies")

go_rules_dependencies()

go_register_toolchains(version = "1.17.1")


load("@com_google_protobuf//:protobuf_deps.bzl", "protobuf_deps")

protobuf_deps()
-- http_archives.bzl --
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

def http_archives():
    http_archive(
        name = "io_bazel_rules_go",
        sha256 = "2b1641428dff9018f9e85c0384f03ec6c10660d935b750e3fa1492a281a53b0f",
        urls = [
            "https://mirror.bazel.build/github.com/bazelbuild/rules_go/releases/download/v0.29.0/rules_go-v0.29.0.zip",
            "https://github.com/bazelbuild/rules_go/releases/download/v0.29.0/rules_go-v0.29.0.zip",
        ],
    )
    http_archive(
        name = "com_google_protobuf",
        sha256 = "d0f5f605d0d656007ce6c8b5a82df3037e1d8fe8b121ed42e536f569dec16113",
        strip_prefix = "protobuf-3.14.0",
        urls = [
            "https://mirror.bazel.build/github.com/protocolbuffers/protobuf/archive/v3.14.0.tar.gz",
            "https://github.com/protocolbuffers/protobuf/archive/v3.14.0.tar.gz",
        ],
    )
  1. Run bazel query 'buildfiles(//external:io_bazel_rules_go)' in it.

The query only returns //external:WORKSPACE, while @io_bazel_rules_go is actually defined in http_archives.bzl.

What operating system are you running Bazel on?

macOS 12.0.1 and Linux 5.4.129

What's the output of bazel info release?

release 4.2.1

Any other information, logs, or outputs that you want to share?

  • bazel query 'loadfiles(//external:io_bazel_rules_go)' returns "Empty results".
  • bazel query --output=xml '//external:io_bazel_rules_go' returns
<?xml version="1.1" encoding="UTF-8" standalone="no"?>
<query version="2">
    <rule class="http_archive" location="/private/tmp/loadfiles/WORKSPACE:3:14" name="//external:io_bazel_rules_go">
        <string name="name" value="io_bazel_rules_go"/>
        <string name="generator_name" value="io_bazel_rules_go"/>
        <string name="generator_function" value="http_archives"/>
        <string name="generator_location"/>
        <list name="urls">
            <string value="https://mirror.bazel.build/github.com/bazelbuild/rules_go/releases/download/v0.29.0/rules_go-v0.29.0.zip"/>
            <string value="https://github.com/bazelbuild/rules_go/releases/download/v0.29.0/rules_go-v0.29.0.zip"/>
        </list>
        <string name="sha256" value="2b1641428dff9018f9e85c0384f03ec6c10660d935b750e3fa1492a281a53b0f"/>
    </rule>
</query>

The generator_location is empty.

@libratiger
Copy link
Contributor

@linzhp If possible you can try on other OS, such as on Linux.

@linzhp
Copy link
Contributor Author

linzhp commented Nov 16, 2021

Tried on Linux, and got the same result. Updated the description.

@gregestren gregestren added team-OSS Issues for the Bazel OSS team: installation, release processBazel packaging, website type: feature request untriaged labels Nov 18, 2021
@libratiger
Copy link
Contributor

The io_bazel_rules_go only used in the workplace, so this is not a problem。

@linzhp
Copy link
Contributor Author

linzhp commented Nov 20, 2021

@gregestren I think this is a bug, not a feature request.
@libratiger This is a problem for us. Because of this, we can't find all .bzl files Bazel needs to start a build in a git sparse checkout.

@gregestren
Copy link
Contributor

@linzhp noted. I marked it as such because I'm not sure if this is query processing something incorrectly vs. just not being updated for external repo support for cases like this.

I'm happy to let you and the domain experts (labeled on the issue) clarify as makes sense between you.

@Wyverald Wyverald added P2 We'll consider working on this in future. (Assignee optional) team-ExternalDeps External dependency handling, remote repositiories, WORKSPACE file. and removed untriaged labels Jan 24, 2022
brentleyjones pushed a commit to brentleyjones/bazel that referenced this issue Mar 7, 2022
This is an alternative approach to fix bazelbuild#14280. It adds transitive closure of Starlark dependencies to `//external` package when loading `WORKSPACE` file, so it can be processed in the same way as `BUILD` files during query execution. Comparing to the approach taken in bazelbuild#14497, this approach is less intrusive, but not able to distinguish the extension files needed by `//external:foo` and `//external:bar`, meaning `buildfiles(//external:foo)` returns the same result as `buildfiles(//external:*)`. However, this behavior is consistent with other packages. For example, `buildfiles(//foo:bar)` has the same result as `buildfiles(//foo:*)`.

Closes bazelbuild#14630.

PiperOrigin-RevId: 424092916
(cherry picked from commit a6c3f23)
brentleyjones pushed a commit to brentleyjones/bazel that referenced this issue Mar 7, 2022
This is an alternative approach to fix bazelbuild#14280. It adds transitive closure of Starlark dependencies to `//external` package when loading `WORKSPACE` file, so it can be processed in the same way as `BUILD` files during query execution. Comparing to the approach taken in bazelbuild#14497, this approach is less intrusive, but not able to distinguish the extension files needed by `//external:foo` and `//external:bar`, meaning `buildfiles(//external:foo)` returns the same result as `buildfiles(//external:*)`. However, this behavior is consistent with other packages. For example, `buildfiles(//foo:bar)` has the same result as `buildfiles(//foo:*)`.

Closes bazelbuild#14630.

PiperOrigin-RevId: 424092916
(cherry picked from commit a6c3f23)
Wyverald pushed a commit that referenced this issue Mar 8, 2022
* Adding Starlark dependencies to the package //external

This is an alternative approach to fix #14280. It adds transitive closure of Starlark dependencies to `//external` package when loading `WORKSPACE` file, so it can be processed in the same way as `BUILD` files during query execution. Comparing to the approach taken in #14497, this approach is less intrusive, but not able to distinguish the extension files needed by `//external:foo` and `//external:bar`, meaning `buildfiles(//external:foo)` returns the same result as `buildfiles(//external:*)`. However, this behavior is consistent with other packages. For example, `buildfiles(//foo:bar)` has the same result as `buildfiles(//foo:*)`.

Closes #14630.

PiperOrigin-RevId: 424092916
(cherry picked from commit a6c3f23)

* Avoid bazel_module_test timeout on macos

https://buildkite.com/bazel/bazel-bazel/builds/17785#9a1fb564-c6f9-4e69-ac8f-87c422a002b0
By setting the test size to "large".

RELNOTES:None
PiperOrigin-RevId: 409114345
(cherry picked from commit 1d99391)

Co-authored-by: Zhongpeng Lin <zplin@uber.com>
Co-authored-by: pcloudy <pcloudy@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P2 We'll consider working on this in future. (Assignee optional) team-ExternalDeps External dependency handling, remote repositiories, WORKSPACE file. team-OSS Issues for the Bazel OSS team: installation, release processBazel packaging, website type: feature request
Projects
None yet
4 participants