Targets placed under source code subdirectory named "external" fail to compile with "file not found" errors

[oakad]

Description of the bug:

Consider a source code tree which has one of its directories named "external":

"source root"
  | - foo
  |   | - dir with targets
  |   | - dir with more targets
  | - external
  |   | - dir with broken targets
  |   |   | - dir with more broken targets

It appears, that bazel (at least the version I'm using) can not correctly compile targets nested under top level "external" subdirectory.

I'm aware that bazel uses a subdirectory called "external" to store dependencies in the build tree, but this should not infringe on the ability of users to have a directory, so called, in their source tree.

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Rename one of the immediate subdirs of any source tree to "external", try to build targets under that directory. I, obviously, had not tried all possible rules, but those I tried are broken (spurious "file not found" errors).

Which operating system are you running Bazel on?

Linux

What is the output of bazel info release?

release 5.2.0

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Have you found anything relevant by searching the web?

Tough luck searching for issues with "external" key word in the query. :-)

Any other information, logs, or outputs that you want to share?

No response

[sgowroji]

Hello @oakad, Could you please provide the above observation with a sample example(Repo) to reproduce with complete steps at our end. Thanks!

[oakad]

https://github.com/oakad/bazel_issue_16220

Here.

This works:

bazel run externa:external

This does not:

bazel run external:external

The subdirs are a copy of each other, up to 1 letter in the name.

[Wyverald]

This is a long-standing known bug in Bazel which IIUC unfortunately does not have a short-term solution. The migration of AOSP onto Bazel encountered similar problems. @lberki could you comment on how AOSP dealt with this, and whether the solution is applicable to other Bazel users?

[lberki]

The good news is that we do have a solution: if you add the --experimental_sibling_repository_layout command line flag, external repositories will be under ../$REPO instead of external/$REPO and things under the external/ directory will be usable.

We are planning to flip the default value of that flag to true (see #12821) but it's a very incompatible change and we haven't yet had the determination to do so.

@Wyverald do you think we should make an effort to do the flip after Bazel 6.0? (definitely not before it's cut; there is too little time left for that even if we really wanted to do so)

[Wyverald]

We could flip after 6.0 of course; just need an owner to watch out for downstream breakages and coordinate fixes.

Just for my curiosity, wouldn't there still be some sort of conflict regarding the //external package? Like, you probably can't refer to anything right under external, since it would be "rerouted" to the "fake" external package containing repo rules instead.

[lberki]

Let's not flip it in 6.0; unresolved symlinks and bzlmod are large enough chunks of work and I personally can't take care of downstream breakages and I don't know of anyone who would be able to.

From memory, the package //external is still problematic due to the "fake" package, but packages under it (e.g. //external/foo work. The example in the description of the bug mentioned directories under external/ and that's the common use case (a subdirectory for each dependency) so I assumed that was good enough.

Then, the next step after that would be that when bzlmod lands and we can switch off WORKSPACE for good, we can also remove //external. Technically, we can do it earlier, since bind() is a nice thing, but it happens to be used by android_{sdk,ndk}_repository so we'd need to find a solution for that (it is in progress for unrelated reasons but I don't think this particular bug should increase the urgency of providing a replacement for these two Android repositories)

[fmeum]

The first thing that breaks with --experimental_sibling_repository_layout is Stardoc, so I expect the downstream pipeline to be deeply red. If @tetromino's ongoing work ends up resolving that, that could be a good time to run a meaningful first pipeline.