repository_ctx.symlink error when testing remotely using RBE #9979

davido · 2019-10-09T22:18:41Z

When running tests on Gerrit@HEAD using RBE (see this documentation section), I am getting this error, when running one specific test:

  $ bazel test --config=remote \
   --remote_instance_name=projects/${PROJECT}/instances/default_instance \
   //javatests/com/google/gerrit/pgm:pgm_license_test
ERROR: An error occurred during the fetch of repository 'local_config_cc':
   Traceback (most recent call last):
	File "/home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/bazel_tools/tools/cpp/cc_configure.bzl", line 104
		repository_ctx.symlink(<2 more arguments>)
	File "/home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/bazel_tools/tools/cpp/cc_configure.bzl", line 104, in repository_ctx.symlink
		paths(<1 more arguments>)
'dict' object is not callable
ERROR: /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/bazel_tools/tools/cpp/cc_configure.bzl:180:5: no such package '@local_config_cc//': Traceback (most recent call last):
	File "/home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/bazel_tools/tools/cpp/cc_configure.bzl", line 104
		repository_ctx.symlink(<2 more arguments>)
	File "/home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/bazel_tools/tools/cpp/cc_configure.bzl", line 104, in repository_ctx.symlink
		paths(<1 more arguments>)
'dict' object is not callable and referenced by '//external:cc_toolchain'
WARNING: /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/remote_java_tools_linux/BUILD:702:1: in hdrs attribute of cc_library rule @remote_java_tools_linux//:combiners: Artifact 'external/remote_java_tools_linux/java_tools/src/tools/singlejar/zip_headers.h' is duplicated (through '@remote_java_tools_linux//:transient_bytes' and '@remote_java_tools_linux//:zip_headers'). Since this rule was created by the macro 'cc_library', the error might have been caused by the macro implementation
ERROR: Analysis of target '//javatests/com/google/gerrit/pgm:pgm_license_test' failed; build aborted: no such package '@local_config_cc//': Traceback (most recent call last):
	File "/home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/bazel_tools/tools/cpp/cc_configure.bzl", line 104
		repository_ctx.symlink(<2 more arguments>)
	File "/home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/bazel_tools/tools/cpp/cc_configure.bzl", line 104, in repository_ctx.symlink
		paths(<1 more arguments>)
'dict' object is not callable
INFO: Elapsed time: 1.178s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded, 2 targets configured)
FAILED: Build did NOT complete successfully (0 packages loaded, 2 targets configured)

Note, that other tests work as expected, e.g.:

  $ bazel test --config=remote \
  --remote_instance_name=projects/${PROJECT}/default_instance \
  //javatests/com/google/gerrit/acceptance/...

//CC @hanwen @buchgr

The text was updated successfully, but these errors were encountered:

buchgr · 2019-10-10T14:43:49Z

Thanks for reporting! You are right, we have not implemented support for repository_ctx.symlink yet.

davido · 2019-10-10T14:49:45Z

@buchgr Thank for the quick answer and confirming the problem.

How can I mark the test to not run on remote config? My current problem that I would like to run the whole nine yard tests using something like: bazel build --config=remote //... that doesn't work, as this problematic test is in the middle of our package tree: //javatests/com/google/gerrit/pgm:pgm_license_test.

buchgr · 2019-10-10T14:55:22Z

java_test(
  tags = ["no-remote"],
)

davido · 2019-10-10T14:58:54Z

Thx!

davido · 2019-10-10T15:36:06Z

Unfortunately it doesn't work. It's sh_test and not java_test rule. I used this diff: [1] on Gerrit stable-2.14 branch.

And it's still failing with the same error message as in description of this issue:

  $ bazel test --config=remote \
  --remote_instance_name=projects/${PROJECT}/instances/default_instance \
  gerrit-pgm:pgm_license_test

The original code is here: [2]. What am I missing?

Update: I have sent this CL: [3], that didn't fix the problem.

[1] http://paste.openstack.org/show/782730
[2] https://github.com/GerritCodeReview/gerrit/blob/stable-2.14/tools/bzl/license.bzl#L34-L57
[3] https://gerrit-review.googlesource.com/c/gerrit/+/240248

scottmin0r · 2019-10-10T16:53:24Z

I'm running into a similar issue updating from 0.28.0 to 1.0.0.

This happens when I migrate default cpp toolchains from @bazel_tools//tools/cpp:default-toolchain to @bazel_tools//tools/cpp:toolchain (see #8546). I guess this changes the toolchain type from being static to cc_autoconf, which triggers this issue building remotely:

INFO: Call stack for the definition of repository 'local_config_cc' which is a cc_autoconf (rule definition at /usr/local/google/home/minors/.cache/bazel/_bazel_minors/86021668b1f881516e569ba9a9189b20/external/bazel_tools/tools/cpp/cc_configure.bzl:143:15):                                                                                                                                                                                                                                       
 - /usr/local/google/home/minors/.cache/bazel/_bazel_minors/86021668b1f881516e569ba9a9189b20/external/bazel_tools/tools/cpp/cc_configure.bzl:179:5                                                                                                                                                                                                                                                                                                                                                      
 - /DEFAULT.WORKSPACE.SUFFIX:385:1                                                        
ERROR: An error occurred during the fetch of repository 'local_config_cc':
   Traceback (most recent call last):           
        File "/usr/local/google/home/minors/.cache/bazel/_bazel_minors/86021668b1f881516e569ba9a9189b20/external/bazel_tools/tools/cpp/cc_configure.bzl", line 104
                repository_ctx.symlink(paths("@bazel_tools//tools/cpp:B..."), ...")
        File "/usr/local/google/home/minors/.cache/bazel/_bazel_minors/86021668b1f881516e569ba9a9189b20/external/bazel_tools/tools/cpp/cc_configure.bzl", line 104, in repository_ctx.symlink                                                                                                                                                                                                                                                                                                           
                paths("@bazel_tools//tools/cpp:BUILD.e...")
'dict' object is not callable                        
ERROR: /usr/local/google/home/minors/.cache/bazel/_bazel_minors/86021668b1f881516e569ba9a9189b20/external/bazel_tools/tools/cpp/cc_configure.bzl:180:5: //external:cc_toolchain depends on @local_config_cc//:toolchain in repository @local_config_cc which failed to fetch. no such package '@local_config_cc//': Traceback (most recent call last):                                                                                                                                                  
        File "/usr/local/google/home/minors/.cache/bazel/_bazel_minors/86021668b1f881516e569ba9a9189b20/external/bazel_tools/tools/cpp/cc_configure.bzl", line 104
                repository_ctx.symlink(paths("@bazel_tools//tools/cpp:B..."), ...")
        File "/usr/local/google/home/minors/.cache/bazel/_bazel_minors/86021668b1f881516e569ba9a9189b20/external/bazel_tools/tools/cpp/cc_configure.bzl", line 104, in repository_ctx.symlink                                                                                                                                                                                                                                                                                                           
                paths("@bazel_tools//tools/cpp:BUILD.e...")                                                               
'dict' object is not callable

I have a temporary workaround, which is to not migrate from static toolchains and to pass --noincompatible_disable_static_cc_toolchains instead.

davido · 2019-10-11T07:30:33Z

Not sure what changed, but now it always failing for me, even on other rules:

  $ bazel test --config=remote --remote_instance_name=projects/${PROJECT}/instances/default_instance //javatests/com/google/gerrit/acceptance/...
[...]
  Internal error thrown during build. Printing stack trace: java.lang.IllegalArgumentException: action terminated with unexpected exception: no such package '@local_config_cc//': Traceback (most recent call last):
	File "/home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/bazel_tools/tools/cpp/cc_configure.bzl", line 104
		repository_ctx.symlink(<2 more arguments>)
	File "/home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/bazel_tools/tools/cpp/cc_configure.bzl", line 104, in repository_ctx.symlink
		paths(<1 more arguments>)
'dict' object is not callable

davido · 2019-10-11T14:13:52Z

Not sure what is going on here, but it seems non-deterministic. Now it works here, for rule: //javatests/com/google/gerrit/acceptance/... that failed yesterday:

  $ bazel test --config=remote \
  --remote_instance_name=projects/${PROJECT}/instances/default_instance \
  //javatests/com/google/gerrit/acceptance/...
  [...]
INFO: Analyzed 77 targets (12 packages loaded, 5567 targets configured).
INFO: Found 77 test targets...
INFO: Elapsed time: 378.232s, Critical Path: 363.42s
INFO: 688 processes: 378 remote cache hit, 310 remote.
INFO: Build completed successfully, 1052 total actions
//javatests/com/google/gerrit/acceptance:acceptance_framework_tests      PASSED in 37.2s
//javatests/com/google/gerrit/acceptance/annotation:annotation           PASSED in 56.9s
//javatests/com/google/gerrit/acceptance/api/accounts:api_account        PASSED in 70.5s
//javatests/com/google/gerrit/acceptance/api/change:api_change           PASSED in 125.5s
//javatests/com/google/gerrit/acceptance/api/config:api_config           PASSED in 20.6s
//javatests/com/google/gerrit/acceptance/api/group:api_group             PASSED in 56.5s
//javatests/com/google/gerrit/acceptance/api/plugin:api_plugin           PASSED in 78.3s
//javatests/com/google/gerrit/acceptance/api/project:api_project         PASSED in 41.9s
//javatests/com/google/gerrit/acceptance/api/revision:api_revision       PASSED in 138.1s
//javatests/com/google/gerrit/acceptance/edit:edit                       PASSED in 48.4s
//javatests/com/google/gerrit/acceptance/git:ForcePushIT                 PASSED in 20.4s
//javatests/com/google/gerrit/acceptance/git:GitmodulesIT                PASSED in 21.7s
//javatests/com/google/gerrit/acceptance/git:HttpPushForReviewIT         PASSED in 67.9s
//javatests/com/google/gerrit/acceptance/git:ImplicitMergeCheckIT        PASSED in 24.1s
//javatests/com/google/gerrit/acceptance/git:PushPermissionsIT           PASSED in 30.5s
//javatests/com/google/gerrit/acceptance/git:RefAdvertisementIT          PASSED in 79.0s
//javatests/com/google/gerrit/acceptance/git:RefOperationValidationIT    PASSED in 22.3s
//javatests/com/google/gerrit/acceptance/git:SshPushForReviewIT          PASSED in 98.7s
//javatests/com/google/gerrit/acceptance/git:SubmitOnPushIT              PASSED in 31.3s
//javatests/com/google/gerrit/acceptance/git:SubmoduleSubscriptionsIT    PASSED in 48.0s
//javatests/com/google/gerrit/acceptance/git:SubmoduleSubscriptionsWholeTopicMergeIT PASSED in 99.1s
//javatests/com/google/gerrit/acceptance/pgm:pgm                         PASSED in 97.2s
//javatests/com/google/gerrit/acceptance/rest:rest_bindings_collection   PASSED in 70.3s
//javatests/com/google/gerrit/acceptance/rest/account:rest_account       PASSED in 58.0s
//javatests/com/google/gerrit/acceptance/rest/auth:auth                  PASSED in 16.8s
//javatests/com/google/gerrit/acceptance/rest/binding:rest_bindings      PASSED in 115.1s
//javatests/com/google/gerrit/acceptance/rest/change:SubmitByCherryPickIT PASSED in 84.7s
//javatests/com/google/gerrit/acceptance/rest/change:SubmitByFastForwardIT PASSED in 65.0s
//javatests/com/google/gerrit/acceptance/rest/change:SubmitByMergeAlwaysIT PASSED in 73.7s
//javatests/com/google/gerrit/acceptance/rest/change:SubmitByMergeIfNecessaryIT PASSED in 94.8s
//javatests/com/google/gerrit/acceptance/rest/change:SubmitByRebaseAlwaysIT PASSED in 80.2s
//javatests/com/google/gerrit/acceptance/rest/change:SubmitByRebaseIfNecessaryIT PASSED in 94.8s
//javatests/com/google/gerrit/acceptance/rest/change:SubmitResolvingMergeCommitIT PASSED in 35.4s
//javatests/com/google/gerrit/acceptance/rest/change:rest_change_other   PASSED in 146.7s
//javatests/com/google/gerrit/acceptance/rest/config:rest_config         PASSED in 42.7s
//javatests/com/google/gerrit/acceptance/rest/group:rest_group           PASSED in 21.1s
//javatests/com/google/gerrit/acceptance/rest/project:AccessIT           PASSED in 29.5s
//javatests/com/google/gerrit/acceptance/rest/project:BanCommitIT        PASSED in 22.3s
//javatests/com/google/gerrit/acceptance/rest/project:CheckMergeabilityIT PASSED in 26.0s
//javatests/com/google/gerrit/acceptance/rest/project:CreateBranchIT     PASSED in 23.8s
//javatests/com/google/gerrit/acceptance/rest/project:CreateProjectIT    PASSED in 29.5s
//javatests/com/google/gerrit/acceptance/rest/project:DeleteBranchIT     PASSED in 24.7s
//javatests/com/google/gerrit/acceptance/rest/project:DeleteBranchesIT   PASSED in 22.9s
//javatests/com/google/gerrit/acceptance/rest/project:DeleteTagIT        PASSED in 22.6s
//javatests/com/google/gerrit/acceptance/rest/project:DeleteTagsIT       PASSED in 17.3s
//javatests/com/google/gerrit/acceptance/rest/project:FileBranchIT       PASSED in 20.9s
//javatests/com/google/gerrit/acceptance/rest/project:GarbageCollectionIT PASSED in 29.3s
//javatests/com/google/gerrit/acceptance/rest/project:GetChildProjectIT  PASSED in 22.5s
//javatests/com/google/gerrit/acceptance/rest/project:GetCommitIT        PASSED in 21.9s
//javatests/com/google/gerrit/acceptance/rest/project:GetProjectIT       PASSED in 21.8s
//javatests/com/google/gerrit/acceptance/rest/project:ListBranchesIT     PASSED in 21.7s
//javatests/com/google/gerrit/acceptance/rest/project:ListChildProjectsIT PASSED in 21.8s
//javatests/com/google/gerrit/acceptance/rest/project:ListCommitFilesIT  PASSED in 24.8s
//javatests/com/google/gerrit/acceptance/rest/project:ListProjectsIT     PASSED in 48.8s
//javatests/com/google/gerrit/acceptance/rest/project:PluginAccessIT     PASSED in 21.4s
//javatests/com/google/gerrit/acceptance/rest/project:ProjectLevelConfigIT PASSED in 22.6s
//javatests/com/google/gerrit/acceptance/rest/project:PushAnnotatedTagIT PASSED in 25.1s
//javatests/com/google/gerrit/acceptance/rest/project:PushLightweightTagIT PASSED in 36.5s
//javatests/com/google/gerrit/acceptance/rest/project:TagsIT             PASSED in 23.1s
//javatests/com/google/gerrit/acceptance/rest/revision:rest_revision     PASSED in 22.5s
//javatests/com/google/gerrit/acceptance/server/account:server_account   PASSED in 19.7s
//javatests/com/google/gerrit/acceptance/server/change:server_change     PASSED in 62.6s
//javatests/com/google/gerrit/acceptance/server/event:server_event       PASSED in 23.9s
//javatests/com/google/gerrit/acceptance/server/git/receive:receive      PASSED in 26.4s
//javatests/com/google/gerrit/acceptance/server/mail:ChangeNotificationsIT PASSED in 109.3s
//javatests/com/google/gerrit/acceptance/server/mail:EmailValidatorIT    PASSED in 24.0s
//javatests/com/google/gerrit/acceptance/server/mail:ListMailFilterIT    PASSED in 29.4s
//javatests/com/google/gerrit/acceptance/server/mail:MailIT              PASSED in 23.1s
//javatests/com/google/gerrit/acceptance/server/mail:MailMetadataIT      PASSED in 24.0s
//javatests/com/google/gerrit/acceptance/server/mail:MailProcessorIT     PASSED in 30.3s
//javatests/com/google/gerrit/acceptance/server/mail:MailSenderIT        PASSED in 27.7s
//javatests/com/google/gerrit/acceptance/server/mail:NotificationMailFormatIT PASSED in 24.8s
//javatests/com/google/gerrit/acceptance/server/notedb:server_notedb     PASSED in 19.5s
//javatests/com/google/gerrit/acceptance/server/project:server_project   PASSED in 37.9s
//javatests/com/google/gerrit/acceptance/server/quota:server_quota       PASSED in 29.8s
//javatests/com/google/gerrit/acceptance/server/rules:server_rules       PASSED in 23.8s
//javatests/com/google/gerrit/acceptance/ssh:ssh                         PASSED in 90.2s

Executed 77 out of 77 tests: 77 tests pass.
INFO: Build completed successfully, 1052 total actions

hanwen · 2019-11-05T14:53:50Z

I can get the tests to run if I disable

license_test()

by inserting a return at the top, but this is clearly not something we want to submit. Tips?

ulfjack · 2019-12-05T00:25:49Z

It's not the test that's failing, but the setup of the C++ toolchain that's failing. The built-in sh_test implicitly depends on the C++ toolchain, and apparently the rest of your codebase does not. I suspect that's not easily fixable in Bazel.

What you could do is write your own test rule in Starlark, or rewrite test_license.sh as a Java test (or in some other language). I wonder if there's already Starlark code for a forbidden dependency check somewhere that you could reuse.

davido · 2019-12-05T03:42:25Z

The built-in sh_test implicitly depends on the C++ toolchain, and apparently the rest of your codebase does not.

That's strange, because Gerrit depends on rules_closure that depends on protobuf, and clean gerrit build would rebuild protobuf project from source and would rebuild C++ source code base.

hanwen · 2019-12-05T10:01:23Z

Thanks, Ulf, I'll try that.

(I don't quite understand though: we depend on protobuf, which is C++. Does that work because the protobuf dependency is in the @protobuf repository?)

hanwen · 2019-12-05T11:56:52Z

Looks like py_test has the same problem.

hanwen · 2019-12-05T12:20:47Z

Is there any publicly available rule set for low-overhead programming (python, shell) languages you can recommend?

ulfjack · 2019-12-05T12:29:44Z

My diagnosis is apparently incorrect, interesting. Maybe it's the genrule that's triggering the C++ toolchain config, not the test? Can you build the genrule, or does that also trigger the error?

hanwen · 2019-12-05T15:51:02Z

@ulfjack

It looks like the CC toolchain gets dragged in through the genquery, see

https://gerrit-review.googlesource.com/c/gerrit/+/247613

The genquery is just above the changed code. I'm only interested in dependencies that might reach into //lib:LICENSE-xxx files. I tried add --noimplicit_deps , --notool_deps, but it's in vain.

davido · 2020-01-25T14:14:23Z

This seems to work now on Bazel@HEAD (5b95663):

  $ bazel test --config=remote --remote_instance_name=<project> javatests/com/google/gerrit/pgm:pgm_license_test
[...]
INFO: Analyzed target //javatests/com/google/gerrit/pgm:pgm_license_test (12 packages loaded, 7 targets configured).
INFO: Found 1 test target...
Target //javatests/com/google/gerrit/pgm:pgm_license_test up-to-date:
  bazel-bin/javatests/com/google/gerrit/pgm/pgm_license_test
INFO: Elapsed time: 13.632s, Critical Path: 10.92s
INFO: 2 processes: 2 remote.
INFO: Build completed successfully, 6 total actions
//javatests/com/google/gerrit/pgm:pgm_license_test                       PASSED in 5.6s

Executed 1 out of 1 test: 1 test passes.
INFO: Build completed successfully, 6 total actions

hanwen · 2020-02-04T12:41:29Z

still broken on

$ bazel version
Build label: 2.0.0
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Thu Dec 19 12:30:18 2019 (1576758618)
Build timestamp: 1576758618
Build timestamp as int: 1576758618

davido · 2020-02-04T14:08:06Z

Have you noticed my last comment?

This seems to work now on Bazel@HEAD (5b95663)

hanwen · 2020-02-04T14:21:34Z

yes, I did. It's more a note to self.

I'll ask bazel team when the next release will be out.

iirina added team-Remote-Exec Issues and PRs for the Execution (Remote) team untriaged labels Oct 10, 2019

buchgr added type: feature request and removed untriaged labels Oct 10, 2019

davido closed this as completed Jan 25, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

repository_ctx.symlink error when testing remotely using RBE #9979

repository_ctx.symlink error when testing remotely using RBE #9979

davido commented Oct 9, 2019 •

edited

Loading

buchgr commented Oct 10, 2019

davido commented Oct 10, 2019 •

edited

Loading

buchgr commented Oct 10, 2019

davido commented Oct 10, 2019

davido commented Oct 10, 2019 •

edited

Loading

scottmin0r commented Oct 10, 2019

davido commented Oct 11, 2019 •

edited

Loading

davido commented Oct 11, 2019 •

edited

Loading

hanwen commented Nov 5, 2019

ulfjack commented Dec 5, 2019

davido commented Dec 5, 2019 •

edited

Loading

hanwen commented Dec 5, 2019

hanwen commented Dec 5, 2019

hanwen commented Dec 5, 2019

ulfjack commented Dec 5, 2019

hanwen commented Dec 5, 2019

davido commented Jan 25, 2020 •

edited

Loading

hanwen commented Feb 4, 2020

davido commented Feb 4, 2020

hanwen commented Feb 4, 2020

repository_ctx.symlink error when testing remotely using RBE #9979

repository_ctx.symlink error when testing remotely using RBE #9979

Comments

davido commented Oct 9, 2019 • edited Loading

buchgr commented Oct 10, 2019

davido commented Oct 10, 2019 • edited Loading

buchgr commented Oct 10, 2019

davido commented Oct 10, 2019

davido commented Oct 10, 2019 • edited Loading

scottmin0r commented Oct 10, 2019

davido commented Oct 11, 2019 • edited Loading

davido commented Oct 11, 2019 • edited Loading

hanwen commented Nov 5, 2019

ulfjack commented Dec 5, 2019

davido commented Dec 5, 2019 • edited Loading

hanwen commented Dec 5, 2019

hanwen commented Dec 5, 2019

hanwen commented Dec 5, 2019

ulfjack commented Dec 5, 2019

hanwen commented Dec 5, 2019

davido commented Jan 25, 2020 • edited Loading

hanwen commented Feb 4, 2020

davido commented Feb 4, 2020

hanwen commented Feb 4, 2020

davido commented Oct 9, 2019 •

edited

Loading

davido commented Oct 10, 2019 •

edited

Loading

davido commented Oct 10, 2019 •

edited

Loading

davido commented Oct 11, 2019 •

edited

Loading

davido commented Oct 11, 2019 •

edited

Loading

davido commented Dec 5, 2019 •

edited

Loading

davido commented Jan 25, 2020 •

edited

Loading