Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

oci_image_index() fails when using remote execution on Bazel 7 #477

Closed
mortenmj opened this issue Jan 26, 2024 · 2 comments
Closed

oci_image_index() fails when using remote execution on Bazel 7 #477

mortenmj opened this issue Jan 26, 2024 · 2 comments
Labels
bug Something isn't working need: investigation A potential issue which we need to investigate first
Milestone
2.0

Comments

@mortenmj
Copy link
Contributor

When using Bazel 7.0.0 and remote execution, building a multi-architecture image as in the rules_oci example fails with an error like this:

ERROR: /workspaces/infrastructure/services/robotics/video-agent/BUILD.bazel:110:16: OCI Index //services/robotics/video-agent:image_multiarch failed: (Exit 1): image_index_image_multiarch.sh failed: error executing OCIIndex command (from target //services/robotics/video-agent:image_multiarch) 
  (cd /home/vscode/.cache/bazel/_bazel_vscode/3dbdfd10a1e163b9a670d9ed81dfe3af/execroot/_main && \
  exec env - \
  bazel-out/rbe_linux_amd64-fastbuild/bin/services/robotics/video-agent/image_index_image_multiarch.sh '--output=bazel-out/rbe_linux_amd64-fastbuild/bin/services/robotics/video-agent/image_multiarch' '--image=bazel-out/linux_amd64-fastbuild/bin/services/robotics/video-agent/image.image' '--blob=blobs/sha256/09c50f98c494cfeb47773210153a03346536b3ffaac2756ae024ceed63ce83d7' '--blob=blobs/sha256/1b26523ec02008554fd7e97b3b48cca84502a987e75f0fb4c2d44a256fea6488' '--blob=blobs/sha256/1e3d9b7d145208fa8fa3ee1c9612d0adaac7255f1bbc9ddea7e461e0b317805c' '--blob=blobs/sha256/1f1e45d2678886af27304ab6c29b527a5edf8259ed2449466b05e2d01714eaf7' '--blob=blobs/sha256/2b776ada03417eaa87102a617f964324df1de8967698fc4209dc1a1fbdfae8cd' '--blob=blobs/sha256/2fa82a9c76b2292e1c056fb1ed61bd0d29ddb196bbd3ac1a60d29d5eecc9e553' '--blob=blobs/sha256/3996ddec0b8122956c7d44b0ebf3cbfec8720a97fd6470614b5c1054c24e7d6f' '--blob=blobs/sha256/4aa0ea1413d37a58615488592a0b827ea4b2e48fa5a77cf707d0e35f025e613f' '--blob=blobs/sha256/62a9f8728f85475521a1a82ded705c87d02c3e8b9fac4fbeae080720d1820822' '--blob=blobs/sha256/672354a91bfa8da293349f3384c1a32f35a0791ae4658a17967e3b4aedd3554b' '--blob=blobs/sha256/6c1d328a67046a4d2b426d92539496c614602ee5708fe9fc67abdeb5d1c523f8' '--blob=blobs/sha256/7145a5563bee3311f10aa91fc348c00a29ce2d701646763db2b4aeba72efb11d' '--blob=blobs/sha256/8d3d080da81b84ce07bc613e33c9dba0de033c3910ee6e939bad542c7944487e' '--blob=blobs/sha256/98d0b6b328c8900a974d376511124b7b96dcc31113d8b9fa1adef2bebfce0f99' '--blob=blobs/sha256/ad7888c7ea2f06bd1042ec3ee1556efd655a332717974c0e2d9a113c1a51c565' '--blob=blobs/sha256/e8c73c638ae9ec5ad70c49df7e484040d889cca6b4a9af056579c3d058ea93f0' '--blob=blobs/sha256/fcb6f6d2c9986d9cd6a2ea3cc2936e5fc613e09f1af9042329011e43057f3265' '--image=bazel-out/linux_arm64-fastbuild/bin/services/robotics/video-agent/image.image' '--blob=blobs/sha256/1e3d9b7d145208fa8fa3ee1c9612d0adaac7255f1bbc9ddea7e461e0b317805c' '--blob=blobs/sha256/2b776ada03417eaa87102a617f964324df1de8967698fc4209dc1a1fbdfae8cd' '--blob=blobs/sha256/2b9c28259b9a51be6fda265b3861ef80c8fefe1b2b887d7f30d685014803cbed' '--blob=blobs/sha256/2fa82a9c76b2292e1c056fb1ed61bd0d29ddb196bbd3ac1a60d29d5eecc9e553' '--blob=blobs/sha256/3996ddec0b8122956c7d44b0ebf3cbfec8720a97fd6470614b5c1054c24e7d6f' '--blob=blobs/sha256/45c2519a3853b7f16991b74ce5d02f36af27000706f086498152a0e4d2682176' '--blob=blobs/sha256/4aa0ea1413d37a58615488592a0b827ea4b2e48fa5a77cf707d0e35f025e613f' '--blob=blobs/sha256/672354a91bfa8da293349f3384c1a32f35a0791ae4658a17967e3b4aedd3554b' '--blob=blobs/sha256/6b3537144156119ceb6ad319ed62d471ae68bc65e10df78e18afa64962ab83fd' '--blob=blobs/sha256/810b5cc4682abbbceeb9ce0a760ac22545cae3c2f5c27cad93932b0bb828ec46' '--blob=blobs/sha256/972a9f56458fc74293b2fa3a1eac98369d424131796b93f3c1d685a7eb938cf8' '--blob=blobs/sha256/b6b9df2318a1387dae851d5b888b508da16f340427f8399e40a75ff92ed880bb' '--blob=blobs/sha256/d192bb73e81f99d9907a58ab59b63f20e8567a1469137d66ea7ec8f6119f8f87' '--blob=blobs/sha256/ddfc3c49c0f1c8904f08efb7566ec4acad4916c740abc89b3b68d2094f86c2bf' '--blob=blobs/sha256/df11e75a3d28b26a5fbfa5e0805d7629de3d86111848bc5097fd73868c4baf70' '--blob=blobs/sha256/e8c73c638ae9ec5ad70c49df7e484040d889cca6b4a9af056579c3d058ea93f0' '--blob=blobs/sha256/fcb6f6d2c9986d9cd6a2ea3cc2936e5fc613e09f1af9042329011e43057f3265')
# Configuration: eabe50dbaf268ec1ec12679bfacb394f12cbd3b9c9d59b549d8520599966d026
# Execution platform: //:rbe_linux_amd64
ln: 'bazel-out/linux_arm64-fastbuild/bin/services/robotics/video-agent/image.image/blobs/sha256/1e3d9b7d145208fa8fa3ee1c9612d0adaac7255f1bbc9ddea7e461e0b317805c' and 'bazel-out/rbe_linux_amd64-fastbuild/bin/services/robotics/video-agent/image_multiarch/blobs/sha256/1e3d9b7d145208fa8fa3ee1c9612d0adaac7255f1bbc9ddea7e461e0b317805c' are the same file
Target //services/robotics/video-agent:image_multiarch failed to build

Tagging the oci_image_index() target with "no-remote-exec" resolves the issue, but it would be very useful if this rule worked with remote execution.
@thesayyn thesayyn added bug Something isn't working enhancement New feature or request need: investigation A potential issue which we need to investigate first and removed enhancement New feature or request labels Feb 20, 2024
@bmclarnon
Copy link

I ran into a similar issue with oci_image (bazel 6.2.1, rules_oci 1.7.4); I tracked it down to zot_launcher.sh failing because TMPDIR wasn't set: https://github.com/bazel-contrib/rules_oci/blob/v1.7.4/oci/private/registry/zot_launcher.sh.tpl#L18. I suspect TMPDIR is unset because of bazelbuild/bazel#2574.

I was able to work around the issue by providing a fallback (e.g., ${TMPDIR:-/tmp}), but I'm not sure if that's sufficiently portable. I didn't verify whether it also fixed oci_image_index.

bmclarnon added a commit to google-parfait/confidential-federated-compute that referenced this issue Mar 25, 2024
@bmclarnon
Add remote execution configuration.
1994c11 
Enabling remote execution requires passing `--config=cfc-remote`, which
will allow us to roll out remote execution slowly -- and roll it back if
needed.

Several targets and tests are incompatible with remote execution had
have been tagged as such; these targets will be addressed in subsequent
changes:

  * `oci_image` doesn't work with remote execution, though a newer
    version may.
    bazel-contrib/rules_oci#477
  * `pip_parse` builds source dependencies based on the host platform,
    which (a) introduces non-determinism and (b) can result in binaries
    that cannot run on the remote host (e.g., due to incompatible glibc
    versions). We may be able to use annotations to substitute in
    versions of dependencies built by bazel, not pip:
    bazelbuild/rules_python#571.
    But we also might be able to remove the Python targets...

Bug: 321291571
Change-Id: Id4b2ece75f7945736e0e3c6bba056c85842e9618
@thesayyn thesayyn added this to the 2.0 milestone May 8, 2024
@thesayyn
Copy link
Collaborator

Fixed by #525

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working need: investigation A potential issue which we need to investigate first
Projects
None yet
Milestone
2.0
Development

No branches or pull requests
3 participants
@mortenmj @bmclarnon @thesayyn