Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Release 4.1 - May 2021 #13099

Closed
7 of 9 tasks
philwo opened this issue Feb 24, 2021 · 107 comments
Closed
7 of 9 tasks

Release 4.1 - May 2021 #13099

philwo opened this issue Feb 24, 2021 · 107 comments
Assignees
@philwo
Labels
P1 I'll work on this now. (Assignee required) release team-OSS Issues for the Bazel OSS team: installation, release processBazel packaging, website type: process

Comments

@philwo
Copy link
Member

philwo commented Feb 24, 2021 •
edited
Loading

Status of Bazel 4.1

This release will use Bazel 4.0.0 as its baseline and we will apply selected cherry-picks and backports on top of it. Please request cherry-picks that you'd like to get into Bazel 4.1.0 here via a comment.

To report a release-blocking bug, please file a bug using the Release blocker label, and cc me.

Task list:
@philwo philwo added P1 I'll work on this now. (Assignee required) type: process release team-OSS Issues for the Bazel OSS team: installation, release processBazel packaging, website labels Feb 24, 2021
@philwo philwo self-assigned this Feb 24, 2021
@philwo
Copy link
Member Author

philwo commented Feb 24, 2021 •
edited by lberki
Loading

Requested cherry picks

✅ Bazel 4.1.0rc1:

✅ Bazel 4.1.0rc2:

✅ Bazel 4.1.0rc3:

✅ Bazel 4.1.0rc4

⚡ Danger zone:

🙅🏻 Rejected:

@philwo philwo pinned this issue Feb 24, 2021
@philwo philwo mentioned this issue Feb 24, 2021
Closed
10 tasks
@dmivankov
Copy link
Contributor

bef4bbb would be nice to have too

@gjasny
Copy link
Contributor

gjasny commented Feb 24, 2021

Hello,

If it's not already fixed on the branch 24d0864 / #12879 would be nice to have in 4.1

Thanks,
Gregor

@coeuvre
Copy link
Member

coeuvre commented Feb 26, 2021 •
edited
Loading

Remote Execution

External Repository

@EdSchouten
Copy link
Contributor

I’d love it if we could get #12886 and #13085 in there. Both of those are small changes and make life easier for me...

@illicitonion
Copy link
Contributor

Can I add 63bc1c7 to the External Repository list? (It's a follow-up from b243584 which is already in the list)

@philwo
Copy link
Member Author

philwo commented Feb 26, 2021

@meteorcloudy @coeuvre Should we cherry-pick the gRPC version upgrade, too?

#13104
#13105
#13106

@philsc
Copy link
Contributor

philsc commented Feb 27, 2021

Some doc updates would be nice:
2eb1bf5
c71697c

@coeuvre
Copy link
Member

coeuvre commented Mar 1, 2021

@meteorcloudy @coeuvre Should we cherry-pick the gRPC version upgrade, too?

#13104
#13105
#13106

Yes, it's the fix for #12927.

Can I add 63bc1c7 to the External Repository list? (It's a follow-up from b243584 which is already in the list)

Sure.

@scele scele mentioned this issue Mar 1, 2021
Closed
@scele
Copy link
Contributor

scele commented Mar 1, 2021

Yes, it's the fix for #12927.

FYI, the second gRPC upgrade MR #13105 doesn't apply cleanly on 4.0 - I've rebased the commits onto 4.0 in this branch if you want to leverage it when preparing the release branch.

@denyska
Copy link
Contributor

denyska commented Mar 3, 2021

For the external repos 495ac92 would be nice cc @philwo @coeuvre

@coeuvre
Copy link
Member

coeuvre commented Mar 3, 2021

For the external repos 495ac92 would be nice

Added to the list.

@illicitonion
Copy link
Contributor

Also a request for dad9630 so that remote execution stacks can start using the newer APIs.

@liucijus
Copy link
Contributor

liucijus commented Mar 3, 2021

We are waiting for ba60c0b in rules_scala

@philsc
Copy link
Contributor

philsc commented Mar 5, 2021

dfb70ea: This would let users select() on constraints in filegroup() targets.

ad241fb: Let users use cquery to filter out incompatible targets.

@quval
Copy link
Contributor

quval commented Mar 6, 2021 •
edited
Loading

We'd be happy to see 1e258d2 and d067669 get in. (There's also the related f1e0d34, which is a no-op, and 8186fbb with documentation for this change.)

@philsc
Copy link
Contributor

philsc commented Mar 12, 2021

I'd also really like to get 321fe3b in there if possible.

@coeuvre
Copy link
Member

coeuvre commented Mar 12, 2021

Also a request for dad9630 so that remote execution stacks can start using the newer APIs.

Added to the list.

@philwo
Copy link
Member Author

philwo commented Mar 12, 2021

All cherry-picks that have been requested until now have been added to my or Chi's list. I'll try to harvest the cherries now and create 4.1.0rc1. :D

@philwo
Copy link
Member Author

philwo commented Mar 15, 2021 •
edited
Loading

Bazel 4.1.0rc1 is out 🚀 🌔

Notes:

Commands ran:

BASELINE_COMMIT=6b33bdb1e22514304c0e35ce8e067f2175685245 # 4.0.0
CHERRY_PICKS=(
    8811e27353c2c10980faf7e4c5e44b431d2d4f1c  # Fix error message from getPrerequisites to not print internal details. [added]
    27e15ad11410eb1014f5247fd0eeb31a46733c07  # Clean up ConfiguredTargetValueAccessor and ConfiguredTargetAccessor. [added]
    e87feb8ac9573cef993824f82370d0389570521d  # Move getConfigConditions into ConfiguredTarget. [added]
    34d98234324da83e93ba0d5ef5702880d5ac7c5c  # Change ConfiguredTargetQuery to use KeyedConfiguredTarget as a value.
    079bb7d69931705bb2b092c9017090e224ef3043  # Clean up old dependencies that are unused since 34d9823. [added]
    e03cb63e059420847d6578d7cbfe93f05615c95e  # Update bazelbuild/platforms to a current release.
    2eb1bf53d5fef13b89ee440af4f83003d1d0b50a  # Update docs and tests to use the @platforms//:incompatible constraint.
    c71697cf33b0fbbb42fc2910bac83960edc7e855  # Clarify test_suite behaviour in the Platforms docs.
    dfb70ea4cae2ffffb76e9741d86c96505a6d05ad  # Enable toolchain resolution for filegroup targets.
    24d086446f74606819dc53c3a436caa056ff05b7  # PlatformProviderUtils should ignore targets that don't have the needed provider.
    ba60c0b3f9bbd00975c984244839b155e84b4c5d  # ijar: fix manifest sections handling.
    58bb42ad7ca263a75c6eeef51482f805726663a5  # Revert "Switch to -fdebug-compilation-dir".
    bef4bbbb47d47befe3711d06f358782ee12554f9  # Update turbine. Built at google/turbine@1bbb136.
    ad241fbebd90a9f0ad65ccd0658838f57030db68  # Allow cquery to filter out incompatible targets.
    1782f0ae751569607ef88930c822ac460a1f8bb3  # Patch grpc to fix cares selecting the wrong source when building for darwin_arm64 cpu. [added]
    8f7bc2f67fafcaa8d25cfc77eaaedbf8eed2984a  # [1/3] Bump grpc to 1.33.1 to fix corruption when downloading CAS blobs.
    a573be53493c0ca9dfa45b84f855c4e1d6141fc5  # [2/3] Bump grpc to 1.33.1 to fix corruption when downloading CAS blobs.
    9b30172547f2093acb56aedf159a77d5dceffda2  # [3/3] Bump grpc to 1.33.1 to fix corruption when downloading CAS blobs
    1e258d2a7a5221613047e5cee0aaec5b56045d2b  # Allow exec groups to inherit from the rule or other exec groups.
    d0676693310215407224c1b8e8aea9e3eddc183d  # Support execution constraints per exec group.
    f1e0d346c8235c855e61afc2adb870e4b895e002  # Clean up RuleContext to use a Table instead of a Map of Maps.
    8186fbb47ab964a9affa9a0fc6315fcdbde2b5aa  # Documentation for #13110.
    321fe3b6b4e892821ee7dbf2d17dd8ae6a541913  # Prevent --repo_env from triggering unnecessary fetches.
    3ebf658cba43bbab1efc36518f0795a7d65e2d46  # Prevent a crash when using --repo_env=VAR without a value.
    913a985a5c2fc3842b12c6e5f29af0fa1bccfd6a  # Report digest of failed uploads.
    5122617b8a22fee7acd86c9c48f2c2737709ca3f  # Status error presentation with details.
    9a70805db543e2fb910e1c55ef3b3567362adf30  # Fix double shutdown of BuildEventArtifactUploader when BES+File output enabled. [added]
    325eb956c92530bdfda54a36a186cae4245a4f7b  # Add rxjava3 to third_party.
    b6e3ba8abf033c6d3a318be8484021eff6a40dde  # remote: set executable bit of an input file based on its real value.
    5b786da75837c5e29714e1d708c3cdf9a67ed32d  # Remote: correctly implement equals and hashCode.
    48648503729d53fdee1322fde2c8e6c05e99cff9  # Fixed an error that bazel binary is not executable when testing with remote execution.
    bc54c648aa1f99509c7c36d5e6b570d066689209  # Remote: Use parameters instead of thread-local storage to provide tracing metadata.
    92955e617b5c41713a5163dc0437c2a024b31815  # Remote: Use parameters instead of thread-local storage to provide tracing metadata. (Part 2)
    75bd1ff8ab56d241916bde36291301fa026b2bab  # Remote: Use parameters instead of thread-local storage to provide tracing metadata. (Part 3)
    37ee252f3744abc4511f55b5089cc52abd3ba09d  # Remote: Use parameters instead of thread-local storage to provide tracing metadata. (Part 4)
    f8d49faaa886321ea26e6d18c741368c129a37dc  # Remote: Use parameters instead of thread-local storage to provide tracing metadata. (Part 5)
    32fc451600b6e94a015263eb1c8a63e974f6f4cc  # Write/QueryWriteStatus logging refinement/addition.
    97963c5bb24ac79eb3646dd61bfcf2f8a648af54  # Remote: gRPC load balancing. (Part 1)
    e2b9a42a61596b0d24f0cadd6b7157b7f1efb221  # Remote: gRPC load balancing. (Part 2)
    6667ad7dd77f8d97952133052c17e7779c1430ec  # Remote: gRPC load balancing. (Part 3)
    7c081eb020186bfb16d4ef1c3832a8e946e99da1  # Remote: gRPC load balancing. (Part 4)
    17afbe4e224b359fee6415a5bd71bbedaa7843eb  # Implement getMessage for BulkTransferException.
    de8f69dd9ef2e06d2809a2ae3eedc97810e0d181  # Remote: gRPC load balancing. (Part 5)
    7a62c2d4e27e398f440910c81eacc384f38ca8be  # Remote: Add interoperability between Rx and ListenableFuture.
    1fcb18a0b455bfcb8e9940778f37d8c82c5ed5a0  # Update to latest remote-execution proto.
    dad96301d12aa77eb67399e08265a5f30f5ffd6a  # Set Platform on Action not just Command
    6c5a3ee0dcbb4b804f4aa85c038a378fb70eb1f9  # Remote: Add AsyncTaskCache which is used to deduplicate task executions and cache the results.
    7dc6f57cb639417c2d80cb19def22676d2b63b50  # Remote: Use AsyncTaskCache inside RemoteActionInputFetcher.
    9804ba5b88e241ac94e25aac6288b307789f0db5  # Add --experimental_repository_disable_download to allow users disable download for external repos.
    b243584a479eb4481a9bf4f69acc899610a3b630  # Report errors parsing rewriter config file.
    63bc1c7d0853dc187e4b96a490d733fb29f79664  # Downloader rewriter config has all_blocked_message.
    495ac923f398443be45c20ab29d183fe47e08911  # Allow UrlRewriter to change protocol, i.e. https->http, and http->https.
    8dbbde0037264c1db4b229a09f98a61ab4ca06b0  # Allow overriding the hostname and instance name in bytestream:// URIs.
)
git clone https://github.com/bazelbuild/bazel.git ~/bazel-release-$RELEASE_NUMBER
cd ~/bazel-release-$RELEASE_NUMBER
git remote add scele https://github.com/scele/bazel.git
git fetch scele issue_12927_bazel4
scripts/release/release.sh create $RELEASE_NUMBER $BASELINE_COMMIT "${CHERRY_PICKS[@]}"
scripts/release/release.sh push

@liucijus
Copy link
Contributor

We are waiting for ba60c0b in rules_scala

@philwo, I've tested 4.1.0rc1. I guess to properly include ba60c0b, bazel tools updated version needs to be included, right? Currently it seems to pick up previous stamping functionality.

@philwo
Copy link
Member Author

philwo commented Mar 15, 2021

@liucijus Indeed, sorry about that and thanks a lot for testing and letting me know!

I just asked my colleague @comius for help and he will prepare a java_tools release that we can cherry-pick into 4.1.0rc2 to ensure that we pick up the Turbine and ijar updates. 👍

@st0rm23
Copy link

st0rm23 commented May 11, 2021

@philwo can we cherry-pick this commit 2d990cf ? I hope we can get rid of python2 dependency ASAP.

@coeuvre coeuvre mentioned this issue May 12, 2021
Closed
@larsrc-google
Copy link
Contributor

@philwo Could you cherrypick the following:

5b95d91 - fixes occasional crash with local_lockfree_output
779d660, f4b5e02 - fixes that dynamic execution was still waiting for workers to finish after remote winning
b2231c5 - fixes a fallback to local execution that circumvents the strategy order
1a35dc5 - fixes occasional crash when trying to copy a non-readable output

@brentleyjones
Copy link
Contributor

If another rc is being cut, can aaae8ce make it in? It helps Apple Silicon support.

@wrowe
Copy link
Contributor

wrowe commented May 18, 2021

I don't yet see a 4.1.0 release, but also don't see a 4.2.0 fix issue so I'll bring this here;

6dc941e fixes a critical issue for windows software releases from GCP RBE, a bandaid to preserve any release-build symbolic debugging information. If this can either hit an rc of 4.1.0 or if a 4.2.x ticket is opened, it would be helpful to consider this asap, I have no practical means to test if it isn't on the 4.x branch (experimenting with current master is a little ahead of the project's progress.)

@robfig
Copy link

robfig commented May 18, 2021

c9f9eed is an important fix to support static analysis tooling with rules_go. The original bug was reported over a year ago, so would love to get this in 4.1. Thank you!

@fmeum
Copy link
Collaborator

fmeum commented May 20, 2021 •
edited
Loading

7acf9ea causes build failures explained in #13464, could it be reverted for 4.1.0?

@philwo
Copy link
Member Author

philwo commented May 20, 2021

Bazel 4.1.0rc5 with the reverted problematic commits has been created:

I would like to release this beginning of next week. @glukasiknuro could you please verify that this fixes the issue you saw with 4.1.0rc4?

I'll see that we get the remaining requests into 4.1.1 or 4.2.0 as soon as possible. I acknowledge that they look important, but I wanted to get 4.1.0 out, at last, without introducing more risks.

@fmeum The culprit you identified is already part of Bazel 4.0.0, thus we cannot revert it for 4.1.0. A fix or revert of that commit has to be created and pushed to the master branch first. If it doesn't cause issues, we can include the cherry-pick of that revert into the next 4.x release then!

Commands ran

git checkout -b release-4.1.0rc4-patches release-4.1.0rc4
git revert --no-edit --signoff \
    5f9456354a789436e8b5bcb03e7c0c8813afa148 \
    1aa544b60465765f77b714cdbf40fa082b0573a7 \
    cbefbd9e5453c084069219ef89403db36b591675 \
    3486b783735531f4edf265babdc40a8730cc427b

RELEASE_NUMBER=4.1.0
BASELINE_COMMIT=d7371772a884a7e630eb158c0e2cb4f7f8c143db # 4.1.0rc4
CHERRY_PICKS=(
    0e00b2703b79bd435d42536aed4bfcb8a2f75e6c  # Revert "Documentation for #13110"
    45cffa9b5e11fb32a97a51cd8f733434cdaf18af  # Revert "Clean up RuleContext to use a Table instead of a Map of Maps."
    9ba6469e3164c444bc2e534f4c8590049cddd3e4  # Revert "Support execution constraints per exec group"
    3ead920f83b5a0061682ca33d3d20428bf3852fb  # Revert "Allow exec groups to inherit from the rule or other exec groups."
)
scripts/release/release.sh create --force_rc=5 $RELEASE_NUMBER $BASELINE_COMMIT "${CHERRY_PICKS[@]}"
git shortlog release-4.1.0rc4...release-4.1.0rc5
git diff release-4.1.0rc4...release-4.1.0rc5
bazel clean --expunge
bazel build //src:bazel
bazel build --build_tests_only //scripts/... //src/java_tools/... //src/test/... //src/tools/execlog/... //src/tools/singlejar/... //src/tools/workspacelog/... //third_party/ijar/... //tools/android/... //tools/aquery_differ/... //tools/python/...
scripts/release/release.sh push

@glukasiknuro
Copy link
Contributor

@philwo confirming that the issue with exec_compatible_with is fixed in rc5 - both in reproduction in #13450 and actual codebase where the problem was detected. Thank you!

@philwo
Copy link
Member Author

philwo commented May 21, 2021

Bazel 4.1.0 has been released. Thank you all for your patience and help with getting this out the door.

Cherry-picks that could not be included will be considered for 4.1.1 or 4.2.0 asap.

@philwo philwo closed this as completed May 21, 2021
@philwo
Copy link
Member Author

philwo commented May 21, 2021 •
edited
Loading

Hey folks,

we noticed that bazel-toolchains has a broken version check and Bazel 4.1.0 triggers a code path in it that will cause all builds using bazel-toolchains to fail. This is not a bug in Bazel and there's nothing we could have done to prevent this, apart from never bumping our version to 4.1.0 in the first place. :/

The fix is to upgrade to bazel-toolchains 5.1.0, which has a fixed version check. Because bazel-toolchains has changed in a major way with version 5.0.0, this might not be straightforward (but still recommended). We are looking into providing a simple patch file that would fix the version check for bazel-toolchains 4.0.0 and which might be simpler to apply if you can't upgrade yet to 5.0.0.

We are tracking our own breakage here: #13502

@brentleyjones
Copy link
Contributor

brentleyjones commented May 21, 2021 •
edited
Loading

We are looking into providing a simple patch file that would fix the version check for bazel-toolchains 4.0.0 and which might be simpler to apply if you can't upgrade yet to 5.0.0

Could a 4.0.x release be made?

@philwo
Copy link
Member Author

philwo commented May 21, 2021

The patch for bazel-toolchains 4.0.0 is available here:

https://github.com/bazelbuild/bazel/blob/5b4320ec65f8da791354f9cc058f3ec2e2107cfb/third_party/bazel-toolchains/bazel-toolchains-4.0.0-fix-version-check.patch

You can use it in your http_archive rule in the WORKSPACE file via patches and patch_args like this:

f8f9dc8#diff-a7b969a66e4cf15e076089c4b3ddca51a3e02750d07938fbb7607af2645fdb9e

Could a 4.0.x release be made?

Done: https://github.com/bazelbuild/bazel-toolchains/releases/tag/4.1.0

@philwo philwo changed the title Release 4.1 - March 2021 Release 4.1 - May 2021 May 21, 2021
@philwo philwo mentioned this issue May 22, 2021
Closed
@timothyklim timothyklim mentioned this issue May 23, 2021
Merged
9 tasks
@msiebuhr
Copy link

I'd like to ask for the 4.1.0 (or indeed all versions) to have docker images available at https://l.gcr.io/google/bazel? I expect this is sort-of Google-internal thing, but I suspect a few Googles to be involved here :)

Alternatively, the documentation at https://docs.bazel.build/versions/4.0.0/bazel-container.html should be updated to point at an up-to-date registry, as :latest on l.gcr.io/google/bazel refers to Bazel 3.5.0.

@philwo
Copy link
Member Author

philwo commented May 25, 2021

@msiebuhr We will provide Docker containers for Bazel versions shortly. 👍

This was referenced May 25, 2021
Merged
Merged
@gregestren gregestren mentioned this issue May 26, 2021
Closed
@vbatts
Copy link

vbatts commented May 29, 2021

sorry for the delay. Fedora and Centos builds done.
Fedora 34 and rawhide(x86_64 and aarch64) builds failed. Looking like another issue with latest gcc.
https://copr.fedorainfracloud.org/coprs/vbatts/bazel/build/2215183/

�[1AERROR: /builddir/build/BUILD/bazel-4.1.0/third_party/ijar/BUILD:47:11: Compiling third_party/ijar/zlib_client.cc failed: (Exit 1): gcc failed: error executing command 
  (cd /var/tmp/bazel_TIraKP60/out/execroot/io_bazel && \
  exec env - \
    PATH=/builddir/build/BUILD/bazel-4.1.0/bin-hack:/builddir/.local/bin:/builddir/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/sbin \
    PWD=/proc/self/cwd \
  /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections '-std=c++0x' -MD -MF bazel-out/k8-opt/bin/third_party/ijar/_objs/zlib_client/zlib_client.d '-frandom-seed=bazel-out/k8-opt/bin/third_party/ijar/_objs/zlib_client/zlib_client.o' -iquote . -iquote bazel-out/k8-opt/bin -isystem third_party/zlib -isystem bazel-out/k8-opt/bin/third_party/zlib -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c third_party/ijar/zlib_client.cc -o bazel-out/k8-opt/bin/third_party/ijar/_objs/zlib_client/zlib_client.o)
Execution platform: //:default_host_platform
[401 / 995] Compiling src/main/cpp/blaze.cc; 4s local

�[1AIn file included from third_party/ijar/zlib_client.cc:20:
./third_party/ijar/zlib_client.h:64:46: error: 'numeric_limits' is not a member of 'std'
   64 |   static const size_t MAX_BUFFER_SIZE = std::numeric_limits<int32_t>::max();
      |                                              ^~~~~~~~~~~~~~
./third_party/ijar/zlib_client.h:64:68: error: expected primary-expression before '>' token
   64 |   static const size_t MAX_BUFFER_SIZE = std::numeric_limits<int32_t>::max();
      |                                                                    ^
./third_party/ijar/zlib_client.h:64:71: error: '::max' has not been declared; did you mean 'std::max'?
   64 |   static const size_t MAX_BUFFER_SIZE = std::numeric_limits<int32_t>::max();
      |                                                                       ^~~
      |                                                                       std::max
In file included from /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/algorithm:62,
                 from third_party/ijar/zlib_client.cc:16:
/usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/stl_algo.h:3467:5: note: 'std::max' declared here
 3467 |     max(initializer_list<_Tp> __l, _Compare __comp)
      |     ^~~

https://download.copr.fedorainfracloud.org/results/vbatts/bazel/fedora-34-x86_64/02215183-bazel4/build.log.gz

@vbatts vbatts mentioned this issue May 29, 2021
Closed
@davido
Copy link
Contributor

davido commented May 29, 2021

@vbatts This is a known issue with gcc 11: gRPC and absl_cpp must be bumped.

@vbatts
Copy link

vbatts commented May 29, 2021 via email

This was referenced Jun 7, 2021
Closed
Closed
@veprbl veprbl mentioned this issue Aug 19, 2021
Closed
@philwo philwo unpinned this issue Sep 15, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@philwo philwo

Labels
P1 I'll work on this now. (Assignee required) release team-OSS Issues for the Bazel OSS team: installation, release processBazel packaging, website type: process
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests