[BEAM-11227] Preparing vendored gRPC upgrade from 1.26 to 1.35 #14028

suztomo · 2021-02-20T04:56:13Z

This PR prepares the vendored gRPC version upgrade from 1.26 to 1.35

Todo

See how this new version breaks compilation. It compiled how about tests?
- It compiled without changing code manually. Tests have passed except Java PostCommit that has been failing since Feb 27th. (-> sent a mail)
- With this hack, the pull request checks can run against the vendored gRPC 1.36.0 without publishing it to Maven Central.
Have a release manager to publish the vendored gRPC 1.36.0 (http://s.apache.org/beam-release-vendored-artifacts) with votes in dev@beam.apache.
Fix anything that fails with the vendored gRPC 1.36.0
Remove the directory for the old gRPC 1.26.0
Merge this PR.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang	Dataflow	Samza	Twister2
Go	---	---	---
Java
Python		---	---
XLang		---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---		---	---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

netty-tcnative-boringssl-static version from https://github.com/grpc/grpc-java/blob/master/SECURITY.md#netty

codecov · 2021-02-20T05:28:04Z

Codecov Report

Merging #14028 (4f5c13c) into master (08a9d54) will increase coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #14028      +/-   ##
==========================================
+ Coverage   83.15%   83.16%   +0.01%     
==========================================
  Files         469      938     +469     
  Lines       58972   117948   +58976     
==========================================
+ Hits        49037    98092   +49055     
- Misses       9935    19856    +9921

Impacted Files	Coverage Δ
...m/runners/portability/flink_uber_jar_job_server.py
...uild/srcs/sdks/python/apache_beam/coders/coders.py
...srcs/sdks/python/apache_beam/dataframe/doctests.py
...e_beam/portability/api/beam_runner_api_pb2_urns.py
...srcs/sdks/python/apache_beam/io/gcp/dicomclient.py
...examples/snippets/transforms/aggregation/latest.py
...thon/apache_beam/runners/portability/job_server.py
...ython/apache_beam/io/aws/clients/s3/fake_client.py
...thon/apache_beam/testing/metric_result_matchers.py
...hon/apache_beam/portability/api/schema_pb2_grpc.py
... and 1397 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 15ef51f...dc4ac01. Read the comment docs.

suztomo · 2021-02-20T13:25:56Z

Run Java Precommit

suztomo · 2021-02-21T03:23:10Z

Run Python_PVR_Flink PreCommit

suztomo · 2021-02-23T17:00:24Z

Discussion in https://issues.apache.org/jira/browse/BEAM-11227 : does this really mark the vendored gRPC artifact out of the CVE?

kennknowles · 2021-02-23T18:49:29Z

vendor/README.md

+$ ../../gradlew shadowJar
+$ find . -name '*.jar'
+./build/libs/beam-vendor-grpc-1_26_0-0.1.jar
+$ mvn install:install-file \


Isn't the idea of the command to test the vendored artifact built from the unsubmitted and unpublished code?

Yes, that's the purpose of my change. The original command does not seem to install the vendors gRPC artifact locally.

suztomo · 2021-03-02T16:28:39Z

I'll try to apply the change to use the new vendored gRPC version.

suztomo · 2021-03-03T19:51:30Z

buildSrc/src/main/groovy/org/apache/beam/gradle/GrpcVendoring_1_36_0.groovy

+
+  static def guava_version = "30.1-jre"
+  // protobuf version from https://search.maven.org/artifact/io.grpc/grpc-protobuf/1.36.0/jar
+  static def protobuf_version = "3.12.0"


According to the Linkage Checker (output), this version 3.12.0 suffers from protocolbuffers/protobuf#7827.

java.nio.ByteBuffer's method position(int) is expected to return java.nio.ByteBuffer but instead returns java.nio.Buffer;

suztomo · 2021-03-03T20:26:34Z

buildSrc/src/main/groovy/org/apache/beam/gradle/GrpcVendoring_1_36_0.groovy

+class GrpcVendoring_1_36_0 {
+
+  static def guava_version = "30.1-jre"
+  static def protobuf_version = "3.15.3"


This is the latest available version of protobuf-java. We need a version higher than 3.14.0 otherwise it hits Java 8 incompatible problem in protobuf-java 3.13.0. (comment in the ticket)

suztomo · 2021-03-03T20:51:41Z

GitHub Actions failed:

> Task :sdks:java:core:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
compiler message file broken: key=compiler.misc.msg.bug arguments=11.0.10, {1}, {2}, {3}, {4}, {5}, {6}, {7}
java.lang.NoSuchMethodError: 'void com.sun.tools.javac.util.Log.error(com.sun.tools.javac.util.JCDiagnostic$DiagnosticPosition, java.lang.String, java.lang.Object[])'
	at com.google.errorprone.ErrorProneError.logFatalError(ErrorProneError.java:55)
	at com.google.errorprone.ErrorProneAnalyzer.finished(ErrorProneAnalyzer.java:155)
	at jdk.compiler/com.sun.tools.javac.api.MultiTaskListener.finished(MultiTaskListener.java:132)
	at jdk.compiler/com.sun.tools.javac.main.JavaCompiler.flow(JavaCompiler.java:1418)
	at jdk.compiler/com.sun.tools.javac.main.JavaCompiler.flow(JavaCompiler.java:1365)
	at jdk.compiler/com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:960)
	at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.lambda$doCall$0(JavacTaskImpl.java:104)
	at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.handleExceptions(JavacTaskImpl.java:147)
	at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.doCall(JavacTaskImpl.java:100)
	at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.call(JavacTaskImpl.java:94)

This seems the problem in the build (before this PR).

suztomo · 2021-03-03T21:04:32Z

"Java Tests / Java Unit Tests (macos-latest)" passed.

suztomo · 2021-03-03T22:07:17Z

.test-infra/jenkins/PrecommitJobBuilder.groovy

@@ -116,7 +116,7 @@ class PrecommitJobBuilder {
      steps {
        gradle {
          rootBuildScriptDir(commonJobProperties.checkoutDir)
-          tasks(gradleTask)
+          tasks(':installVendoredGrpc', gradleTask)


Jenkins does not respect this until we tell Jenkins to do so. (That might affect other PRs?)

suztomo · 2021-03-03T22:07:21Z

.test-infra/jenkins/job_PostCommit_Java.groovy

@@ -40,7 +40,7 @@ PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java', 'Run Java PostCommit'
      steps {
        gradle {
          rootBuildScriptDir(commonJobProperties.checkoutDir)
-          tasks(':javaPostCommit')
+          tasks(':installVendoredGrpc', ':javaPostCommit')


Jenkins does not respect this until we tell Jenkins to do so. (That might affect other PRs?)

suztomo · 2021-03-03T22:48:23Z

build.gradle.kts

+// Because :model:job-management:runtimeClasspath requires the vendored gRPC at configuration phase
+// (before execution phase), we cannot rely on task dependencies.
+if (!project.hasProperty("installVendoredGrpcFlag")) {
+  project.exec {
+    commandLine = listOf(
+            "./gradlew", ":installVendoredGrpc", "-PinstallVendoredGrpcFlag", "--info", "--stacktrace"
+    )
+  }
+}


It worked! (I'll remove this piece of code before merging this to master)

suztomo · 2021-03-13T05:12:23Z

Run Java Precommit

suztomo · 2021-03-15T14:39:49Z

Run SQL Postcommit

suztomo · 2021-03-15T14:43:04Z

Run Go PreCommit

suztomo · 2021-03-15T14:43:12Z

Run Java_Examples_Dataflow_Java11 PreCommit

suztomo · 2021-03-15T14:43:16Z

Run PythonDocker PreCommit

suztomo · 2021-03-15T14:43:23Z

Run Website_Stage_GCS PreCommit

suztomo · 2021-03-15T15:05:11Z

Run GoPortable PreCommit

suztomo · 2021-03-15T15:49:54Z

Run Python PreCommit

suztomo · 2021-03-15T15:50:11Z

"SQL Post Commit Tests" succeeded.

suztomo · 2021-03-15T16:03:57Z

Run Java Postcommit

suztomo · 2021-03-15T16:04:34Z

Run Java HadoopFormatIO Performance Test

suztomo · 2021-03-15T16:04:40Z

Run BigQueryIO Streaming Performance Test Java

suztomo · 2021-03-15T16:04:46Z

Run Dataflow ValidatesRunner

suztomo · 2021-03-15T16:04:51Z

Run Spark ValidatesRunner

kennknowles · 2021-03-15T17:44:49Z

Just to understand more about this, my understanding of the steps would be:

Check in the new vendored gRPC
Release the new vendored gRPC
Check in the migration to the new vendored gRPC
Delete the old vendored gRPC

Does that sound right?

suztomo · 2021-03-15T19:06:17Z

@kennknowles Yes, that matches my understanding. Although some tests failed due to timeout, they passed at dd2df58 . I just removed unnecessary build.gradle for testing (The BEAM-11227_parallel fails the GitHub checks because the vendored artifact is not available in Maven Central yet). My branch BEAM-11227_parallel is reach for you to publish the vendored artifact to staging location.

http://s.apache.org/beam-release-vendored-artifacts says:

./gradlew -p vendor/grpc-1_36_0 -PisRelease -PvendoredDependenciesOnly publish

This uploads the artifacts to the staging Apache Nexus repository. Would you perform the staging release?

kennknowles · 2021-03-15T22:01:11Z

We should commit the code for the thing before I publish it, though.

suztomo · 2021-03-15T22:24:09Z

Sure, let me prepare this PR so that this can be merged to master. (I will need to remove the references to the new vendored classes)

kennknowles · 2021-03-15T22:26:45Z

This PR is good for testing but you could just cherrypick the commit with the new vendoring to a separate PR. Up to you!

suztomo · 2021-03-15T23:08:17Z

@kennknowles I also think another PR is better. I just created #14242

kennknowles · 2021-03-22T17:38:22Z

OK we should be able to adjust this to just the changes to move the SDK to the new version.

suztomo · 2021-03-22T17:45:17Z

@kennknowles I created another PR #14295 (I was reluctant to manage conflicts)

suztomo · 2021-03-31T16:11:54Z

The other PR 14295 has been merged.

suztomo added 3 commits February 19, 2021 18:03

grpc 1.26 and 1.35 in parallel

dcdbe16

Enriching README with install intruction

867c5cb

netty versions to match gRPC 1.35

21e6bcf

netty-tcnative-boringssl-static version from https://github.com/grpc/grpc-java/blob/master/SECURITY.md#netty

kennknowles self-requested a review February 22, 2021 23:51

kennknowles reviewed Feb 23, 2021

View reviewed changes

suztomo added 6 commits March 2, 2021 11:29

Merge remote-tracking branch 'origin/master' into BEAM-11227_parallel

74ec8e0

vendored gRPC 1.36.0

4cf462a

vendored gRPC 1.36 part of the build

5ee37dd

Updating readme

f89cd60

replaced the usage of 1.26.0 to 1.36.0

a92d463

replaced org.apache.beam.vendor.grpc.v1p26p0 to v1p36p0

3f08e25

suztomo commented Mar 3, 2021

View reviewed changes

protobuf-version from latest

b8ea472

suztomo commented Mar 3, 2021

View reviewed changes

installVendoredGrpc task

c1144fc

suztomo force-pushed the BEAM-11227_parallel branch from d56edc0 to c1144fc Compare March 3, 2021 21:48

suztomo commented Mar 3, 2021

View reviewed changes

suztomo added 3 commits March 3, 2021 17:30

a hack to install vendored gRPC

5eac9b7

removing new flag for mvn

ce1e425

mvn full path and info options

4cdba24

suztomo commented Mar 3, 2021

View reviewed changes

suztomo added 2 commits March 3, 2021 18:18

/home/jenkins/tools/maven/apache-maven-3.5.4/bin as PATH

5fa1bc1

PATH environment before invoking Gradle

0b1ae99

suztomo added 3 commits March 15, 2021 09:38

Merge remote-tracking branch 'origin/master' into BEAM-11227_parallel

89cb6fc

removed local test code

c69916e

Adding the test code to confirm SQL Postcommit

4f5c13c

removing test hack that runs checks with unpublished artifacts

dc4ac01

suztomo mentioned this pull request Mar 15, 2021

[BEAM-11227] Preparing vendored gRPC 1.36.0 #14242

Merged

4 tasks

suztomo closed this Mar 31, 2021

suztomo deleted the BEAM-11227_parallel branch March 31, 2021 16:13

[BEAM-11227] Preparing vendored gRPC upgrade from 1.26 to 1.35 #14028

[BEAM-11227] Preparing vendored gRPC upgrade from 1.26 to 1.35 #14028

Conversation

suztomo commented Feb 20, 2021 • edited Loading

Todo

Post-Commit Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

GitHub Actions Tests Status (on master branch)

codecov bot commented Feb 20, 2021 • edited Loading

Codecov Report

suztomo commented Feb 20, 2021

suztomo commented Feb 21, 2021

suztomo commented Feb 23, 2021

kennknowles Feb 23, 2021

Choose a reason for hiding this comment

suztomo Feb 23, 2021

Choose a reason for hiding this comment

suztomo commented Mar 2, 2021

suztomo Mar 3, 2021 • edited Loading

Choose a reason for hiding this comment

suztomo Mar 3, 2021

Choose a reason for hiding this comment

suztomo commented Mar 3, 2021 • edited Loading

suztomo commented Mar 3, 2021

suztomo Mar 3, 2021

Choose a reason for hiding this comment

suztomo Mar 3, 2021

Choose a reason for hiding this comment

suztomo Mar 3, 2021 • edited Loading

Choose a reason for hiding this comment

suztomo commented Mar 13, 2021

suztomo commented Mar 15, 2021

suztomo commented Mar 15, 2021

suztomo commented Mar 15, 2021

suztomo commented Mar 15, 2021

suztomo commented Mar 15, 2021

suztomo commented Mar 15, 2021

suztomo commented Mar 15, 2021

suztomo commented Mar 15, 2021

suztomo commented Mar 15, 2021

suztomo commented Mar 15, 2021

suztomo commented Mar 15, 2021

suztomo commented Mar 15, 2021

suztomo commented Mar 15, 2021

kennknowles commented Mar 15, 2021

suztomo commented Mar 15, 2021 • edited Loading

kennknowles commented Mar 15, 2021

suztomo commented Mar 15, 2021

kennknowles commented Mar 15, 2021

suztomo commented Mar 15, 2021

kennknowles commented Mar 22, 2021

suztomo commented Mar 22, 2021

suztomo commented Mar 31, 2021

suztomo commented Feb 20, 2021 •

edited

Loading

codecov bot commented Feb 20, 2021 •

edited

Loading

suztomo Mar 3, 2021 •

edited

Loading

suztomo commented Mar 3, 2021 •

edited

Loading

suztomo Mar 3, 2021 •

edited

Loading

suztomo commented Mar 15, 2021 •

edited

Loading