From 274b12c07a95feadf004e8f05eb212aa0e9e9c0f Mon Sep 17 00:00:00 2001 From: Blake Li Date: Tue, 10 Jan 2023 14:05:44 -0800 Subject: [PATCH 01/14] Update DEVELOPMENT.md for local development. --- DEVELOPMENT.md | 60 +++++++++++++++++++++++++++++++++++--------------- 1 file changed, 42 insertions(+), 18 deletions(-) diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index e7c508066b..6b709017e3 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -160,7 +160,7 @@ $ cd showcase $ mvn compile -P update ``` -## Running the Plugin +## Running the Plugin under googleapis with local gapic-generator-java See also above section "Showcase Integration Testing". @@ -171,28 +171,52 @@ To generate a production GAPIC API: 2. Point to local gapic-generator-java Normally, googleapis's build pulls in googleapis/gapic-generator-java from the - Internet: - - ``` - # Java microgenerator. - … - _gapic_generator_java_version = "2.1.0" + Internet, we need to build a local SNAPSHOT jar first and point googleapis to the SNAPSHOT jar and local copy of the repo. - http_archive( - name = "gapic_generator_java", - … - urls = ["https://github.com/googleapis/gapic-generator-java/archive/v%s.zip" % _gapic_generator_java_version], - ) + Before: + ``` + _gapic_generator_java_version = "2.13.0" + + maven_install( + artifacts = [ + "com.google.api:gapic-generator-java:" + _gapic_generator_java_version, + ], + #Update this False for local development + fail_on_missing_checksum = True, + repositories = [ + "m2Local", + "https://repo.maven.apache.org/maven2/", + ] + ) + + http_archive( + name = "gapic_generator_java", + strip_prefix = "gapic-generator-java-%s" % _gapic_generator_java_version, + urls = ["https://github.com/googleapis/gapic-generator-java/archive/v%s.zip" % _gapic_generator_java_version], + ) ``` - By replacing this portion using the built-in local_repository rule, you can make - it refer to your local development repo: + After ``` - local_repository( - name = "gapic_generator_java", - path = "/home//gapic-generator-java", - ) + _gapic_generator_java_version = "2.13.1-SNAPSHOT" + + maven_install( + artifacts = [ + "com.google.api:gapic-generator-java:" + _gapic_generator_java_version, + ], + #Update this False for local development + fail_on_missing_checksum = False, + repositories = [ + "m2Local", + "https://repo.maven.apache.org/maven2/", + ] + ) + + local_repository( + name = "gapic_generator_java", + path = "/absolute/path/to/your/local/gapic-generator-java", + ) ``` 3. Build the new target. From 995342c50dc6de012a779f66892ab79357b7b3d4 Mon Sep 17 00:00:00 2001 From: Lawrence Qiu Date: Wed, 11 Jan 2023 11:22:34 -0500 Subject: [PATCH 02/14] docs: Update section with cleaning local m2 --- DEVELOPMENT.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 6b709017e3..73c5383894 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -230,6 +230,10 @@ To generate a production GAPIC API: ``` bazel build //google/cloud/speech/v2:google-cloud-speech-v2-java ``` - - - + + Note: If you are running into bazel build issues, you can try to remove gapic-generator-java cached in your local m2 + Try running this command: + ``` + rm -rf ~/.m2/repository/com/google/api/ + ``` + and then rebuild gapic-generator-java (`mvn clean install`). \ No newline at end of file From bbf502d37ab960bb7accac39276bd706e9892766 Mon Sep 17 00:00:00 2001 From: Lawrence Qiu Date: Wed, 11 Jan 2023 13:14:15 -0500 Subject: [PATCH 03/14] Update DEVELOPMENT.md Co-authored-by: Tomo Suzuki --- DEVELOPMENT.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 73c5383894..4489a95ce3 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -231,7 +231,7 @@ To generate a production GAPIC API: bazel build //google/cloud/speech/v2:google-cloud-speech-v2-java ``` - Note: If you are running into bazel build issues, you can try to remove gapic-generator-java cached in your local m2 + Note: If you are running into bazel build issues, you can try removing gapic-generator-java cached in your local Maven repository. Try running this command: ``` rm -rf ~/.m2/repository/com/google/api/ From 1dc5c1ad5a1ab3d1826b4a7103f4496fe8ab01d6 Mon Sep 17 00:00:00 2001 From: Lawrence Qiu Date: Wed, 11 Jan 2023 13:14:31 -0500 Subject: [PATCH 04/14] Update DEVELOPMENT.md Co-authored-by: Tomo Suzuki --- DEVELOPMENT.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 4489a95ce3..765c26491c 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -201,7 +201,7 @@ To generate a production GAPIC API: ``` _gapic_generator_java_version = "2.13.1-SNAPSHOT" - maven_install( + maven_install( artifacts = [ "com.google.api:gapic-generator-java:" + _gapic_generator_java_version, ], From 9b2bae6efd55e8be2ef9522ad7e507dbed7dc676 Mon Sep 17 00:00:00 2001 From: Lawrence Qiu Date: Wed, 11 Jan 2023 13:30:15 -0500 Subject: [PATCH 05/14] Update DEVELOPMENT.md Co-authored-by: Burke Davison <40617934+burkedavison@users.noreply.github.com> --- DEVELOPMENT.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 765c26491c..843cef68c4 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -168,7 +168,7 @@ To generate a production GAPIC API: 1. Clone [googleapis](https://github.com/googleapis/googleapis). -2. Point to local gapic-generator-java +2. Modify `googleapis/WORKSPACE` to point to local gapic-generator-java Normally, googleapis's build pulls in googleapis/gapic-generator-java from the Internet, we need to build a local SNAPSHOT jar first and point googleapis to the SNAPSHOT jar and local copy of the repo. From 04a9c3fb31273b48b31a6aa8f5d32f320c2e6022 Mon Sep 17 00:00:00 2001 From: Lawrence Qiu Date: Tue, 17 Jan 2023 13:44:03 -0500 Subject: [PATCH 06/14] docs: Update with a FAQ section --- DEVELOPMENT.md | 56 +++++++++++++++++++++++++++++++++----------------- 1 file changed, 37 insertions(+), 19 deletions(-) diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 843cef68c4..851eef1606 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -41,7 +41,7 @@ mvn test ``` -- Run a single or multiple unit tests: +- Run a single or multiple unit tests: ```sh mvn test -Dtest=JavaCodeGeneratorTest @@ -51,30 +51,30 @@ - Update all unit test golden files: - ```sh - mvn test -DupdateUnitGoldens - ``` + ```sh + mvn test -DupdateUnitGoldens + ``` - Update a single unit test golden file, for example `JavaCodeGeneratorTest.java`: - ```sh - mvn test -DupdateUnitGoldens -Dtest=JavaCodeGeneratorTest - ``` + ```sh + mvn test -DupdateUnitGoldens -Dtest=JavaCodeGeneratorTest + ``` - Run a single integration test for API like `Redis`, it generates Java source code using the Java microgenerator and compares them with the goldens files in `test/integration/goldens/redis`. - ```sh - bazel test //test/integration:redis - ``` + ```sh + bazel test //test/integration:redis + ``` - Update integration test golden files, for example `Redis`. This clobbers all the files in `test/integration/goldens/redis`. - ```sh - bazel run //test/integration:update_redis - ``` + ```sh + bazel run //test/integration:update_redis + ``` ## Showcase Integration Testing @@ -171,8 +171,9 @@ To generate a production GAPIC API: 2. Modify `googleapis/WORKSPACE` to point to local gapic-generator-java Normally, googleapis's build pulls in googleapis/gapic-generator-java from the - Internet, we need to build a local SNAPSHOT jar first and point googleapis to the SNAPSHOT jar and local copy of the repo. - + Internet, we need to build a local SNAPSHOT jar first and point googleapis to the SNAPSHOT jar and local copy of the + repo. + Before: ``` _gapic_generator_java_version = "2.13.0" @@ -230,10 +231,27 @@ To generate a production GAPIC API: ``` bazel build //google/cloud/speech/v2:google-cloud-speech-v2-java ``` - - Note: If you are running into bazel build issues, you can try removing gapic-generator-java cached in your local Maven repository. + + Note: If you are running into bazel build issues, you can try to remove gapic-generator-java cached in your local m2 Try running this command: ``` - rm -rf ~/.m2/repository/com/google/api/ + rm -rf ~/.m2/repository/com/google/api/ ``` - and then rebuild gapic-generator-java (`mvn clean install`). \ No newline at end of file + and then rebuild gapic-generator-java (`mvn clean install`). + +## FAQ + +### Error in workspace: workspace() got unexpected keyword argument 'managed_directories' + +Full Error: + +``` +ERROR: Traceback (most recent call last): + File "/home/alicejli/googleapis/WORKSPACE", line 1, column 10, in + workspace( +Error in workspace: workspace() got unexpected keyword argument 'managed_directories' +ERROR: Error computing the main repository mapping: Encountered error while reading extension file 'tools/build_defs/repo/http.bzl': no such package '@bazel_tools//tools/build_defs/repo': error loading package 'external': Could not load //external package +``` + +You may be using the latest version of bazel which this project does not support yet. Try installing bazelisk to force +bazel to use the version specified in `.bazeliskrc` \ No newline at end of file From 8357c3698a68ba9d414d18746db0fcad32259a11 Mon Sep 17 00:00:00 2001 From: Lawrence Qiu Date: Wed, 18 Jan 2023 11:20:42 -0500 Subject: [PATCH 07/14] chore: Update with note about pre-commit hooks --- gapic-generator-java/DEVELOPMENT.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/gapic-generator-java/DEVELOPMENT.md b/gapic-generator-java/DEVELOPMENT.md index 5eb2740fa0..3b0b64a666 100644 --- a/gapic-generator-java/DEVELOPMENT.md +++ b/gapic-generator-java/DEVELOPMENT.md @@ -4,13 +4,19 @@ 1. Clone this repo. -2. Copy the Git pre-commit hooks. This will automatically check the build, run +2. (OPTIONAL) Copy the Git pre-commit hooks. This will automatically check the build, run tests, and perform linting before each commit. (Symlinks don't seem to work, but if you find a way, please add it here!) ```sh cp .githooks/pre-commit .git/hooks/pre-commit ``` + + ### Note: You may see this error with the pre-commits due to the monorepo migration: + ``` + [ERROR] No plugin found for prefix 'fmt' in the current project and in the plugin groups [org.apache.maven.plugins, org.codehaus.mojo] available from the repositories [local (/home/alicejli/.m2/repository), central (https://repo.maven.apache.org/maven2)] -> [Help 1] + ``` + Remove the pre-commit hooks. Tracking the issue in https://github.com/googleapis/gapic-generator-java/issues/1253 ## Code Formatting From 083e1e5d62f9ab437f5516dd2b9e82e810cda5a2 Mon Sep 17 00:00:00 2001 From: Lawrence Qiu Date: Wed, 18 Jan 2023 11:23:38 -0500 Subject: [PATCH 08/14] chore: Update the main development.md --- DEVELOPMENT.md | 18 ++++++++++++------ gapic-generator-java/DEVELOPMENT.md | 18 ++++++------------ 2 files changed, 18 insertions(+), 18 deletions(-) diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 851eef1606..2800973766 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -4,13 +4,19 @@ 1. Clone this repo. -2. Copy the Git pre-commit hooks. This will automatically check the build, run - tests, and perform linting before each commit. (Symlinks don't seem to work, - but if you find a way, please add it here!) +2. (OPTIONAL) Copy the Git pre-commit hooks. This will automatically check the build, run + tests, and perform linting before each commit. (Symlinks don't seem to work, + but if you find a way, please add it here!) - ```sh - cp .githooks/pre-commit .git/hooks/pre-commit - ``` + ```sh + cp .githooks/pre-commit .git/hooks/pre-commit + ``` + + ### Note: You may see this error with the pre-commits due to the monorepo migration: + ``` + [ERROR] No plugin found for prefix 'fmt' in the current project and in the plugin groups [org.apache.maven.plugins, org.codehaus.mojo] available from the repositories [local (/home/alicejli/.m2/repository), central (https://repo.maven.apache.org/maven2)] -> [Help 1] + ``` + Remove the pre-commit hooks. Tracking the issue in https://github.com/googleapis/gapic-generator-java/issues/1253 ## Code Formatting diff --git a/gapic-generator-java/DEVELOPMENT.md b/gapic-generator-java/DEVELOPMENT.md index 3b0b64a666..7fe37586be 100644 --- a/gapic-generator-java/DEVELOPMENT.md +++ b/gapic-generator-java/DEVELOPMENT.md @@ -4,19 +4,13 @@ 1. Clone this repo. -2. (OPTIONAL) Copy the Git pre-commit hooks. This will automatically check the build, run - tests, and perform linting before each commit. (Symlinks don't seem to work, - but if you find a way, please add it here!) +2. Copy the Git pre-commit hooks. This will automatically check the build, run + tests, and perform linting before each commit. (Symlinks don't seem to work, + but if you find a way, please add it here!) - ```sh - cp .githooks/pre-commit .git/hooks/pre-commit - ``` - - ### Note: You may see this error with the pre-commits due to the monorepo migration: - ``` - [ERROR] No plugin found for prefix 'fmt' in the current project and in the plugin groups [org.apache.maven.plugins, org.codehaus.mojo] available from the repositories [local (/home/alicejli/.m2/repository), central (https://repo.maven.apache.org/maven2)] -> [Help 1] - ``` - Remove the pre-commit hooks. Tracking the issue in https://github.com/googleapis/gapic-generator-java/issues/1253 + ```sh + cp .githooks/pre-commit .git/hooks/pre-commit + ``` ## Code Formatting From 7edade28fcacdf0a2a709f8bf66e9bf36bdad9c2 Mon Sep 17 00:00:00 2001 From: Lawrence Qiu Date: Wed, 18 Jan 2023 11:24:33 -0500 Subject: [PATCH 09/14] chore: Update spacing --- gapic-generator-java/DEVELOPMENT.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/gapic-generator-java/DEVELOPMENT.md b/gapic-generator-java/DEVELOPMENT.md index 7fe37586be..5eb2740fa0 100644 --- a/gapic-generator-java/DEVELOPMENT.md +++ b/gapic-generator-java/DEVELOPMENT.md @@ -4,13 +4,13 @@ 1. Clone this repo. -2. Copy the Git pre-commit hooks. This will automatically check the build, run - tests, and perform linting before each commit. (Symlinks don't seem to work, - but if you find a way, please add it here!) +2. Copy the Git pre-commit hooks. This will automatically check the build, run + tests, and perform linting before each commit. (Symlinks don't seem to work, + but if you find a way, please add it here!) - ```sh - cp .githooks/pre-commit .git/hooks/pre-commit - ``` + ```sh + cp .githooks/pre-commit .git/hooks/pre-commit + ``` ## Code Formatting From 25561a4389462712122a878c35e7dad349044a0c Mon Sep 17 00:00:00 2001 From: Lawrence Qiu Date: Wed, 18 Jan 2023 13:42:24 -0500 Subject: [PATCH 10/14] Update DEVELOPMENT.md Co-authored-by: Tomo Suzuki --- DEVELOPMENT.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 2800973766..97a2e13a1d 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -4,7 +4,7 @@ 1. Clone this repo. -2. (OPTIONAL) Copy the Git pre-commit hooks. This will automatically check the build, run +2. (OPTIONAL) Copy the Git pre-commit hooks. This will automatically check the build, run tests, and perform linting before each commit. (Symlinks don't seem to work, but if you find a way, please add it here!) From 663e5955f841df6a36ad77ead52e8ce76967cfbf Mon Sep 17 00:00:00 2001 From: Lawrence Qiu Date: Wed, 18 Jan 2023 14:12:12 -0500 Subject: [PATCH 11/14] chore: Update docs per comments --- DEVELOPMENT.md | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 97a2e13a1d..b6bf51099b 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -176,13 +176,13 @@ To generate a production GAPIC API: 2. Modify `googleapis/WORKSPACE` to point to local gapic-generator-java - Normally, googleapis's build pulls in googleapis/gapic-generator-java from the - Internet, we need to build a local SNAPSHOT jar first and point googleapis to the SNAPSHOT jar and local copy of the - repo. + Normally, googleapis's build pulls in gapic-generator-java from Maven Central. + For a local run, we first need to build a local SNAPSHOT jar of the generator. Then we point googleapis to + both the local SNAPSHOT jar and the local copy of the generator. Before: ``` - _gapic_generator_java_version = "2.13.0" + _gapic_generator_java_version = "2.13.0" maven_install( artifacts = [ @@ -206,11 +206,11 @@ To generate a production GAPIC API: After ``` - _gapic_generator_java_version = "2.13.1-SNAPSHOT" + _gapic_generator_java_version = "2.13.1-SNAPSHOT" - maven_install( + maven_install( artifacts = [ - "com.google.api:gapic-generator-java:" + _gapic_generator_java_version, + "com.google.api:gapic-generator-java:" + _gapic_generator_java_version, ], #Update this False for local development fail_on_missing_checksum = False, @@ -220,11 +220,13 @@ To generate a production GAPIC API: ] ) - local_repository( - name = "gapic_generator_java", - path = "/absolute/path/to/your/local/gapic-generator-java", - ) + local_repository( + name = "gapic_generator_java", + path = "/absolute/path/to/your/local/gapic-generator-java", + ) ``` + + Note: At the time of writing, the gapic-generator version was `2.13.0`. Update the version to the latest version in the pom.xml 3. Build the new target. From 1490865632fd8a4b1d71b9969baa9690de71a8d0 Mon Sep 17 00:00:00 2001 From: Blake Li Date: Tue, 31 Jan 2023 11:26:01 -0500 Subject: [PATCH 12/14] Apply suggestions from code review Co-authored-by: Burke Davison <40617934+burkedavison@users.noreply.github.com> --- DEVELOPMENT.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index b6bf51099b..8744c9fb41 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -14,7 +14,10 @@ ### Note: You may see this error with the pre-commits due to the monorepo migration: ``` - [ERROR] No plugin found for prefix 'fmt' in the current project and in the plugin groups [org.apache.maven.plugins, org.codehaus.mojo] available from the repositories [local (/home/alicejli/.m2/repository), central (https://repo.maven.apache.org/maven2)] -> [Help 1] +[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.10.1:compile (default-compile) on project gapic-generator-java: Compilation failure: Compilation failure: +[ERROR] gapic-generator-java/gapic-generator-java/src/main/java/com/google/api/generator/gapic/composer/rest/ServiceClientTestClassComposer.java:[19,43] package com.google.api.gax.httpjson.testing does not exist +[ERROR] gapic-generator-java/gapic-generator-java/src/main/java/com/google/api/generator/gapic/composer/rest/ServiceClientTestClassComposer.java:[24,38] package com.google.api.gax.rpc.testing does not exist +[ERROR] gapic-generator-java/gapic-generator-java/src/main/java/com/google/api/generator/gapic/composer/grpc/ServiceClientTestClassComposer.java:[18,39] package com.google.api.gax.grpc.testing does not exist ``` Remove the pre-commit hooks. Tracking the issue in https://github.com/googleapis/gapic-generator-java/issues/1253 From 5f702025b7b3e933fa6be196391ba874d9dc4390 Mon Sep 17 00:00:00 2001 From: Blake Li Date: Tue, 31 Jan 2023 17:44:48 -0500 Subject: [PATCH 13/14] Update DEVELOPMENT.md --- DEVELOPMENT.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 6a25ac0dac..06984dc65f 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -42,17 +42,17 @@ - Run all unit and integration tests. ```sh - mvn test # unit tests + mvn install # unit tests, maven test wouldn't work in root folder because gapic-generator-java is dependant on test jars of gax-java bazel test //... # integration tests ``` - Run all unit tests. ```sh - mvn test + mvn install ``` - -- Run a single or multiple unit tests: +- For running unit tests in `gapic-generator-java` submodule, first build all modules with `mvn install -DskipTests`, then `cd` into `gapic-generator-java` submodule for the following commands: + - Run a single or multiple unit tests: ```sh mvn test -Dtest=JavaCodeGeneratorTest @@ -60,13 +60,13 @@ mvn test "-Dtest=Basic*, !%regex[.*.Unstable.*], !%regex[.*.MyTest.class#one.*|two.*], %regex[#fast.*|slow.*]" ``` -- Update all unit test golden files: + - Update all unit test golden files: ```sh mvn test -DupdateUnitGoldens ``` -- Update a single unit test golden file, for example `JavaCodeGeneratorTest.java`: + - Update a single unit test golden file, for example `JavaCodeGeneratorTest.java`: ```sh mvn test -DupdateUnitGoldens -Dtest=JavaCodeGeneratorTest @@ -185,7 +185,7 @@ To generate a production GAPIC API: For a local run, we first need to build a local SNAPSHOT jar of the generator. Then we point googleapis to both the local SNAPSHOT jar and the local copy of the generator. - Before: + Replace the following section in googleapis ``` _gapic_generator_java_version = "2.13.0" @@ -208,7 +208,7 @@ To generate a production GAPIC API: ) ``` - After + to ``` _gapic_generator_java_version = "2.13.1-SNAPSHOT" From 6d6cc528a045ccd48cd1d17192061ef56b48f7bb Mon Sep 17 00:00:00 2001 From: Blake Li Date: Tue, 31 Jan 2023 18:01:10 -0500 Subject: [PATCH 14/14] Update DEVELOPMENT.md --- DEVELOPMENT.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 06984dc65f..f813d7de4a 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -173,7 +173,7 @@ $ mvn compile -P update ## Running the Plugin under googleapis with local gapic-generator-java -See also above section "Showcase Integration Testing". +For running the Plugin with showcase protos and local gapic-generator-java, see above section "Showcase Integration Testing". To generate a production GAPIC API: