chore: transfer java-related synthtool code into the library_generati…

…on image (#3011)

This PR moves `java.py` and the minimum required dependencies from
[synthtool](https://github.com/googleapis/synthtool/tree/master/synthtool).

It contains the changes of @JoeWang1127's
googleapis/synthtool@696c4bf

### Details of code transfer

Some files have been simplified and others simply verbatim-copied. Here
is a detailed list of changes (the paths are relative to
`library_generation/owlbot`:
* __`synthtool/__init__.py`__: No changes made. This is the module
initialization configuration to allow statements such as `import
synthtool as s` combined with `s.copy`, `s.replace`, etc. Without this,
we would have to use `import synthtool.transforms as s`.
* __`synthtool/_tracked_paths.py`__: No changes made. It allows to use
relative paths internally when working with library files.
* __`synthtool/gcp/common.py`__: Simplified to use java-only functions.
This file originally had a set of functions to support postprocessing of
languages written in multiple languages. The most important function is
`common_templates`, which renders the templates for the library (e.g.
workflow files, kokoro files). Note that `common_templates` was modified
in order to require and only allow specifying the path to the templates
via the `SYNTHTOOL_TEMPLATES` env var, as opposed to its original
support of 3 separate ways, including cloning synthtool and reading the
templates from there, because this is now an internal detail of how the
`library_generation` image will work.
* __`synthtool/gcp/samples.py`__: No changes made. This is a helper to
obtain path and metadata about the generated samples of a library. It is
then used [when rendering the
README](https://github.com/googleapis/sdk-platform-java/blob/554329eae8d6970223dc64a920f7714246afe7a1/library_generation/owlbot/templates/java_library/README.md?plain=1#L141-L150).
* __`synthtool/gcp/snippets.py`__: No changes made. Similar to
`samples.py` in the way it's
[used](https://github.com/googleapis/sdk-platform-java/blob/554329eae8d6970223dc64a920f7714246afe7a1/library_generation/owlbot/templates/java_library/README.md?plain=1#L68-L80)
to render a library's README
* __`synthool/languages/java.py`__: Small modifications around the fact
that we dropped several files (e.g. use `gcp.common.CommonTemplates`
instead of `gcp.CommonTemplates` to save us from an extra
`gcp/__init__.py`).
* __`synthtool/sources/templates.py`__: No changes made. Internally used
by `common_templates`. Contains the underlying usage of jinja2 to render
the templates.
* __`synthtool/transforms.py`__: No changes made. Contains a few
functions that are commonly used by `owlbot.py` files
([example](https://github.com/googleapis/java-bigtable/blob/45732201880a13eeced3d0332bd172aae0f73dbe/owlbot.py#L50))

### Changes in Dockerfile
We will not clone `synthtool` anymore. We will instead install it as a
separate package whose source code is within sdk-platform-java.

.cloudbuild/library_generation/library_generation.Dockerfile

@@ -17,7 +17,6 @@ FROM gcr.io/cloud-devrel-public-resources/python
 
 SHELL [ "/bin/bash", "-c" ]
 
-ARG SYNTHTOOL_COMMITTISH=696c4bff721f5541cd75fdc97d413f8f39d2a2c1
 ARG OWLBOT_CLI_COMMITTISH=ac84fa5c423a0069bbce3d2d869c9730c8fdf550
 ARG PROTOC_VERSION=25.3
 ENV HOME=/home
@@ -44,19 +43,11 @@ RUN ln -s $(which python3.11) /usr/local/bin/python
 RUN ln -s $(which python3.11) /usr/local/bin/python3
 RUN python -m pip install --upgrade pip
 
-# install scripts as a python package
+# install main scripts as a python package
 WORKDIR /src
 RUN python -m pip install -r requirements.txt
 RUN python -m pip install .
 
-# install synthtool
-WORKDIR /tools
-RUN git clone https://github.com/googleapis/synthtool
-WORKDIR /tools/synthtool
-RUN git checkout "${SYNTHTOOL_COMMITTISH}"
-RUN python3 -m pip install --no-deps -e .
-RUN python3 -m pip install -r requirements.in
-
 # Install nvm with node and npm
 ENV NODE_VERSION 20.12.0
 WORKDIR /home

.github/snippet-bot.yml

@@ -5,3 +5,4 @@ ignoreFiles:
   - test/**
   - showcase/**
   - library_generation/owlbot/templates/java_library/samples/install-without-bom/pom.xml
+  - library_generation/owlbot/synthtool/gcp/snippets.py

library_generation/owlbot/synthtool/__init__.py

@@ -0,0 +1,34 @@
+# Copyright 2024 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Synthtool synthesizes libraries from disparate sources."""
+
+import sys
+
+from synthtool.transforms import (
+    move,
+    replace,
+    get_staging_dirs,
+    remove_staging_dirs,
+)
+
+copy = move
+
+__all__ = [
+    "copy",
+    "move",
+    "replace",
+    "get_staging_dirs",
+    "remove_staging_dirs",
+]

library_generation/owlbot/synthtool/_tracked_paths.py

@@ -0,0 +1,39 @@
+# Copyright 2024 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Tracked paths.
+
+This is a bit of a hack (imported from original synthtool library).
+"""
+
+import pathlib
+
+
+_tracked_paths = []
+
+
+def add(path):
+    _tracked_paths.append(pathlib.Path(path))
+    # Reverse sort the list, so that the deepest paths get matched first.
+    _tracked_paths.sort(key=lambda s: -len(str(s)))
+
+
+def relativize(path):
+    path = pathlib.Path(path)
+    for tracked_path in _tracked_paths:
+        try:
+            return path.relative_to(tracked_path)
+        except ValueError:
+            pass
+    raise ValueError(f"The root for {path} is not tracked.")

library_generation/owlbot/synthtool/gcp/common.py

@@ -0,0 +1,153 @@
+# Copyright 2024 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import json
+import os
+import re
+import sys
+import shutil
+import yaml
+from pathlib import Path
+from typing import Dict, List, Optional
+import jinja2
+import logging
+
+from synthtool import _tracked_paths
+from synthtool.sources import templates
+
+logger = logging.getLogger()
+logger.setLevel(logging.DEBUG)
+
+
+DEFAULT_TEMPLATES_PATH = "synthtool/gcp/templates"
+LOCAL_TEMPLATES: Optional[str] = os.environ.get("SYNTHTOOL_TEMPLATES")
+
+# Originally brought from gcp/partials.py.
+# These are the default locations to look up
+_DEFAULT_PARTIAL_FILES = [
+    ".readme-partials.yml",
+    ".readme-partials.yaml",
+    ".integration-partials.yaml",
+]
+
+
+class CommonTemplates:
+    def __init__(self, template_path: Optional[Path] = None):
+        if LOCAL_TEMPLATES is None:
+            logger.error("env var SYNTHTOOL_TEMPLATES must be set")
+            sys.exit(1)
+        logger.debug(f"Using local templates at {LOCAL_TEMPLATES}")
+        self._template_root = Path(LOCAL_TEMPLATES)
+        self._templates = templates.Templates(self._template_root)
+        self.excludes = []  # type: List[str]
+
+    def _generic_library(self, directory: str, relative_dir=None, **kwargs) -> Path:
+        # load common repo meta information (metadata that's not language specific).
+        if "metadata" in kwargs:
+            self._load_generic_metadata(kwargs["metadata"], relative_dir=relative_dir)
+            # if no samples were found, don't attempt to render a
+            # samples/README.md.
+            if "samples" not in kwargs["metadata"] or not kwargs["metadata"]["samples"]:
+                self.excludes.append("samples/README.md")
+
+        t = templates.TemplateGroup(self._template_root / directory, self.excludes)
+
+        result = t.render(**kwargs)
+        _tracked_paths.add(result)
+
+        return result
+
+    def java_library(self, **kwargs) -> Path:
+        # kwargs["metadata"] is required to load values from .repo-metadata.json
+        if "metadata" not in kwargs:
+            kwargs["metadata"] = {}
+        return self._generic_library("java_library", **kwargs)
+
+    def render(self, template_name: str, **kwargs) -> Path:
+        template = self._templates.render(template_name, **kwargs)
+        _tracked_paths.add(template)
+        return template
+
+    def _load_generic_metadata(self, metadata: Dict, relative_dir=None):
+        """
+        loads additional meta information from .repo-metadata.json.
+        """
+        metadata["partials"] = load_partials()
+
+        # Loads repo metadata information from the default location if it
+        # hasn't already been set. Some callers may have already loaded repo
+        # metadata, so we don't need to do it again or overwrite it. Also, only
+        # set the "repo" key.
+        if "repo" not in metadata:
+            metadata["repo"] = _load_repo_metadata(relative_dir=relative_dir)
+
+
+def _load_repo_metadata(
+    relative_dir=None, metadata_file: str = "./.repo-metadata.json"
+) -> Dict:
+    """Parse a metadata JSON file into a Dict.
+
+    Currently, the defined fields are:
+    * `name` - The service's API name
+    * `name_pretty` - The service's API title. This will be used for generating titles on READMEs
+    * `product_documentation` - The product documentation on cloud.google.com
+    * `client_documentation` - The client library reference documentation
+    * `issue_tracker` - The public issue tracker for the product
+    * `release_level` - The release level of the client library. One of: alpha, beta,
+      ga, deprecated, preview, stable
+    * `language` - The repo language. One of dotnet, go, java, nodejs, php, python, ruby
+    * `repo` - The GitHub repo in the format {owner}/{repo}
+    * `distribution_name` - The language-idiomatic package/distribution name
+    * `api_id` - The API ID associated with the service. Fully qualified identifier use to
+      enable a service in the cloud platform (e.g. monitoring.googleapis.com)
+    * `requires_billing` - Whether or not the API requires billing to be configured on the
+      customer's acocunt
+
+    Args:
+        metadata_file (str, optional): Path to the metadata json file
+
+    Returns:
+        A dictionary of metadata. This may not necessarily include all the defined fields above.
+    """
+    if relative_dir is not None:
+        if os.path.exists(Path(relative_dir, metadata_file).resolve()):
+            with open(Path(relative_dir, metadata_file).resolve()) as f:
+                return json.load(f)
+    elif os.path.exists(metadata_file):
+        with open(metadata_file) as f:
+            return json.load(f)
+    return {}
+
+
+def load_partials(files: List[str] = []) -> Dict:
+    """
+    hand-crafted artisanal markdown can be provided in a .readme-partials.yml.
+    The following fields are currently supported:
+
+    body: custom body to include in the usage section of the document.
+    samples_body: an optional body to place below the table of contents
+        in samples/README.md.
+    introduction: a more thorough introduction than metadata["description"].
+    title: provide markdown to use as a custom title.
+    deprecation_warning: a warning to indicate that the library has been
+        deprecated and a pointer to an alternate option
+    """
+    result: Dict[str, Dict] = {}
+    cwd_path = Path(os.getcwd())
+    for file in files + _DEFAULT_PARTIAL_FILES:
+        partials_file = cwd_path / file
+        if os.path.exists(partials_file):
+            with open(partials_file) as f:
+                result.update(yaml.load(f, Loader=yaml.SafeLoader))
+    return result

library_generation/owlbot/synthtool/gcp/samples.py

@@ -0,0 +1,91 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import glob
+import logging
+import re
+import os
+import yaml
+from typing import List, Dict
+
+logger = logging.getLogger()
+logger.setLevel(logging.DEBUG)
+
+
+def _read_sample_metadata_comment(sample_file: str) -> Dict:
+    """Additional meta-information can be provided through embedded comments:
+
+    // sample-metadata:
+    //   title: ACL (Access Control)
+    //   description: Demonstrates setting access control rules.
+    //   usage: node iam.js --help
+    """
+    sample_metadata = {}  # type: Dict[str, str]
+    with open(sample_file) as f:
+        contents = f.read()
+        match = re.search(
+            r"(?P<metadata>// *sample-metadata:([^\n]+|\n//)+)", contents, re.DOTALL
+        )
+        if match:
+            # the metadata yaml is stored in a comments, remove the
+            # prefix so that we can parse the yaml contained.
+            sample_metadata_string = re.sub(r"((#|//) ?)", "", match.group("metadata"))
+            try:
+                sample_metadata = yaml.load(
+                    sample_metadata_string, Loader=yaml.SafeLoader
+                )["sample-metadata"]
+            except yaml.scanner.ScannerError:
+                # warn and continue on bad metadata
+                logger.warning(f"bad metadata detected in {sample_file}")
+    return sample_metadata
+
+
+def _sample_metadata(file: str) -> Dict[str, str]:
+    metadata = {
+        "title": _decamelize(os.path.splitext(os.path.basename(file))[0]),
+        "file": file,
+    }
+    return {**metadata, **_read_sample_metadata_comment(file)}
+
+
+def all_samples(sample_globs: List[str]) -> List[Dict[str, str]]:
+    """Walks samples directory and builds up samples data-structure
+
+    Args:
+        sample_globs: (List[str]): List of path globs to search for samples
+
+    Returns:
+        A list of sample metadata in the format:
+        {
+            "title": "Requester Pays",
+            "file": "samples/requesterPays.js"
+        }
+        The file path is the relative path from the repository root.
+    """
+    files = []
+    for sample_glob in sample_globs:
+        for file in glob.glob(sample_glob, recursive=True):
+            files.append(file)
+    return [_sample_metadata(file) for file in sorted(files)]
+
+
+def _decamelize(value: str):
+    """Parser to convert fooBar.js to Foo Bar."""
+    if not value:
+        return ""
+    str_decamelize = re.sub("^.", value[0].upper(), value)  # apple -> Apple.
+    str_decamelize = re.sub(
+        "([A-Z]+)([A-Z])([a-z0-9])", r"\1 \2\3", str_decamelize
+    )  # ACLBatman -> ACL Batman.
+    return re.sub("([a-z0-9])([A-Z])", r"\1 \2", str_decamelize)  # FooBar -> Foo Bar.