Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: generate selected libraries #2598

Merged
merged 10 commits into from
Mar 26, 2024
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 80 additions & 0 deletions library_generation/cli/generate_repo.py
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this is supposed to be the endtrypoint in the future? If it is, can we rename this file to avoid confusion? Otherwise I think keeping the cli together with the logic is fine.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this is supposed to be the entrypoint in the future?

Yes.

We can change the name, then before combining generate_pr_description.py, we will have some like this in google-cloud-java

entrypoint.py ... // generate libraries
generate_pr_description.py ... // generate PR description

Do you think this is fine?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, I think that's better than having two generate_repo.py. In general, if you have a design in your mind but will not implement it immediately, it's better to define the interfaces first so that others are easy to follow.

Copy link
Collaborator Author

@JoeWang1127 JoeWang1127 Mar 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I think this (having two generate_repo.py is confusing) makes sense.

I decided to remove cli/generate_repo.py and keep the cli part in generate_repo.py (as before) and modified description accordingly.

I'll introduce entrypoint.py in follow PR.

Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
# Copyright 2024 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import click
from library_generation.generate_repo import generate_from_yaml


@click.group(invoke_without_command=False)
@click.pass_context
@click.version_option(message="%(version)s")
def main(ctx):
pass


@main.command()
@click.option(
"--generation-config-yaml",
required=True,
type=str,
help="""
Path to generation_config.yaml that contains the metadata about
library generation
""",
)
@click.option(
"--target-library-names",
required=False,
default=None,
type=str,
help="""
A list of libraries will be generated.

If specified, only the `library` whose library_name is in
target-library-names will be generated.
If not specified, all libraries in the configuration yaml will be generated.

The input string will be parsed to a list of string with comma as the
separator.

For example, apigeeconnect,alloydb-connectors will be parsed as a
list of two strings, apigeeconnect and alloydb-connectors.
""",
)
@click.option(
"--repository-path",
required=False,
default=".",
type=str,
help="""
If specified, the generated files will be sent to this location.
If not specified, the repository will be generated to the current working
directory.
""",
)
def generate(
generation_config_yaml: str,
target_library_names: str,
repository_path: str,
):
generate_from_yaml(
generation_config_yaml=generation_config_yaml,
repository_path=repository_path,
target_library_names=target_library_names.split(",")
if target_library_names is not None
else target_library_names,
)


if __name__ == "__main__":
main()
88 changes: 25 additions & 63 deletions library_generation/generate_repo.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,87 +13,33 @@
# See the License for the specific language governing permissions and
# limitations under the License.

import click
import library_generation.utilities as util
import os
from library_generation.generate_composed_library import generate_composed_library
from library_generation.model.generation_config import GenerationConfig
from library_generation.model.generation_config import from_yaml
from library_generation.model.library_config import LibraryConfig
from library_generation.utils.monorepo_postprocessor import monorepo_postprocessing


@click.group(invoke_without_command=False)
@click.pass_context
@click.version_option(message="%(version)s")
def main(ctx):
pass


@main.command()
@click.option(
"--generation-config-yaml",
required=True,
type=str,
help="""
Path to generation_config.yaml that contains the metadata about
library generation
""",
)
@click.option(
"--target-library-api-shortname",
required=False,
type=str,
help="""
If specified, only the `library` whose api_shortname equals to
target-library-api-shortname will be generated.
If not specified, all libraries in the configuration yaml will be generated.
""",
)
@click.option(
"--repository-path",
required=False,
default=".",
type=str,
help="""
If specified, the generated files will be sent to this location.
If not specified, the repository will be generated to the current working
directory.
""",
)
def generate(
generation_config_yaml: str,
target_library_api_shortname: str,
repository_path: str,
):
generate_from_yaml(
generation_config_yaml=generation_config_yaml,
repository_path=repository_path,
target_library_api_shortname=target_library_api_shortname,
)


def generate_from_yaml(
generation_config_yaml: str,
repository_path: str,
target_library_api_shortname: str = None,
target_library_names: list[str] = None,
) -> None:
"""
Parses a config yaml and generates libraries via
generate_composed_library.py
"""
# convert paths to absolute paths so they can be correctly referenced in
# convert paths to absolute paths, so they can be correctly referenced in
# downstream scripts
generation_config_yaml = os.path.abspath(generation_config_yaml)
repository_path = os.path.abspath(repository_path)

config = from_yaml(generation_config_yaml)
target_libraries = config.libraries
if target_library_api_shortname is not None:
target_libraries = [
library
for library in config.libraries
if library.api_shortname == target_library_api_shortname
]

target_libraries = __get_target_libraries(
config=config, target_library_names=target_library_names
)
repo_config = util.prepare_repo(
gen_config=config, library_config=target_libraries, repo_path=repository_path
)
Expand All @@ -118,5 +64,21 @@ def generate_from_yaml(
)


if __name__ == "__main__":
main()
def __get_target_libraries(
config: GenerationConfig, target_library_names: list[str] = None
) -> list[LibraryConfig]:
"""
Returns LibraryConfig objects whose library_name is in target_library_names.

:param config: a GenerationConfig object.
:param target_library_names: library_name of target libraries.
:return:
"""
if target_library_names is None:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if the list is empty? Does it mean we generate nothing or generate everything?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if the list is empty

We generate nothing.

The rational is if we pass a list, then the libraries will be selected from the list. Therefore, an empty list means no generation.

If we don't pass a list, the default value is None and we generate everything.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SG. Can you mention the behavior in the comment above? I wonder how it is going to be implemented in generate_repo.py later, maybe just no-op if the list is empty?

Copy link
Collaborator Author

@JoeWang1127 JoeWang1127 Mar 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adde comments.

I wonder how it is going to be implemented in generate_repo.py later, maybe just no-op if the list is empty?

It will not call generate_composed_library.py but it will do a repo level post processing, which is no-op if no library is changed.

return config.libraries
target_libraries = set(target_library_names)
return [
library
for library in config.libraries
if library.get_library_name() in target_libraries
]
6 changes: 1 addition & 5 deletions library_generation/utilities.py
Original file line number Diff line number Diff line change
Expand Up @@ -131,11 +131,7 @@ def prepare_repo(
os.makedirs(output_folder, exist_ok=True)
libraries = {}
for library in library_config:
library_name = (
f"{language}-{library.library_name}"
if library.library_name
else f"{language}-{library.api_shortname}"
)
library_name = f"{language}-{library.get_library_name()}"
library_path = (
f"{repo_path}/{library_name}" if gen_config.is_monorepo else f"{repo_path}"
)
Expand Down
Loading