diff --git a/octavia-cli/README.md b/octavia-cli/README.md index f7f1c53424f8..bc9fd98e571c 100644 --- a/octavia-cli/README.md +++ b/octavia-cli/README.md @@ -5,7 +5,6 @@ The project is in **alpha** version. Readers can refer to our [opened GitHub issues](https://github.com/airbytehq/airbyte/issues?q=is%3Aopen+is%3Aissue+label%3Aarea%2Foctavia-cli) to check the ongoing work on this project. - ## What is `octavia` CLI? Octavia CLI is a tool to manage Airbyte configurations in YAML. @@ -44,7 +43,7 @@ Feel free to share your use cases with the community in [#octavia-cli](https://a ### 1. Generate local YAML files for sources or destinations -1. Retrieve the *definition id* of the connector you want to use using `octavia list command`. +1. Retrieve the _definition id_ of the connector you want to use using `octavia list command`. 2. Generate YAML configuration running `octavia generate source ` or `octavia generate destination `. ### 2. Edit your local YAML configurations @@ -67,7 +66,7 @@ Feel free to share your use cases with the community in [#octavia-cli](https://a ### 6. Update your configurations -Changes in your local configurations can be propagated to your Airbyte instance using `octavia apply`. You will be prompted for validation of changes. You can bypass the validation step using the `--force` flag. +Changes in your local configurations can be propagated to your Airbyte instance using `octavia apply`. You will be prompted for validation of changes. You can bypass the validation step using the `--force` flag. ## Secret management @@ -79,7 +78,7 @@ configuration: password: ${MY_PASSWORD} ``` -If you have set a `MY_PASSWORD` environment variable, `octavia apply` will load its value into the `password` field. +If you have set a `MY_PASSWORD` environment variable, `octavia apply` will load its value into the `password` field. ## Install @@ -138,27 +137,32 @@ docker-compose run octavia-cli ` ### `octavia` command flags -| **Flag** | **Description** | **Env Variable** | **Default** | -|--------------------------------------------|-----------------------------------------------------------------------------------|------------------------------|--------------------------------------------------------| -| `--airbyte-url` | Airbyte instance URL. | `AIRBYTE_URL` | `http://localhost:8000` | -| `--workspace-id` | Airbyte workspace id. | `AIRBYTE_WORKSPACE_ID` | The first workspace id found on your Airbyte instance. | -| `--enable-telemetry/--disable-telemetry` | Enable or disable the sending of telemetry data. | `OCTAVIA_ENABLE_TELEMETRY` | True | -| `--api-http-header` | HTTP Header value pairs passed while calling Airbyte's API | not supported. | None | None | -| `--api-http-headers-file-path` | Path to the YAML file that contains custom HTTP Headers to send to Airbyte's API. | None | None | +| **Flag** | **Description** | **Env Variable** | **Default** | +| ---------------------------------------- | --------------------------------------------------------------------------------- | -------------------------- | ------------------------------------------------------ | ---- | +| `--airbyte-url` | Airbyte instance URL. | `AIRBYTE_URL` | `http://localhost:8000` | +| `--workspace-id` | Airbyte workspace id. | `AIRBYTE_WORKSPACE_ID` | The first workspace id found on your Airbyte instance. | +| `--enable-telemetry/--disable-telemetry` | Enable or disable the sending of telemetry data. | `OCTAVIA_ENABLE_TELEMETRY` | True | +| `--api-http-header` | HTTP Header value pairs passed while calling Airbyte's API | not supported. | None | None | +| `--api-http-headers-file-path` | Path to the YAML file that contains custom HTTP Headers to send to Airbyte's API. | None | None | #### Using custom HTTP headers -You can set custom HTTP headers to send to Airbyte's API with options: + +You can set custom HTTP headers to send to Airbyte's API with options: + ```bash octavia --api-http-header Header-Name Header-Value --api-http-header Header-Name-2 Header-Value-2 list connectors sources ``` You can also use a custom YAML file (one is already created on init in `api_http_headers.yaml`) to declare the HTTP headers to send to the API: + ```yaml headers: Authorization: Basic foobar== User-Agent: octavia-cli/0.0.0 ``` + Environment variable expansion is available in this Yaml file + ```yaml headers: Authorization: Bearer ${MY_API_TOKEN} @@ -168,18 +172,21 @@ headers: ### `octavia` subcommands -| **Command** | **Usage** | -|-----------------------------------------|-------------------------------------------------------------------------------------| -| **`octavia init`** | Initialize required directories for the project. | -| **`octavia list connectors sources`** | List all sources connectors available on the remote Airbyte instance. | -| **`octavia list connectors destination`** | List all destinations connectors available on the remote Airbyte instance. | -| **`octavia list workspace sources`** | List existing sources in current the Airbyte workspace. | -| **`octavia list workspace destinations`** | List existing destinations in the current Airbyte workspace. | -| **`octavia list workspace connections`** | List existing connections in the current Airbyte workspace. | -| **`octavia generate source`** | Generate a local YAML configuration for a new source. | -| **`octavia generate destination`** | Generate a local YAML configuration for a new destination. | -| **`octavia generate connection`** | Generate a local YAML configuration for a new connection. | -| **`octavia apply`** | Create or update Airbyte remote resources according to local YAML configurations. | +| **Command** | **Usage** | +| ----------------------------------------- | ---------------------------------------------------------------------------------------- | +| **`octavia init`** | Initialize required directories for the project. | +| **`octavia list connectors sources`** | List all sources connectors available on the remote Airbyte instance. | +| **`octavia list connectors destination`** | List all destinations connectors available on the remote Airbyte instance. | +| **`octavia list workspace sources`** | List existing sources in current the Airbyte workspace. | +| **`octavia list workspace destinations`** | List existing destinations in the current Airbyte workspace. | +| **`octavia list workspace connections`** | List existing connections in the current Airbyte workspace. | +| **`octavia get source`** | Get the JSON representation of an existing source in current the Airbyte workspace. | +| **`octavia get destination`** | Get the JSON representation of an existing destination in the current Airbyte workspace. | +| **`octavia get connection`** | Get the JSON representation of an existing connection in the current Airbyte workspace. | +| **`octavia generate source`** | Generate a local YAML configuration for a new source. | +| **`octavia generate destination`** | Generate a local YAML configuration for a new destination. | +| **`octavia generate connection`** | Generate a local YAML configuration for a new connection. | +| **`octavia apply`** | Create or update Airbyte remote resources according to local YAML configurations. | #### `octavia init` @@ -264,13 +271,227 @@ NAME CONNECTION ID STATUS SOURCE ID weather_to_pg a4491317-153e-436f-b646-0b39338f9aab active c4aa8550-2122-4a33-9a21-adbfaa638544 c0c977c2-48e7-46fe-9f57-576285c26d42 ``` +#### `octavia get source or ` + +Get an existing source in current the Airbyte workspace. You can use a source ID or name. + +| **Argument** | **Description** | +| --------------| -----------------| +| `SOURCE_ID` | The source id. | +| `SOURCE_NAME` | The source name. | + +**Examples**: + +```bash +$ octavia get source c0c977c2-48e7-46fe-9f57-576285c26d42 +{'connection_configuration': {'key': '**********', + 'start_date': '2010-01-01T00:00:00.000Z', + 'token': '**********'}, + 'name': 'Pokemon', + 'source_definition_id': 'b08e4776-d1de-4e80-ab5c-1e51dad934a2', + 'source_id': 'c0c977c2-48e7-46fe-9f57-576285c26d42', + 'source_name': 'My Poke', + 'workspace_id': 'c4aa8550-2122-4a33-9a21-adbfaa638544'} +``` + +```bash +$ octavia get source "My Poke" +{'connection_configuration': {'key': '**********', + 'start_date': '2010-01-01T00:00:00.000Z', + 'token': '**********'}, + 'name': 'Pokemon', + 'source_definition_id': 'b08e4776-d1de-4e80-ab5c-1e51dad934a2', + 'source_id': 'c0c977c2-48e7-46fe-9f57-576285c26d42', + 'source_name': 'My Poke', + 'workspace_id': 'c4aa8550-2122-4a33-9a21-adbfaa638544'} +``` + +#### `octavia get destination or ` + +Get an existing destination in current the Airbyte workspace. You can use a destination ID or name. + +| **Argument** | **Description** | +| ------------------ | ----------------------| +| `DESTINATION_ID` | The destination id. | +| `DESTINATION_NAME` | The destination name. | + +**Examples**: + +```bash +$ octavia get destination c0c977c2-48e7-46fe-9f57-576285c26d42 +{ + "destinationDefinitionId": "c0c977c2-48e7-46fe-9f57-576285c26d42", + "destinationId": "18102e7c-5160-4000-841b-15e8ec48c301", + "workspaceId": "18102e7c-5160-4000-883a-30bc7cd65601", + "connectionConfiguration": { + "user": "charles" + }, + "name": "pg", + "destinationName": "Postgres" +} +``` + +```bash +$ octavia get destination pg +{ + "destinationDefinitionId": "18102e7c-5160-4000-821f-4d7cfdf87201", + "destinationId": "18102e7c-5160-4000-841b-15e8ec48c301", + "workspaceId": "18102e7c-5160-4000-883a-30bc7cd65601", + "connectionConfiguration": { + "user": "charles" + }, + "name": "string", + "destinationName": "string" +} +``` + +#### `octavia get connection or ` + +Get an existing connection in current the Airbyte workspace. You can use a connection ID or name. + +| **Argument** | **Description** | +| ------------------ | ----------------------| +| `CONNECTION_ID` | The connection id. | +| `CONNECTION_NAME` | The connection name. | + +**Example**: + +```bash +$ octavia get connection c0c977c2-48e7-46fe-9f57-576285c26d42 +{ + "connectionId": "c0c977c2-48e7-46fe-9f57-576285c26d42", + "name": "Poke To PG", + "namespaceDefinition": "source", + "namespaceFormat": "${SOURCE_NAMESPACE}", + "prefix": "string", + "sourceId": "18102e7c-5340-4000-8eaa-4a86f844b101", + "destinationId": "18102e7c-5340-4000-8e58-6bed49c24b01", + "operationIds": [ + "18102e7c-5340-4000-8ef0-f35c05a49a01" + ], + "syncCatalog": { + "streams": [ + { + "stream": { + "name": "string", + "jsonSchema": {}, + "supportedSyncModes": [ + "full_refresh" + ], + "sourceDefinedCursor": false, + "defaultCursorField": [ + "string" + ], + "sourceDefinedPrimaryKey": [ + [ + "string" + ] + ], + "namespace": "string" + }, + "config": { + "syncMode": "full_refresh", + "cursorField": [ + "string" + ], + "destinationSyncMode": "append", + "primaryKey": [ + [ + "string" + ] + ], + "aliasName": "string", + "selected": false + } + } + ] + }, + "schedule": { + "units": 0, + "timeUnit": "minutes" + }, + "status": "active", + "resourceRequirements": { + "cpu_request": "string", + "cpu_limit": "string", + "memory_request": "string", + "memory_limit": "string" + }, + "sourceCatalogId": "18102e7c-5340-4000-85f3-204ab7715801" +} +``` + +```bash +$ octavia get connection "Poke To PG" +{ + "connectionId": "c0c977c2-48e7-46fe-9f57-576285c26d42", + "name": "Poke To PG", + "namespaceDefinition": "source", + "namespaceFormat": "${SOURCE_NAMESPACE}", + "prefix": "string", + "sourceId": "18102e7c-5340-4000-8eaa-4a86f844b101", + "destinationId": "18102e7c-5340-4000-8e58-6bed49c24b01", + "operationIds": [ + "18102e7c-5340-4000-8ef0-f35c05a49a01" + ], + "syncCatalog": { + "streams": [ + { + "stream": { + "name": "string", + "jsonSchema": {}, + "supportedSyncModes": [ + "full_refresh" + ], + "sourceDefinedCursor": false, + "defaultCursorField": [ + "string" + ], + "sourceDefinedPrimaryKey": [ + [ + "string" + ] + ], + "namespace": "string" + }, + "config": { + "syncMode": "full_refresh", + "cursorField": [ + "string" + ], + "destinationSyncMode": "append", + "primaryKey": [ + [ + "string" + ] + ], + "aliasName": "string", + "selected": false + } + } + ] + }, + "schedule": { + "units": 0, + "timeUnit": "minutes" + }, + "status": "active", + "resourceRequirements": { + "cpu_request": "string", + "cpu_limit": "string", + "memory_request": "string", + "memory_limit": "string" + }, + "sourceCatalogId": "18102e7c-5340-4000-85f3-204ab7715801" +} +``` #### `octavia generate source ` Generate a YAML configuration for a source. The YAML file will be stored at `./sources//configuration.yaml`. -| **Argument** | **Description** | -|-----------------|-----------------------------------------------------------------------------------------------| +| **Argument** | **Description** | +| --------------- | --------------------------------------------------------------------------------------------- | | `DEFINITION_ID` | The source connector definition id. Can be retrieved using `octavia list connectors sources`. | | `SOURCE_NAME` | The name you want to give to this source in Airbyte. | @@ -287,7 +508,7 @@ Generate a YAML configuration for a destination. The YAML file will be stored at `./destinations//configuration.yaml`. | **Argument** | **Description** | -|--------------------|---------------------------------------------------------------------------------------------------------| +| ------------------ | ------------------------------------------------------------------------------------------------------- | | `DEFINITION_ID` | The destination connector definition id. Can be retrieved using `octavia list connectors destinations`. | | `DESTINATION_NAME` | The name you want to give to this destination in Airbyte. | @@ -303,13 +524,13 @@ $ octavia generate destination 25c5221d-dce2-4163-ade9-739ef790f503 my_db Generate a YAML configuration for a connection. The YAML file will be stored at `./connections//configuration.yaml`. -| **Option** | **Required** | **Description** | -|-------------------|--------------|--------------------------------------------------------------------------------------------| -| `--source` | Yes | Path to the YAML configuration file of the source you want to create a connection from. | -| `--destination` | Yes | Path to the YAML configuration file of the destination you want to create a connection to. | +| **Option** | **Required** | **Description** | +| --------------- | ------------ | ------------------------------------------------------------------------------------------ | +| `--source` | Yes | Path to the YAML configuration file of the source you want to create a connection from. | +| `--destination` | Yes | Path to the YAML configuration file of the destination you want to create a connection to. | | **Argument** | **Description** | -|-------------------|----------------------------------------------------------| +| ----------------- | -------------------------------------------------------- | | `CONNECTION_NAME` | The name you want to give to this connection in Airbyte. | **Example**: @@ -326,10 +547,10 @@ If the resource was not found on your Airbyte instance, **apply** will **create* If the resource was found on your Airbyte instance, **apply** will prompt you for validation of the changes and will run an **update** of your resource. Please note that if a secret field was updated on your configuration, **apply** will run this change without prompt. -| **Option** | **Required** | **Description** | -|-----------------|--------------|--------------------------------------------------------------------------------------------| -| `--file` | No | Path to the YAML configuration files you want to create or update. | -| `--force` | No | Run update without prompting for changes validation. | +| **Option** | **Required** | **Description** | +| ---------- | ------------ | ------------------------------------------------------------------ | +| `--file` | No | Path to the YAML configuration files you want to create or update. | +| `--force` | No | Run update without prompting for changes validation. | **Example**: @@ -373,23 +594,26 @@ $ octavia apply 7. Make sure the build passes (step 0) before opening a PR. ## Telemetry + This CLI has some telemetry tooling to send Airbyte some data about the usage of this tool. We will use this data to improve the CLI and measure its adoption. The telemetry sends data about: -* Which command was run (not the arguments or options used). -* Success or failure of the command run and the error type (not the error payload). -* The current Airbyte workspace id if the user has not set the *anonymous data collection* on their Airbyte instance. + +- Which command was run (not the arguments or options used). +- Success or failure of the command run and the error type (not the error payload). +- The current Airbyte workspace id if the user has not set the _anonymous data collection_ on their Airbyte instance. You can disable telemetry by setting the `OCTAVIA_ENABLE_TELEMETRY` environment variable to `False` or using the `--disable-telemetry` flag. ## Changelog -| Version | Date | Description | PR | -|----------|------------|----------------------------------------------------|----------------------------------------------------------| -| 0.39.19 | 2022-06-16 | Allow connection management on multiple workspaces | [#12727](https://github.com/airbytehq/airbyte/pull/12727)| -| 0.39.19 | 2022-06-15 | Allow users to set custom HTTP headers | [#12893](https://github.com/airbytehq/airbyte/pull/12893) | -| 0.39.14 | 2022-05-12 | Enable normalization on connection | [#12727](https://github.com/airbytehq/airbyte/pull/12727)| -| 0.37.0 | 2022-05-05 | Use snake case in connection fields | [#12133](https://github.com/airbytehq/airbyte/pull/12133)| -| 0.35.68 | 2022-04-15 | Improve telemetry | [#12072](https://github.com/airbytehq/airbyte/issues/11896)| -| 0.35.68 | 2022-04-12 | Add telemetry | [#11896](https://github.com/airbytehq/airbyte/issues/11896)| -| 0.35.61 | 2022-04-07 | Alpha release | [EPIC](https://github.com/airbytehq/airbyte/issues/10704)| +| Version | Date | Description | PR | +| ------- | ---------- | ------------------------------------------------------------ | ----------------------------------------------------------- | +| 0.39.27 | 2022-06-24 | Create get command to retrieve resources JSON representation | [#13254](https://github.com/airbytehq/airbyte/pull/13254) | +| 0.39.19 | 2022-06-16 | Allow connection management on multiple workspaces | [#13070](https://github.com/airbytehq/airbyte/pull/12727) | +| 0.39.19 | 2022-06-15 | Allow users to set custom HTTP headers | [#12893](https://github.com/airbytehq/airbyte/pull/12893) | +| 0.39.14 | 2022-05-12 | Enable normalization on connection | [#12727](https://github.com/airbytehq/airbyte/pull/12727) | +| 0.37.0 | 2022-05-05 | Use snake case in connection fields | [#12133](https://github.com/airbytehq/airbyte/pull/12133) | +| 0.35.68 | 2022-04-15 | Improve telemetry | [#12072](https://github.com/airbytehq/airbyte/issues/11896) | +| 0.35.68 | 2022-04-12 | Add telemetry | [#11896](https://github.com/airbytehq/airbyte/issues/11896) | +| 0.35.61 | 2022-04-07 | Alpha release | [EPIC](https://github.com/airbytehq/airbyte/issues/10704) | diff --git a/octavia-cli/octavia_cli/apply/resources.py b/octavia-cli/octavia_cli/apply/resources.py index 2124f9e760a1..b846fa675a98 100644 --- a/octavia-cli/octavia_cli/apply/resources.py +++ b/octavia-cli/octavia_cli/apply/resources.py @@ -57,10 +57,6 @@ from .yaml_loaders import EnvVarLoader -class DuplicateResourceError(click.ClickException): - pass - - class NonExistingResourceError(click.ClickException): pass diff --git a/octavia-cli/octavia_cli/entrypoint.py b/octavia-cli/octavia_cli/entrypoint.py index 6d914bc2b3a7..e42846cb243b 100644 --- a/octavia-cli/octavia_cli/entrypoint.py +++ b/octavia-cli/octavia_cli/entrypoint.py @@ -14,11 +14,18 @@ from .apply import commands as apply_commands from .check_context import check_api_health, check_is_initialized, check_workspace_exists from .generate import commands as generate_commands +from .get import commands as get_commands from .init import commands as init_commands from .list import commands as list_commands from .telemetry import TelemetryClient, build_user_agent -AVAILABLE_COMMANDS: List[click.Command] = [list_commands._list, init_commands.init, generate_commands.generate, apply_commands.apply] +AVAILABLE_COMMANDS: List[click.Command] = [ + list_commands._list, + get_commands.get, + init_commands.init, + generate_commands.generate, + apply_commands.apply, +] def set_context_object( diff --git a/octavia-cli/octavia_cli/generate/templates/source_or_destination.yaml.j2 b/octavia-cli/octavia_cli/generate/templates/source_or_destination.yaml.j2 index 3ea86d4902a1..9f5131e1789b 100644 --- a/octavia-cli/octavia_cli/generate/templates/source_or_destination.yaml.j2 +++ b/octavia-cli/octavia_cli/generate/templates/source_or_destination.yaml.j2 @@ -32,7 +32,7 @@ definition_version: {{ definition.docker_image_tag }} {%- macro render_one_of(field) %} -{{ field.name }}: +{{ field.name }}: {%- for one_of_value in field.one_of_values %} {%- if loop.first %} ## -------- Pick one valid structure among the examples below: -------- @@ -41,17 +41,17 @@ definition_version: {{ definition.docker_image_tag }} ## -------- Another valid structure for {{ field.name }}: -------- {{- render_sub_fields(one_of_value, True)|indent(2, False) }} {%- endif %} -{%- endfor %} +{%- endfor %} {%- endmacro %} {%- macro render_object_field(field) %} -{{ field.name }}: - {{- render_sub_fields(field.object_properties, is_commented=False)|indent(2, False)}} +{{ field.name }}: + {{- render_sub_fields(field.object_properties, is_commented=False)|indent(2, False)}} {%- endmacro %} {%- macro render_array_of_objects(field) %} -{{ field.name }}: - {{- render_array_sub_fields(field.array_items, is_commented=False)|indent(2, False)}} +{{ field.name }}: + {{- render_array_sub_fields(field.array_items, is_commented=False)|indent(2, False)}} {%- endmacro %} {%- macro render_root(root, is_commented) %} diff --git a/octavia-cli/octavia_cli/get/__init__.py b/octavia-cli/octavia_cli/get/__init__.py new file mode 100644 index 000000000000..1100c1c58cf5 --- /dev/null +++ b/octavia-cli/octavia_cli/get/__init__.py @@ -0,0 +1,3 @@ +# +# Copyright (c) 2022 Airbyte, Inc., all rights reserved. +# diff --git a/octavia-cli/octavia_cli/get/commands.py b/octavia-cli/octavia_cli/get/commands.py new file mode 100644 index 000000000000..fea5c6d96377 --- /dev/null +++ b/octavia-cli/octavia_cli/get/commands.py @@ -0,0 +1,108 @@ +# +# Copyright (c) 2022 Airbyte, Inc., all rights reserved. +# + +import uuid +from typing import List, Optional, Tuple, Type, Union + +import airbyte_api_client +import click +from octavia_cli.base_commands import OctaviaCommand + +from .resources import Connection, Destination, Source + +COMMON_HELP_MESSAGE_PREFIX = "Get a JSON representation of a remote" + + +def build_help_message(resource_type: str) -> str: + """Helper function to build help message consistently for all the commands in this module. + + Args: + resource_type (str): source, destination or connection + + Returns: + str: The generated help message. + """ + return f"Get a JSON representation of a remote {resource_type}." + + +def get_resource_id_or_name(resource: str) -> Tuple[Optional[str], Optional[str]]: + """Helper function to detect if the resource argument passed to the CLI is a resource ID or name. + + Args: + resource (str): the resource ID or name passed as an argument to the CLI. + + Returns: + Tuple[Optional[str], Optional[str]]: the resource_id and resource_name, the not detected kind is set to None. + """ + resource_id, resource_name = None, None + try: + uuid.UUID(resource) + resource_id = resource + except ValueError: + resource_name = resource + return resource_id, resource_name + + +def get_json_representation( + api_client: airbyte_api_client.ApiClient, + workspace_id: str, + ResourceCls: Type[Union[Source, Destination, Connection]], + resource_to_get: str, +) -> str: + """Helper function to retrieve a resource json representation and avoid repeating the same logic for Source/Destination and connection. + + + Args: + api_client (airbyte_api_client.ApiClient): The Airbyte API client. + workspace_id (str): Current workspace id. + ResourceCls (Type[Union[Source, Destination, Connection]]): Resource class to use + resource_to_get (str): resource name or id to get JSON representation for. + + Returns: + str: The resource's JSON representation. + """ + resource_id, resource_name = get_resource_id_or_name(resource_to_get) + resource = ResourceCls(api_client, workspace_id, resource_id=resource_id, resource_name=resource_name) + return resource.to_json() + + +@click.group( + "get", + help=f'{build_help_message("source, destination or connection")} ID or name can be used as argument. Example: \'octavia get source "My Pokemon source"\' or \'octavia get source cb5413b2-4159-46a2-910a-dc282a439d2d\'', +) +@click.pass_context +def get(ctx: click.Context): # pragma: no cover + pass + + +@get.command(cls=OctaviaCommand, name="source", help=build_help_message("source")) +@click.argument("resource", type=click.STRING) +@click.pass_context +def source(ctx: click.Context, resource: str): + click.echo(get_json_representation(ctx.obj["API_CLIENT"], ctx.obj["WORKSPACE_ID"], Source, resource)) + + +@get.command(cls=OctaviaCommand, name="destination", help=build_help_message("destination")) +@click.argument("resource", type=click.STRING) +@click.pass_context +def destination(ctx: click.Context, resource: str): + click.echo(get_json_representation(ctx.obj["API_CLIENT"], ctx.obj["WORKSPACE_ID"], Destination, resource)) + + +@get.command(cls=OctaviaCommand, name="connection", help=build_help_message("connection")) +@click.argument("resource", type=click.STRING) +@click.pass_context +def connection(ctx: click.Context, resource: str): + click.echo(get_json_representation(ctx.obj["API_CLIENT"], ctx.obj["WORKSPACE_ID"], Connection, resource)) + + +AVAILABLE_COMMANDS: List[click.Command] = [source, destination, connection] + + +def add_commands_to_list(): + for command in AVAILABLE_COMMANDS: + get.add_command(command) + + +add_commands_to_list() diff --git a/octavia-cli/octavia_cli/get/resources.py b/octavia-cli/octavia_cli/get/resources.py new file mode 100644 index 000000000000..aef89cda00c0 --- /dev/null +++ b/octavia-cli/octavia_cli/get/resources.py @@ -0,0 +1,193 @@ +# +# Copyright (c) 2022 Airbyte, Inc., all rights reserved. +# + +import abc +import json +from typing import Optional, Union + +import airbyte_api_client +import click +from airbyte_api_client.api import destination_api, source_api, web_backend_api +from airbyte_api_client.model.destination_id_request_body import DestinationIdRequestBody +from airbyte_api_client.model.destination_read import DestinationRead +from airbyte_api_client.model.source_id_request_body import SourceIdRequestBody +from airbyte_api_client.model.source_read import SourceRead +from airbyte_api_client.model.web_backend_connection_read import WebBackendConnectionRead +from airbyte_api_client.model.web_backend_connection_request_body import WebBackendConnectionRequestBody +from airbyte_api_client.model.workspace_id_request_body import WorkspaceIdRequestBody + + +class DuplicateResourceError(click.ClickException): + pass + + +class ResourceNotFoundError(click.ClickException): + pass + + +class BaseResource(abc.ABC): + @property + @abc.abstractmethod + def api( + self, + ): # pragma: no cover + pass + + @property + @abc.abstractmethod + def name( + self, + ) -> str: # pragma: no cover + pass + + @property + @abc.abstractmethod + def get_function_name( + self, + ) -> str: # pragma: no cover + pass + + @property + def _get_fn(self): + return getattr(self.api, self.get_function_name) + + @property + @abc.abstractmethod + def get_payload( + self, + ): # pragma: no cover + pass + + @property + @abc.abstractmethod + def list_for_workspace_function_name( + self, + ) -> str: # pragma: no cover + pass + + @property + def _list_for_workspace_fn(self): + return getattr(self.api, self.list_for_workspace_function_name) + + @property + def list_for_workspace_payload( + self, + ): + return WorkspaceIdRequestBody(workspace_id=self.workspace_id) + + def __init__( + self, + api_client: airbyte_api_client.ApiClient, + workspace_id: str, + resource_id: Optional[str] = None, + resource_name: Optional[str] = None, + ): + if resource_id is None and resource_name is None: + raise ValueError("resource_id and resource_name keyword arguments can't be both None.") + if resource_id is not None and resource_name is not None: + raise ValueError("resource_id and resource_name keyword arguments can't be both set.") + self.resource_id = resource_id + self.resource_name = resource_name + self.api_instance = self.api(api_client) + self.workspace_id = workspace_id + + def _find_by_resource_name( + self, + ) -> Union[WebBackendConnectionRead, SourceRead, DestinationRead]: + """Retrieve a remote resource from its name by listing the available resources on the Airbyte instance. + + Raises: + ResourceNotFoundError: Raised if no resource was found with the current resource_name. + DuplicateResourceError: Raised if multiple resources were found with the current resource_name. + + Returns: + Union[WebBackendConnectionRead, SourceRead, DestinationRead]: The remote resource model instance. + """ + + api_response = self._list_for_workspace_fn(self.api_instance, self.list_for_workspace_payload) + matching_resources = [] + for resource in getattr(api_response, f"{self.name}s"): + if resource.name == self.resource_name: + matching_resources.append(resource) + if not matching_resources: + raise ResourceNotFoundError(f"The {self.name} {self.resource_name} was not found in your current Airbyte workspace.") + if len(matching_resources) > 1: + raise DuplicateResourceError( + f"{len(matching_resources)} {self.name}s with the name {self.resource_name} were found in your current Airbyte workspace." + ) + return matching_resources[0] + + def _find_by_resource_id( + self, + ) -> Union[WebBackendConnectionRead, SourceRead, DestinationRead]: + """Retrieve a remote resource from its id by calling the get endpoint of the resource type. + + Returns: + Union[WebBackendConnectionRead, SourceRead, DestinationRead]: The remote resource model instance. + """ + return self._get_fn(self.api_instance, self.get_payload) + + def get_remote_resource(self) -> Union[WebBackendConnectionRead, SourceRead, DestinationRead]: + """Retrieve a remote resource with a resource_name or a resource_id + + Returns: + Union[WebBackendConnectionRead, SourceRead, DestinationRead]: The remote resource model instance. + """ + if self.resource_id is not None: + return self._find_by_resource_id() + else: + return self._find_by_resource_name() + + def to_json(self) -> str: + """Get the JSON representation of the remote resource model instance. + + Returns: + str: The JSON representation of the remote resource model instance. + """ + return json.dumps(self.get_remote_resource().to_dict()) + + +class Source(BaseResource): + name = "source" + api = source_api.SourceApi + get_function_name = "get_source" + list_for_workspace_function_name = "list_sources_for_workspace" + + @property + def get_payload(self) -> Optional[SourceIdRequestBody]: + """Defines the payload to retrieve the remote source according to its resource_id. + Returns: + SourceIdRequestBody: The SourceIdRequestBody payload. + """ + return SourceIdRequestBody(self.resource_id) + + +class Destination(BaseResource): + name = "destination" + api = destination_api.DestinationApi + get_function_name = "get_destination" + list_for_workspace_function_name = "list_destinations_for_workspace" + + @property + def get_payload(self) -> Optional[DestinationIdRequestBody]: + """Defines the payload to retrieve the remote destination according to its resource_id. + Returns: + DestinationIdRequestBody: The DestinationIdRequestBody payload. + """ + return DestinationIdRequestBody(self.resource_id) + + +class Connection(BaseResource): + name = "connection" + api = web_backend_api.WebBackendApi + get_function_name = "web_backend_get_connection" + list_for_workspace_function_name = "web_backend_list_connections_for_workspace" + + @property + def get_payload(self) -> Optional[WebBackendConnectionRequestBody]: + """Defines the payload to retrieve the remote connection according to its resource_id. + Returns: + WebBackendConnectionRequestBody: The WebBackendConnectionRequestBody payload. + """ + return WebBackendConnectionRequestBody(with_refreshed_catalog=False, connection_id=self.resource_id) diff --git a/octavia-cli/unit_tests/test_entrypoint.py b/octavia-cli/unit_tests/test_entrypoint.py index 4b4fa0bbe7e1..0e4a9b7cea33 100644 --- a/octavia-cli/unit_tests/test_entrypoint.py +++ b/octavia-cli/unit_tests/test_entrypoint.py @@ -216,6 +216,7 @@ def test_not_implemented_commands(command): def test_available_commands(): assert entrypoint.AVAILABLE_COMMANDS == [ entrypoint.list_commands._list, + entrypoint.get_commands.get, entrypoint.init_commands.init, entrypoint.generate_commands.generate, entrypoint.apply_commands.apply, diff --git a/octavia-cli/unit_tests/test_get/__init__.py b/octavia-cli/unit_tests/test_get/__init__.py new file mode 100644 index 000000000000..1100c1c58cf5 --- /dev/null +++ b/octavia-cli/unit_tests/test_get/__init__.py @@ -0,0 +1,3 @@ +# +# Copyright (c) 2022 Airbyte, Inc., all rights reserved. +# diff --git a/octavia-cli/unit_tests/test_get/test_commands.py b/octavia-cli/unit_tests/test_get/test_commands.py new file mode 100644 index 000000000000..a380c290689f --- /dev/null +++ b/octavia-cli/unit_tests/test_get/test_commands.py @@ -0,0 +1,102 @@ +# +# Copyright (c) 2022 Airbyte, Inc., all rights reserved. +# + +import pytest +from click.testing import CliRunner +from octavia_cli.get import commands + + +def test_commands_in_get_group(): + get_commands = commands.get.commands.values() + for command in commands.AVAILABLE_COMMANDS: + assert command in get_commands + + +@pytest.fixture +def context_object(mock_api_client, mock_telemetry_client): + return { + "API_CLIENT": mock_api_client, + "WORKSPACE_ID": "my_workspace_id", + "resource_id": "my_resource_id", + "TELEMETRY_CLIENT": mock_telemetry_client, + } + + +def test_available_commands(): + assert commands.AVAILABLE_COMMANDS == [commands.source, commands.destination, commands.connection] + + +def test_build_help_message(): + assert commands.build_help_message("fake_resource_type") == "Get a JSON representation of a remote fake_resource_type." + + +def test_get_resource_id_or_name(): + resource_id, resource_name = commands.get_resource_id_or_name("resource_name") + assert resource_id is None and resource_name == "resource_name" + resource_id, resource_name = commands.get_resource_id_or_name("8c2e8369-3b81-471a-9945-32a3c67c31b7") + assert resource_id == "8c2e8369-3b81-471a-9945-32a3c67c31b7" and resource_name is None + + +def test_get_json_representation(mocker, context_object): + mock_cls = mocker.Mock() + mocker.patch.object(commands.click, "echo") + mock_resource_id = mocker.Mock() + mock_resource_name = mocker.Mock() + mocker.patch.object(commands, "get_resource_id_or_name", mocker.Mock(return_value=(mock_resource_id, mock_resource_name))) + json_repr = commands.get_json_representation(context_object["API_CLIENT"], context_object["WORKSPACE_ID"], mock_cls, "resource_to_get") + commands.get_resource_id_or_name.assert_called_with("resource_to_get") + mock_cls.assert_called_with( + context_object["API_CLIENT"], context_object["WORKSPACE_ID"], resource_id=mock_resource_id, resource_name=mock_resource_name + ) + assert json_repr == mock_cls.return_value.to_json.return_value + + +@pytest.mark.parametrize( + "command, resource_cls, resource", + [ + (commands.source, commands.Source, "my_resource_id"), + (commands.destination, commands.Destination, "my_resource_id"), + (commands.connection, commands.Connection, "my_resource_id"), + ], +) +def test_commands(context_object, mocker, command, resource_cls, resource): + mocker.patch.object(commands, "get_json_representation", mocker.Mock(return_value='{"foo": "bar"}')) + runner = CliRunner() + result = runner.invoke(command, [resource], obj=context_object) + commands.get_json_representation.assert_called_once_with( + context_object["API_CLIENT"], context_object["WORKSPACE_ID"], resource_cls, resource + ) + assert result.exit_code == 0 + + +# @pytest.mark.parametrize( +# "command,resource_id", +# [ +# (commands.destination, "my_resource_id"), +# ], +# ) +# def test_destination(mocker, context_object, command, resource_id): +# runner = CliRunner() +# mocker.patch.object(commands, "Destination", mocker.Mock()) +# mock_renderer = commands.Destination.return_value +# mock_renderer.get_remote_resource.return_value = '{"hello": "world"}' +# result = runner.invoke(command, [resource_id], obj=context_object) +# assert result.exit_code == 0 +# commands.Destination.assert_called_with(context_object["API_CLIENT"], context_object["WORKSPACE_ID"], resource_id) + + +# @pytest.mark.parametrize( +# "command,resource_id", +# [ +# (commands.connection, "my_resource_id"), +# ], +# ) +# def test_connection(mocker, context_object, command, resource_id): +# runner = CliRunner() +# mocker.patch.object(commands, "Connection", mocker.Mock()) +# mock_renderer = commands.Connection.return_value +# mock_renderer.get_remote_resource.return_value = '{"hello": "world"}' +# result = runner.invoke(command, [resource_id], obj=context_object) +# assert result.exit_code == 0 +# commands.Connection.assert_called_with(context_object["API_CLIENT"], context_object["WORKSPACE_ID"], resource_id) diff --git a/octavia-cli/unit_tests/test_get/test_resources.py b/octavia-cli/unit_tests/test_get/test_resources.py new file mode 100644 index 000000000000..3ac680c6a239 --- /dev/null +++ b/octavia-cli/unit_tests/test_get/test_resources.py @@ -0,0 +1,137 @@ +# +# Copyright (c) 2022 Airbyte, Inc., all rights reserved. +# + +import pytest +from airbyte_api_client.api import destination_api, source_api, web_backend_api +from airbyte_api_client.model.destination_id_request_body import DestinationIdRequestBody +from airbyte_api_client.model.source_id_request_body import SourceIdRequestBody +from airbyte_api_client.model.web_backend_connection_request_body import WebBackendConnectionRequestBody +from octavia_cli.get.resources import BaseResource, Connection, Destination, DuplicateResourceError, ResourceNotFoundError, Source + + +class TestBaseResource: + @pytest.fixture + def patch_base_class(self, mocker): + # Mock abstract methods to enable instantiating abstract class + mocker.patch.object(BaseResource, "__abstractmethods__", set()) + mocker.patch.object(BaseResource, "api", mocker.Mock()) + mocker.patch.object(BaseResource, "get_function_name", "get_function_name") + mocker.patch.object(BaseResource, "get_payload", "get_payload") + mocker.patch.object(BaseResource, "list_for_workspace_function_name", "list_for_workspace_function_name") + mocker.patch.object(BaseResource, "name", "fake_resource") + + @pytest.mark.parametrize( + "resource_id, resource_name, expected_error, expected_error_message", + [ + ("my_resource_id", None, None, None), + (None, "my_resource_name", None, None), + (None, None, ValueError, "resource_id and resource_name keyword arguments can't be both None."), + ("my_resource_id", "my_resource_name", ValueError, "resource_id and resource_name keyword arguments can't be both set."), + ], + ) + def test_init(self, patch_base_class, mock_api_client, resource_id, resource_name, expected_error, expected_error_message): + if expected_error: + with pytest.raises(expected_error, match=expected_error_message): + base_resource = BaseResource(mock_api_client, "workspace_id", resource_id=resource_id, resource_name=resource_name) + else: + base_resource = BaseResource(mock_api_client, "workspace_id", resource_id=resource_id, resource_name=resource_name) + base_resource.api.assert_called_with(mock_api_client) + assert base_resource.api_instance == base_resource.api.return_value + assert base_resource.workspace_id == "workspace_id" + assert base_resource._get_fn == getattr(base_resource.api, base_resource.get_function_name) + assert base_resource._list_for_workspace_fn == getattr(base_resource.api, base_resource.list_for_workspace_function_name) + assert base_resource.resource_id == resource_id + assert base_resource.resource_name == resource_name + + @pytest.mark.parametrize( + "resource_name, api_response_resources_names, expected_error, expected_error_message", + [ + ("foo", ["foo", "bar"], None, None), + ("foo", ["bar", "fooo"], ResourceNotFoundError, "The fake_resource foo was not found in your current Airbyte workspace."), + ( + "foo", + ["foo", "foo"], + DuplicateResourceError, + "2 fake_resources with the name foo were found in your current Airbyte workspace.", + ), + ], + ) + def test__find_by_resource_name( + self, mocker, patch_base_class, mock_api_client, resource_name, api_response_resources_names, expected_error, expected_error_message + ): + mock_api_response_records = [] + for fake_resource_name in api_response_resources_names: + mock_api_response_record = mocker.Mock() # We can't set the mock name on creation as it's a reserved attribute + mock_api_response_record.name = fake_resource_name + mock_api_response_records.append(mock_api_response_record) + + mocker.patch.object( + BaseResource, "_list_for_workspace_fn", mocker.Mock(return_value=mocker.Mock(fake_resources=mock_api_response_records)) + ) + base_resource = BaseResource(mock_api_client, "workspace_id", resource_id=None, resource_name=resource_name) + if not expected_error: + found_resource = base_resource._find_by_resource_name() + assert found_resource.name == resource_name + if expected_error: + with pytest.raises(expected_error, match=expected_error_message): + base_resource._find_by_resource_name() + + def test__find_by_id(self, mocker, patch_base_class, mock_api_client): + mocker.patch.object(BaseResource, "_get_fn") + base_resource = BaseResource(mock_api_client, "workspace_id", resource_id="my_resource_id") + base_resource._find_by_resource_id() + base_resource._get_fn.assert_called_with(base_resource.api_instance, base_resource.get_payload) + + @pytest.mark.parametrize("resource_id, resource_name", [("my_resource_id", None), (None, "my_resource_name")]) + def test_get_remote_resource(self, mocker, patch_base_class, mock_api_client, resource_id, resource_name): + mocker.patch.object(BaseResource, "_find_by_resource_id") + mocker.patch.object(BaseResource, "_find_by_resource_name") + base_resource = BaseResource(mock_api_client, "workspace_id", resource_id=resource_id, resource_name=resource_name) + remote_resource = base_resource.get_remote_resource() + if resource_id is not None: + base_resource._find_by_resource_id.assert_called_once() + base_resource._find_by_resource_name.assert_not_called() + assert remote_resource == base_resource._find_by_resource_id.return_value + if resource_name is not None: + base_resource._find_by_resource_id.assert_not_called() + base_resource._find_by_resource_name.assert_called_once() + assert remote_resource == base_resource._find_by_resource_name.return_value + + def test_to_json(self, mocker, patch_base_class, mock_api_client): + mocker.patch.object( + BaseResource, "get_remote_resource", mocker.Mock(return_value=mocker.Mock(to_dict=mocker.Mock(return_value={"foo": "bar"}))) + ) + base_resource = BaseResource(mock_api_client, "workspace_id", resource_id="my_resource_id") + json_repr = base_resource.to_json() + assert json_repr == '{"foo": "bar"}' + + +class TestSource: + def test_init(self, mock_api_client): + assert Source.__base__ == BaseResource + source = Source(mock_api_client, "workspace_id", "resource_id") + assert source.api == source_api.SourceApi + assert source.get_function_name == "get_source" + assert source.list_for_workspace_function_name == "list_sources_for_workspace" + assert source.get_payload == SourceIdRequestBody("resource_id") + + +class TestDestination: + def test_init(self, mock_api_client): + assert Destination.__base__ == BaseResource + destination = Destination(mock_api_client, "workspace_id", "resource_id") + assert destination.api == destination_api.DestinationApi + assert destination.get_function_name == "get_destination" + assert destination.list_for_workspace_function_name == "list_destinations_for_workspace" + assert destination.get_payload == DestinationIdRequestBody("resource_id") + + +class TestConnection: + def test_init(self, mock_api_client): + assert Connection.__base__ == BaseResource + connection = Connection(mock_api_client, "workspace_id", "resource_id") + assert connection.api == web_backend_api.WebBackendApi + assert connection.get_function_name == "web_backend_get_connection" + assert connection.list_for_workspace_function_name == "web_backend_list_connections_for_workspace" + assert connection.get_payload == WebBackendConnectionRequestBody(with_refreshed_catalog=False, connection_id=connection.resource_id)