Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🎉 octavia-cli: Add ability to import existing resources #14137

Merged
merged 47 commits into from
Jul 1, 2022
Merged

🎉 octavia-cli: Add ability to import existing resources #14137

Show file tree
Hide file tree
Changes from 45 commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
aa1ff16
init
danieldiamond May 26, 2022
19a52a2
Update docs and add tests
danieldiamond May 27, 2022
0150ae9
Revert docker run docs
danieldiamond May 27, 2022
7748d63
Add "update" commands
danieldiamond May 29, 2022
58303ea
Add "update sources" and "update destinations" to create yaml files f…
danieldiamond May 29, 2022
d66efbc
Merge branch 'master' into danieldiamond/octavia-cli-generate-existin…
danieldiamond Jun 24, 2022
48676c1
WIP
danieldiamond Jun 24, 2022
89549dc
Revert "WIP"
danieldiamond Jun 24, 2022
87f6b48
Remove update method
danieldiamond Jun 24, 2022
a41fd9b
Merge branch 'master' into danieldiamond/octavia-cli-generate-existin…
danieldiamond Jun 24, 2022
29c3a64
Remove update from tests
danieldiamond Jun 24, 2022
35d9cdd
Remove update related methods in generate_renderers
danieldiamond Jun 24, 2022
a4b03ed
Merge branch 'master' into danieldiamond/octavia-cli-generate-existin…
danieldiamond Jun 26, 2022
528afc1
Reformat get commands
danieldiamond Jun 26, 2022
e376f4e
Remove DuplicateResourceError from apply
danieldiamond Jun 26, 2022
3d96529
Remove ConnectorRenderer from generate
danieldiamond Jun 26, 2022
6f3944d
Update unit tests for commands and resources
danieldiamond Jun 26, 2022
c44903e
Update docs and setup
danieldiamond Jun 26, 2022
c31f2bc
init import commands
danieldiamond Jun 26, 2022
810b089
Add quotes to handle yaml characters
danieldiamond Jun 26, 2022
f147e27
Add source and destination commands
danieldiamond Jun 26, 2022
18f6967
Remove ConnectorRenderer from generate
danieldiamond Jun 26, 2022
3ec8ce3
Merge branch 'master' into danieldiamond/octavia-cli-import
alafanechere Jun 27, 2022
9cc897e
Remove commands for import of multiple resources
danieldiamond Jun 27, 2022
da2e0b5
Improve writes: Avoid intermediate yaml file saved to disk, check if …
danieldiamond Jun 27, 2022
42d37a5
Manage connections and store state
danieldiamond Jun 27, 2022
a55547d
Make resource_requirements an optional field in apply.resources
danieldiamond Jun 28, 2022
654e3a3
Slugify resource_name in get_output_path method in generate.renderers
danieldiamond Jun 28, 2022
108d8a4
Add docstring to import commands
danieldiamond Jun 28, 2022
e1dc6fd
Merge branch 'master' into danieldiamond/octavia-cli-import
danieldiamond Jun 28, 2022
57371e3
Add user warning message to update secrets in source or destination
danieldiamond Jun 28, 2022
8779a00
add slugify dependency
alafanechere Jun 29, 2022
678f8db
write unit tests
alafanechere Jun 29, 2022
cbd183b
wip integration tests
alafanechere Jun 29, 2022
938eef0
fix connection config for integration tests
alafanechere Jun 30, 2022
305bafe
improve connection config cleaning on import
alafanechere Jun 30, 2022
efaa21f
finish integration tests
alafanechere Jun 30, 2022
fa69f09
Merge branch 'master' into danieldiamond/octavia-cli-import
alafanechere Jun 30, 2022
488dc01
update readme
alafanechere Jun 30, 2022
b62796c
update readme
alafanechere Jun 30, 2022
e38c422
update readme
alafanechere Jun 30, 2022
1e26425
update readme
alafanechere Jun 30, 2022
269ffdc
update readme
alafanechere Jun 30, 2022
470824b
Merge branch 'master' into danieldiamond/octavia-cli-import
danieldiamond Jun 30, 2022
3c359fe
Iterate on import_configuration in ConnectionRenderer
danieldiamond Jun 30, 2022
36904d2
fix docstring + clean
alafanechere Jul 1, 2022
2c7dfbb
bump to the right version
alafanechere Jul 1, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 81 additions & 13 deletions octavia-cli/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,11 +138,11 @@ docker-compose run octavia-cli <command>`
### `octavia` command flags

| **Flag** | **Description** | **Env Variable** | **Default** |
| ---------------------------------------- | --------------------------------------------------------------------------------- | -------------------------- | ------------------------------------------------------ | ---- |
| ---------------------------------------- | --------------------------------------------------------------------------------- | -------------------------- | ------------------------------------------------------ |
| `--airbyte-url` | Airbyte instance URL. | `AIRBYTE_URL` | `http://localhost:8000` |
| `--workspace-id` | Airbyte workspace id. | `AIRBYTE_WORKSPACE_ID` | The first workspace id found on your Airbyte instance. |
| `--enable-telemetry/--disable-telemetry` | Enable or disable the sending of telemetry data. | `OCTAVIA_ENABLE_TELEMETRY` | True |
| `--api-http-header` | HTTP Header value pairs passed while calling Airbyte's API | not supported. | None | None |
| `--api-http-header` | HTTP Header value pairs passed while calling Airbyte's API | None | None |
| `--api-http-headers-file-path` | Path to the YAML file that contains custom HTTP Headers to send to Airbyte's API. | None | None |

#### Using custom HTTP headers
Expand All @@ -157,7 +157,7 @@ You can also use a custom YAML file (one is already created on init in `api_http

```yaml
headers:
Authorization: Basic foobar==
Authorization: Bearer my-secret-token
User-Agent: octavia-cli/0.0.0
```

Expand All @@ -183,6 +183,9 @@ headers:
| **`octavia get source`** | Get the JSON representation of an existing source in current the Airbyte workspace. |
| **`octavia get destination`** | Get the JSON representation of an existing destination in the current Airbyte workspace. |
| **`octavia get connection`** | Get the JSON representation of an existing connection in the current Airbyte workspace. |
| **`octavia import source`** | Import an existing source to manage it with octavia-cli. |
| **`octavia import destination`** | Import an existing destination to manage it with octavia-cli. |
| **`octavia import connection`** | Import an existing connection to manage it with octavia-cli. |
| **`octavia generate source`** | Generate a local YAML configuration for a new source. |
| **`octavia generate destination`** | Generate a local YAML configuration for a new destination. |
| **`octavia generate connection`** | Generate a local YAML configuration for a new connection. |
Expand Down Expand Up @@ -485,6 +488,70 @@ $ octavia get connection "Poke To PG"
"sourceCatalogId": "18102e7c-5340-4000-85f3-204ab7715801"
}
```

#### `octavia import source <SOURCE_ID> or <SOURCE_NAME>`

Import an existing source to manage it with octavia-cli. You can use a source ID or name.

| **Argument** | **Description** |
| --------------| -----------------|
| `SOURCE_ID` | The source id. |
| `SOURCE_NAME` | The source name. |

**Examples**:

```bash
$ octavia import source poke
🐙 - Octavia is targetting your Airbyte instance running at http://localhost:8000 on workspace 75658e4f-e5f0-4e35-be0c-bdad33226c94.
✅ - Imported source poke in sources/poke/configuration.yaml. State stored in sources/poke/state_75658e4f-e5f0-4e35-be0c-bdad33226c94.yaml
⚠️ - Please update any secrets stored in sources/poke/configuration.yaml
```
You know have local configuration file for an Airbyte source that was already existing.
You need to edit any secret value that exist in this configuration as secrets are not imported.
You can edit the configuration and run `octavia apply` to continue managing it with octavia-cli.

#### `octavia import destination <DESTINATION_ID> or <DESTINATION_NAME>`

Import an existing destination to manage it with octavia-cli. You can use a destination ID or name.

| **Argument** | **Description** |
| -------------------| ----------------------|
| `DESTINATION_ID` | The destination id. |
| `DESTINATION_NAME` | The destination name. |

**Examples**:

```bash
$ octavia import destination pg
🐙 - Octavia is targetting your Airbyte instance running at http://localhost:8000 on workspace 75658e4f-e5f0-4e35-be0c-bdad33226c94.
✅ - Imported destination pg in destinations/pg/configuration.yaml. State stored in destinations/pg/state_75658e4f-e5f0-4e35-be0c-bdad33226c94.yaml
⚠️ - Please update any secrets stored in destinations/pg/configuration.yaml
```
You know have local configuration file for an Airbyte destination that was already existing.
You need to edit any secret value that exist in this configuration as secrets are not imported.
You can edit the configuration and run `octavia apply` to continue managing it with octavia-cli.

#### `octavia import connection <CONNECTION_ID> or <CONNECTION_NAME>`

Import an existing connection to manage it with octavia-cli. You can use a connection ID or name.

| **Argument** | **Description** |
| ------------------| ---------------------|
| `CONNECTION_ID` | The connection id. |
| `CONNECTION_NAME` | The connection name. |

**Examples**:

```bash
$ octavia import connection poke-to-pg
🐙 - Octavia is targetting your Airbyte instance running at http://localhost:8000 on workspace 75658e4f-e5f0-4e35-be0c-bdad33226c94.
✅ - Imported connection poke-to-pg in connections/poke-to-pg/configuration.yaml. State stored in connections/poke-to-pg/state_75658e4f-e5f0-4e35-be0c-bdad33226c94.yaml
⚠️ - Please update any secrets stored in connections/poke-to-pg/configuration.yaml
```
You know have local configuration file for an Airbyte connection that was already existing.
**N.B.: You first need to import the source and destination used by the connection.**
You can edit the configuration and run `octavia apply` to continue managing it with octavia-cli.

#### `octavia generate source <DEFINITION_ID> <SOURCE_NAME>`

Generate a YAML configuration for a source.
Expand Down Expand Up @@ -607,13 +674,14 @@ You can disable telemetry by setting the `OCTAVIA_ENABLE_TELEMETRY` environment

## Changelog

| Version | Date | Description | PR |
| ------- | ---------- | ------------------------------------------------------------ | ----------------------------------------------------------- |
| 0.39.27 | 2022-06-24 | Create get command to retrieve resources JSON representation | [#13254](https://github.com/airbytehq/airbyte/pull/13254) |
| 0.39.19 | 2022-06-16 | Allow connection management on multiple workspaces | [#13070](https://github.com/airbytehq/airbyte/pull/12727) |
| 0.39.19 | 2022-06-15 | Allow users to set custom HTTP headers | [#12893](https://github.com/airbytehq/airbyte/pull/12893) |
| 0.39.14 | 2022-05-12 | Enable normalization on connection | [#12727](https://github.com/airbytehq/airbyte/pull/12727) |
| 0.37.0 | 2022-05-05 | Use snake case in connection fields | [#12133](https://github.com/airbytehq/airbyte/pull/12133) |
| 0.35.68 | 2022-04-15 | Improve telemetry | [#12072](https://github.com/airbytehq/airbyte/issues/11896) |
| 0.35.68 | 2022-04-12 | Add telemetry | [#11896](https://github.com/airbytehq/airbyte/issues/11896) |
| 0.35.61 | 2022-04-07 | Alpha release | [EPIC](https://github.com/airbytehq/airbyte/issues/10704) |
| Version | Date | Description | PR |
| ------- | ---------- | --------------------------------------------------------------------------------------| ----------------------------------------------------------- |
| 0.39.30 | 2022-06-30 | Create import command to import and manage existing Airbyte resource from octavia-cli | [#14137](https://github.com/airbytehq/airbyte/pull/14137) |
| 0.39.27 | 2022-06-24 | Create get command to retrieve resources JSON representation | [#13254](https://github.com/airbytehq/airbyte/pull/13254) |
| 0.39.19 | 2022-06-16 | Allow connection management on multiple workspaces | [#13070](https://github.com/airbytehq/airbyte/pull/12727) |
| 0.39.19 | 2022-06-15 | Allow users to set custom HTTP headers | [#12893](https://github.com/airbytehq/airbyte/pull/12893) |
| 0.39.14 | 2022-05-12 | Enable normalization on connection | [#12727](https://github.com/airbytehq/airbyte/pull/12727) |
| 0.37.0 | 2022-05-05 | Use snake case in connection fields | [#12133](https://github.com/airbytehq/airbyte/pull/12133) |
| 0.35.68 | 2022-04-15 | Improve telemetry | [#12072](https://github.com/airbytehq/airbyte/issues/11896) |
| 0.35.68 | 2022-04-12 | Add telemetry | [#11896](https://github.com/airbytehq/airbyte/issues/11896) |
| 0.35.61 | 2022-04-07 | Alpha release | [EPIC](https://github.com/airbytehq/airbyte/issues/10704) |
Loading