Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add GCP Stack creation Wizard (CLI) #2826

Merged
merged 84 commits into from
Jul 11, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
84 commits
Select commit Hold shift + click to select a range
4af4307
initial codes
avishniakov Jun 26, 2024
000a021
Merge branch 'develop' into feature/PRD-477-stack-wizard-cli
avishniakov Jun 26, 2024
7461dec
prettify a bit
avishniakov Jun 26, 2024
d03804a
some aws pieces
avishniakov Jun 26, 2024
7399ba8
add container registry
avishniakov Jun 27, 2024
0d5b646
refactor
avishniakov Jun 27, 2024
f6a3154
Merge branch 'develop' into feature/PRD-477-stack-wizard-cli
avishniakov Jun 28, 2024
2cf6ad5
new models
bcdurak Jun 28, 2024
e60489e
new models
bcdurak Jun 28, 2024
a293dc6
new constant
bcdurak Jun 28, 2024
be18c34
converting it into a request
bcdurak Jun 28, 2024
37e664f
defining interface
bcdurak Jun 28, 2024
092ea24
rest zen store
bcdurak Jun 28, 2024
524e4aa
checking permissions and scoping the request in the endpoint
bcdurak Jun 28, 2024
fa437fa
first version of the sql zen store
bcdurak Jun 28, 2024
a8bb05c
rbac fixes
avishniakov Jun 28, 2024
62635a8
misc fixes
avishniakov Jun 28, 2024
8c0b71d
remove names duplicates
avishniakov Jun 28, 2024
3402bf8
multiple service connectors support
avishniakov Jun 28, 2024
8cd5c02
maybe working
avishniakov Jun 28, 2024
a0530b8
likely to work
avishniakov Jun 28, 2024
4c2e37f
some fixes and naming checks
bcdurak Jun 28, 2024
8b847ba
linting
bcdurak Jun 28, 2024
0801c59
analytics, name checks for stacks and cleanup
bcdurak Jun 28, 2024
63d9a35
typo
bcdurak Jun 28, 2024
5e4235e
hamza's feedback
avishniakov Jun 28, 2024
e73590d
minor docs update
avishniakov Jun 28, 2024
b13c09e
various fixes from PR comments
bcdurak Jul 1, 2024
c97afe9
several more fixes
bcdurak Jul 1, 2024
81632cd
reversing the order of the cleanup
bcdurak Jul 1, 2024
6c7e9d3
darglint
avishniakov Jul 1, 2024
37eec51
ruff
avishniakov Jul 1, 2024
19ddc42
mypy
avishniakov Jul 1, 2024
00823f6
ruff
avishniakov Jul 1, 2024
5437a19
ruff
avishniakov Jul 1, 2024
79f835f
removing the extra verification
bcdurak Jul 1, 2024
8574d86
fetch stack by full name
avishniakov Jul 1, 2024
6130187
Merge branch 'feature/PRD-477-stack-wizard-cli' of https://github.com…
avishniakov Jul 1, 2024
35a36b4
mypy
avishniakov Jul 1, 2024
dcd8e5d
workspaced full stack request
avishniakov Jul 2, 2024
63ffd29
fix normal stack register
avishniakov Jul 2, 2024
9a9dd3f
add `service_connector_resource_id`
avishniakov Jul 2, 2024
10a781b
tiny renames
avishniakov Jul 2, 2024
dc66278
use passed `resource_id`
avishniakov Jul 2, 2024
19cd0e3
Fix connector resource type in `zenml <component> describe` CLI command
stefannica Jul 3, 2024
00a1023
use auto config option instead of implicit
avishniakov Jul 3, 2024
eeb8bea
add connector index validator
avishniakov Jul 3, 2024
8bf0a34
suggestions and mypy
avishniakov Jul 3, 2024
2269a4b
Merge branch 'develop' into feature/PRD-477-stack-wizard-cli
avishniakov Jul 3, 2024
a1cf862
ignore local, if rejected
avishniakov Jul 3, 2024
b577cc8
mypy
avishniakov Jul 3, 2024
7f28028
Auto-update of LLM Finetuning template
actions-user Jul 3, 2024
d1c0394
revert fullstack model changes
avishniakov Jul 3, 2024
58fa9e8
Merge branch 'feature/PRD-477-stack-wizard-cli' of https://github.com…
avishniakov Jul 3, 2024
15661e0
solving the linting issue
bcdurak Jul 4, 2024
28a6db0
GCP stack wizard
avishniakov Jul 4, 2024
8dbe58a
Merge branch 'feature/PRD-477-stack-wizard-cli' into feature/PRD-477-…
avishniakov Jul 4, 2024
5e57479
changing the models to include the new labels as well
bcdurak Jul 4, 2024
ad3a438
adding the labels to stacks in the client
bcdurak Jul 4, 2024
6a1d35d
changing the schema
bcdurak Jul 4, 2024
0c32fd3
adding the necessary migration script
bcdurak Jul 4, 2024
c7bfc24
adjusting the sql zen store after the labels change
bcdurak Jul 4, 2024
893a1fc
the correct migration script
bcdurak Jul 4, 2024
7ccdb35
removed autogenerated comments
bcdurak Jul 4, 2024
568b81f
Merge branch 'feature/PRD-477-stack-wizard-cli' into feature/PRD-477-…
avishniakov Jul 4, 2024
03225d4
Merge branch 'develop' into feature/PRD-477-stack-wizard-cli
stefannica Jul 5, 2024
61d4c38
Merge branch 'feature/PRD-477-stack-wizard-cli' into feature/PRD-477-…
avishniakov Jul 5, 2024
09c25d6
Merge branch 'develop' into feature/PRD-477-add-gcp
avishniakov Jul 5, 2024
8742967
fix after merge mess
avishniakov Jul 5, 2024
78d1283
Merge branch 'develop' into feature/PRD-477-add-gcp
avishniakov Jul 8, 2024
2cc1441
lint
avishniakov Jul 8, 2024
5f48793
move timeout closer to rest
avishniakov Jul 10, 2024
25e32ee
generate_temporary_tokens only in skypilot cases
avishniakov Jul 10, 2024
b32bfcb
Merge branch 'develop' into feature/PRD-477-add-gcp
avishniakov Jul 10, 2024
8461da9
rework timeout as suggested
avishniakov Jul 11, 2024
740ca42
rename
avishniakov Jul 11, 2024
3179a7f
Merge branch 'develop' into feature/PRD-477-add-gcp
avishniakov Jul 11, 2024
4c77f8a
get regions from zen_store
avishniakov Jul 11, 2024
b35f2b8
disable wizard with SC for local ZenMLs
avishniakov Jul 11, 2024
dae3ad7
left-over timeout
avishniakov Jul 11, 2024
dbb8e51
add docs
avishniakov Jul 11, 2024
1cde618
formatting
avishniakov Jul 11, 2024
c45b73a
formatting
avishniakov Jul 11, 2024
21c69fd
restructure a bit
avishniakov Jul 11, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
163 changes: 114 additions & 49 deletions docs/book/how-to/stack-deployment/register-a-cloud-stack.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,84 +33,149 @@ In order to register a remote stack over the CLI with the stack wizard,
you can use the following command:

```shell
zenml stack register <STACK_NAME> -p aws
zenml stack register <STACK_NAME> -p {aws|gcp}
```

To register the cloud stack, the first thing that the wizard needs is a service
connector. You can either use an existing connector by providing its ID
`-c <CONNECTOR_ID>` or the wizard will create one for you.
connector. You can either use an existing connector by providing its ID or name
`-sc <SERVICE_CONNECTOR_ID_OR_NAME>` or the wizard will create one for you.

Similar to the service connector, you can also use existing stack components.
However, this is only possible if these component are already configured with
However, this is only possible if these components are already configured with
the same service connector that you provided through the parameter
described above.

{% hint style="warning" %}
Currently, the stack wizard only works on AWS. We are working on bringing
support to GCP and Azure as well. Stay in touch for further updates.
Currently, the stack wizard only works on AWS and GCP. We are working on bringing
support to Azure as well. Stay in touch for further updates.
{% endhint %}

### AWS
### Define Service Connector

Below you will find cloud-specific selection options. Based on your selection, you will have to provide the required parameters listed
below. This will allow ZenML to create a Service Connector and
authenticate you to use your cloud resources.

#### AWS

If you select `aws` as your cloud provider, and you haven't selected a connector
yet, you will be prompted to select an authentication method for your stack.

{% code title="Example Command Output" %}
```
Available authentication methods for aws
┏━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Choice ┃ Name ┃ Required ┃
┡━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ [0] │ AWS Secret Key │ aws_access_key_id (AWS Access Key ID) │
│ │ │ aws_secret_access_key (AWS Secret Access Key) │
│ │ │ region (AWS Region) │
│ │ │ │
├────────┼──────────────────────┼────────────────────────────────────────────────┤
│ [1] │ AWS STS Token │ aws_access_key_id (AWS Access Key ID) │
│ │ │ aws_secret_access_key (AWS Secret Access Key) │
│ │ │ aws_session_token (AWS Session Token) │
│ │ │ region (AWS Region) │
│ │ │ │
├────────┼──────────────────────┼────────────────────────────────────────────────┤
│ [2] │ AWS IAM Role │ aws_access_key_id (AWS Access Key ID) │
│ │ │ aws_secret_access_key (AWS Secret Access Key) │
│ │ │ region (AWS Region) │
│ │ │ role_arn (AWS IAM Role ARN) │
│ │ │ │
├────────┼──────────────────────┼────────────────────────────────────────────────┤
│ [3] │ AWS Session Token │ aws_access_key_id (AWS Access Key ID) │
│ │ │ aws_secret_access_key (AWS Secret Access Key) │
│ │ │ region (AWS Region) │
│ │ │ │
├────────┼──────────────────────┼────────────────────────────────────────────────┤
│ [4] │ AWS Federation Token │ aws_access_key_id (AWS Access Key ID) │
│ │ │ aws_secret_access_key (AWS Secret Access Key) │
│ │ │ region (AWS Region) │
│ │ │ │
└────────┴──────────────────────┴────────────────────────────────────────────────┘
Available authentication methods for aws
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Choice ┃ Name ┃ Required ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ [0] │ AWS Secret Key │ aws_access_key_id (AWS Access │
│ │ │ Key ID) │
│ │ │ aws_secret_access_key (AWS │
│ │ │ Secret Access Key) │
│ │ │ region (AWS Region) │
│ │ │ │
├─────────┼────────────────────────────────┼────────────────────────────────┤
│ [1] │ AWS STS Token │ aws_access_key_id (AWS Access │
│ │ │ Key ID) │
│ │ │ aws_secret_access_key (AWS │
│ │ │ Secret Access Key) │
│ │ │ aws_session_token (AWS │
│ │ │ Session Token) │
│ │ │ region (AWS Region) │
│ │ │ │
├─────────┼────────────────────────────────┼────────────────────────────────┤
│ [2] │ AWS IAM Role │ aws_access_key_id (AWS Access │
│ │ │ Key ID) │
│ │ │ aws_secret_access_key (AWS │
│ │ │ Secret Access Key) │
│ │ │ region (AWS Region) │
│ │ │ role_arn (AWS IAM Role ARN) │
│ │ │ │
├─────────┼────────────────────────────────┼────────────────────────────────┤
│ [3] │ AWS Session Token │ aws_access_key_id (AWS Access │
│ │ │ Key ID) │
│ │ │ aws_secret_access_key (AWS │
│ │ │ Secret Access Key) │
│ │ │ region (AWS Region) │
│ │ │ │
├─────────┼────────────────────────────────┼────────────────────────────────┤
│ [4] │ AWS Federation Token │ aws_access_key_id (AWS Access │
│ │ │ Key ID) │
│ │ │ aws_secret_access_key (AWS │
│ │ │ Secret Access Key) │
│ │ │ region (AWS Region) │
│ │ │ │
└─────────┴────────────────────────────────┴────────────────────────────────┘
```
{% endcode %}

Based on your selection, you will have to provide the required parameters listed
above. This will allow ZenML to create a Service Connector and
authenticate you to use your cloud resources.
#### GCP

If you select `gcp` as your cloud provider, and you haven't selected a connector
yet, you will be prompted to select an authentication method for your stack.

{% code title="Example Command Output" %}
```
Available authentication methods for gcp
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Choice ┃ Name ┃ Required ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ [0] │ GCP User Account │ user_account_json (GCP User │
│ │ │ Account Credentials JSON │
│ │ │ optionally base64 encoded.) │
│ │ │ project_id (GCP Project ID │
│ │ │ where the target resource is │
│ │ │ located.) │
│ │ │ │
├─────────┼────────────────────────────────┼────────────────────────────────┤
│ [1] │ GCP Service Account │ service_account_json (GCP │
│ │ │ Service Account Key JSON │
│ │ │ optionally base64 encoded.) │
│ │ │ │
├─────────┼────────────────────────────────┼────────────────────────────────┤
│ [2] │ GCP External Account │ external_account_json (GCP │
│ │ │ External Account JSON │
│ │ │ optionally base64 encoded.) │
│ │ │ project_id (GCP Project ID │
│ │ │ where the target resource is │
│ │ │ located.) │
│ │ │ │
├─────────┼────────────────────────────────┼────────────────────────────────┤
│ [3] │ GCP Oauth 2.0 Token │ token (GCP OAuth 2.0 Token) │
│ │ │ project_id (GCP Project ID │
│ │ │ where the target resource is │
│ │ │ located.) │
│ │ │ │
├─────────┼────────────────────────────────┼────────────────────────────────┤
│ [4] │ GCP Service Account │ service_account_json (GCP │
│ │ Impersonation │ Service Account Key JSON │
│ │ │ optionally base64 encoded.) │
│ │ │ target_principal (GCP Service │
│ │ │ Account Email to impersonate) │
│ │ │ │
└─────────┴────────────────────────────────┴────────────────────────────────┘
```
{% endcode %}

### Defining cloud components

Next, for each missing component, the available resources will be listed to
you as follows:

{% code title="Example Command Output for Artifact Stores" %}
```
Available AWS storages
┏━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Choice ┃ Storage
┡━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ [0] │ s3://**************
├────────┼─────────────────────────────────────────────────────────────┤
│ [1] │ s3://**************
└────────┴─────────────────────────────────────────────────────────────┘
Available GCP storages
┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Choice ┃ Storage
┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ [0] │ gs://***************************
├───────────────┼───────────────────────────────────────────────────────────┤
│ [1] │ gs://***************************
└───────────────┴───────────────────────────────────────────────────────────┘
```
{% endcode %}

### Final steps

Based on your selection, ZenML will create the stack component and ultimately
register the stack for you.

Expand Down
Loading
Loading