Skip to content

Commit

Permalink
Add helm chart support and update docs
Browse files Browse the repository at this point in the history
  • Loading branch information
@stefannica
stefannica committed Dec 15, 2023
1 parent 02b3d9c commit fe55e50
Show file tree
Hide file tree
Showing 10 changed files with 308 additions and 59 deletions.
37 changes: 30 additions & 7 deletions docs/book/deploying-zenml/zenml-self-hosted/deploy-with-docker.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -75,17 +75,32 @@ The SQL database is used as the default secret store. You only need to configure
These configuration options are only relevant if you're using the AWS Secrets Manager as the secrets store backend.
* **ZENML\_SECRETS\_STORE\_TYPE:** Set this to `aws` in order to set this type of secret store.
* **ZENML\_SECRETS\_STORE\_REGION\_NAME**: The AWS region to use. This must be set to the region where the AWS Secrets Manager service that you want to use is located.
* **ZENML\_SECRETS\_STORE\_AWS\_ACCESS\_KEY\_ID**: The AWS access key ID to use for authentication. This must be set to a valid AWS access key ID that has access to the AWS Secrets Manager service that you want to use. If you are using an IAM role attached to an EKS cluster to authenticate, you can omit this variable. NOTE: this is the same as setting the `AWS_ACCESS_KEY_ID` environment variable.
* **ZENML\_SECRETS\_STORE\_AWS\_SECRET\_ACCESS\_KEY**: The AWS secret access key to use for authentication. This must be set to a valid AWS secret access key that has access to the AWS Secrets Manager service that you want to use. If you are using an IAM role attached to an EKS cluster to authenticate, you can omit this variable. NOTE: this is the same as setting the `AWS_SECRET_ACCESS_KEY` environment variable.
* **ZENML\_SECRETS\_STORE\_AWS\_SESSION\_TOKEN**: Optional AWS session token to use for authentication. NOTE: this is the same as setting the `AWS_SESSION_TOKEN` environment variable.
The AWS Secrets Store uses the ZenML AWS Service Connector under the hood to authenticate with the AWS Secrets Manager API. This means that you can use any of the [authentication methods supported by the AWS Service Connector](https://docs.zenml.io/stacks-and-components/auth-management/aws-service-connector#authentication-methods) to authenticate with the AWS Secrets Manager API. The following configuration options are supported:
* **ZENML\_SECRETS\_STORE\_AUTH\_METHOD**: The AWS Service Connector authentication method to use (e.g. `secret-key` or `iam-role`).
* **ZENML\_SECRETS\_STORE\_AUTH\_CONFIG**: The AWS Service Connector configuration, in JSON format (e.g. `{"role_arn": "arn:aws:iam::123456789012:role/MyRole"}`).
* **ZENML\_SECRETS\_STORE\_SECRET\_LIST\_REFRESH\_TIMEOUT**: AWS' [Secrets Manager](https://aws.amazon.com/secrets-manager) has a known issue where it does not immediately reflect new and updated secrets in the `list_secrets` results. To work around this issue, you can set this refresh timeout value to a non-zero value to get the ZenML server to wait after creating or updating an AWS secret until the changes are reflected in the secrets returned by `list_secrets` or the number of seconds specified by this value has elapsed. Defaults to `0` (disabled). Should not be set to a high value as it may cause thread starvation in the ZenML server on high load.
> **Note:** The remaining configuration options are deprecated and may be removed in a future release. Instead, you should set the `ZENML_SECRETS_STORE_AUTH_METHOD` and `ZENML_SECRETS_STORE_AUTH_CONFIG` variables to use the AWS Service Connector authentication method.
* **ZENML\_SECRETS\_STORE\_REGION\_NAME**: The AWS region to use. This must be set to the region where the AWS Secrets Manager service that you want to use is located.
* **ZENML\_SECRETS\_STORE\_AWS\_ACCESS\_KEY\_ID**: The AWS access key ID to use for authentication. This must be set to a valid AWS access key ID that has access to the AWS Secrets Manager service that you want to use. If you are using an IAM role attached to an EKS cluster to authenticate, you can omit this variable.
* **ZENML\_SECRETS\_STORE\_AWS\_SECRET\_ACCESS\_KEY**: The AWS secret access key to use for authentication. This must be set to a valid AWS secret access key that has access to the AWS Secrets Manager service that you want to use. If you are using an IAM role attached to an EKS cluster to authenticate, you can omit this variable.
{% endtab %}
{% tab title="GCP" %}
These configuration options are only relevant if you're using the GCP Secrets Manager as the secrets store backend.
* **ZENML\_SECRETS\_STORE\_TYPE:** Set this to `gcp` in order to set this type of secret store.
The GCP Secrets Store uses the ZenML GCP Service Connector under the hood to authenticate with the GCP Secrets Manager API. This means that you can use any of the [authentication methods supported by the GCP Service Connector](https://docs.zenml.io/stacks-and-components/auth-management/gcp-service-connector#authentication-methods) to authenticate with the GCP Secrets Manager API. The following configuration options are supported:
* **ZENML\_SECRETS\_STORE\_AUTH\_METHOD**: The GCP Service Connector authentication method to use (e.g. `service-account`).
* **ZENML\_SECRETS\_STORE\_AUTH\_CONFIG**: The GCP Service Connector configuration, in JSON format (e.g. `{"project_id": "my-project", "service_account_json": { ... }}`).
> **Note:** The remaining configuration options are deprecated and may be removed in a future release. Instead, you should set the `ZENML_SECRETS_STORE_AUTH_METHOD` and `ZENML_SECRETS_STORE_AUTH_CONFIG` variables to use the GCP Service Connector authentication method.
* **ZENML\_SECRETS\_STORE\_PROJECT\_ID**: The GCP project ID to use. This must be set to the project ID where the GCP Secrets Manager service that you want to use is located.
* **GOOGLE\_APPLICATION\_CREDENTIALS**: The path to the GCP service account credentials file to use for authentication. This must be set to a valid GCP service account credentials file that has access to the GCP Secrets Manager service that you want to use. If you are using a GCP service account attached to a GKE cluster to authenticate, you can omit this variable. NOTE: the path to the credentials file must be mounted into the container.
{% endtab %}
Expand All @@ -95,9 +110,17 @@ These configuration options are only relevant if you're using Azure Key Vault as
* **ZENML\_SECRETS\_STORE\_TYPE:** Set this to `azure` in order to set this type of secret store.
* **ZENML\_SECRETS\_STORE\_KEY\_VAULT\_NAME**: The name of the Azure Key Vault. This must be set to point to the Azure Key Vault instance that you want to use.
* **ZENML\_SECRETS\_STORE\_AZURE\_CLIENT\_ID**: The Azure application service principal client ID to use to authenticate with the Azure Key Vault API. If you are running the ZenML server hosted in Azure and are using a managed identity to access the Azure Key Vault service, you can omit this variable. NOTE: this is the same as setting the `AZURE_CLIENT_ID` environment variable.
* **ZENML\_SECRETS\_STORE\_AZURE\_CLIENT\_SECRET**: The Azure application service principal client secret to use to authenticate with the Azure Key Vault API. If you are running the ZenML server hosted in Azure and are using a managed identity to access the Azure Key Vault service, you can omit this variable. NOTE: this is the same as setting the `AZURE_CLIENT_SECRET` environment variable.
* **ZENML\_SECRETS\_STORE\_AZURE\_TENANT\_ID**: The Azure application service principal tenant ID to use to authenticate with the Azure Key Vault API. If you are running the ZenML server hosted in Azure and are using a managed identity to access the Azure Key Vault service, you can omit this variable. NOTE: this is the same as setting the `AZURE_TENANT_ID` environment variable.
The Azure Secrets Store uses the ZenML Azure Service Connector under the hood to authenticate with the Azure Key Vault API. This means that you can use any of the [authentication methods supported by the Azure Service Connector](https://docs.zenml.io/stacks-and-components/auth-management/azure-service-connector#authentication-methods) to authenticate with the Azure Key Vault API. The following configuration options are supported:
* **ZENML\_SECRETS\_STORE\_AUTH\_METHOD**: The Azure Service Connector authentication method to use (e.g. `service-account`).
* **ZENML\_SECRETS\_STORE\_AUTH\_CONFIG**: The Azure Service Connector configuration, in JSON format (e.g. `{"tenant_id": "my-tenant-id", "client_id": "my-client-id", "client_secret": "my-client-secret"}`).
> **Note:** The remaining configuration options are deprecated and may be removed in a future release. Instead, you should set the `ZENML_SECRETS_STORE_AUTH_METHOD` and `ZENML_SECRETS_STORE_AUTH_CONFIG` variables to use the Azure Service Connector authentication method.
* **ZENML\_SECRETS\_STORE\_AZURE\_CLIENT\_ID**: The Azure application service principal client ID to use to authenticate with the Azure Key Vault API. If you are running the ZenML server hosted in Azure and are using a managed identity to access the Azure Key Vault service, you can omit this variable.
* **ZENML\_SECRETS\_STORE\_AZURE\_CLIENT\_SECRET**: The Azure application service principal client secret to use to authenticate with the Azure Key Vault API. If you are running the ZenML server hosted in Azure and are using a managed identity to access the Azure Key Vault service, you can omit this variable.
* **ZENML\_SECRETS\_STORE\_AZURE\_TENANT\_ID**: The Azure application service principal tenant ID to use to authenticate with the Azure Key Vault API. If you are running the ZenML server hosted in Azure and are using a managed identity to access the Azure Key Vault service, you can omit this variable.
{% endtab %}
{% tab title="Hashicorp" %}
Expand Down
79 changes: 47 additions & 32 deletions docs/book/deploying-zenml/zenml-self-hosted/deploy-with-helm.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -312,7 +312,10 @@ This method requires you to configure a DNS service like AWS Route 53 or Google
{% tab title="AWS" %}
#### Using the AWS Secrets Manager as a secrets store backend

Unless explicitly disabled or configured otherwise, the ZenML server will use the SQL database as a secrets store backend. If you want to use the AWS Secrets Manager instead, you need to configure it in the Helm values. Depending on where you deploy your ZenML server and how your Kubernetes cluster is configured, you may also need to provide the AWS credentials needed to access the AWS Secrets Manager API:
Unless explicitly disabled or configured otherwise, the ZenML server will use the SQL database as a secrets store backend. If you want to use the AWS Secrets Manager instead, you need to configure it in the Helm values. Depending on where you deploy your ZenML server and how your Kubernetes cluster is configured, you may also need to provide the AWS credentials needed to access the AWS Secrets Manager API.

The AWS Secrets Store uses the ZenML AWS Service Connector under the hood to authenticate with the AWS Secrets Manager API. This means that you can use any of the [authentication methods supported by the AWS Service Connector](https://docs.zenml.io/stacks-and-components/auth-management/aws-service-connector#authentication-methods) to authenticate with the AWS Secrets Manager API:


```yaml
zenml:
Expand All @@ -331,24 +334,28 @@ Unless explicitly disabled or configured otherwise, the ZenML server will use th
# Configuration for the AWS Secrets Manager secrets store
aws:
# The AWS region to use. This must be set to the region where the AWS
# Secrets Manager service that you want to use is located.
region_name: us-east-1
# The AWS credentials to use to authenticate with the AWS Secrets
# Manager instance. You can omit these if you are running the ZenML server
# in an AWS EKS cluster that has an IAM role attached to it that has
# permissions to access the AWS Secrets Manager instance.
aws_access_key_id: <your AWS access key ID>
aws_secret_access_key: <your AWS secret access key>
aws_session_token: <your AWS session token>
# The AWS Service Connector authentication method to use.
authMethod: secret-key
# The AWS Service Connector configuration.
authConfig:
# The AWS region to use. This must be set to the region where the AWS
# Secrets Manager service that you want to use is located.
region: us-east-1
# The AWS credentials to use to authenticate with the AWS Secrets
aws_access_key_id: <your AWS access key ID>
aws_secret_access_key: <your AWS secret access key>
```
{% endtab %}

{% tab title="GCP" %}
#### Using the GCP Secrets Manager as a secrets store backend

Unless explicitly disabled or configured otherwise, the ZenML server will use the SQL database as a secrets store backend. If you want to use the GCP Secrets Manager instead, you need to configure it in the Helm values. Depending on where you deploy your ZenML server and how your Kubernetes cluster is configured, you may also need to provide the GCP credentials needed to access the GCP Secrets Manager API:
Unless explicitly disabled or configured otherwise, the ZenML server will use the SQL database as a secrets store backend. If you want to use the GCP Secrets Manager instead, you need to configure it in the Helm values. Depending on where you deploy your ZenML server and how your Kubernetes cluster is configured, you may also need to provide the GCP credentials needed to access the GCP Secrets Manager API.

The GCP Secrets Store uses the ZenML GCP Service Connector under the hood to authenticate with the GCP Secrets Manager API. This means that you can use any of the [authentication methods supported by the GCP Service Connector](https://docs.zenml.io/stacks-and-components/auth-management/gcp-service-connector#authentication-methods) to authenticate with the GCP Secrets Manager API:


```yaml
zenml:
Expand All @@ -367,18 +374,19 @@ Unless explicitly disabled or configured otherwise, the ZenML server will use th
# Configuration for the GCP Secrets Manager secrets store
gcp:
# The GCP project ID to use. This must be set to the project ID where the
# GCP Secrets Manager service that you want to use is located.
project_id: my-gcp-project
# The GCP Service Connector authentication method to use.
authMethod: service-account
# Path to the GCP credentials file to use to authenticate with the GCP Secrets
# Manager instance. You can omit this if you are running the ZenML server
# in a GCP GKE cluster that uses workload identity to authenticate with
# GCP services without the need for credentials.
# NOTE: the credentials file needs to be copied in the helm chart folder
# and the path configured here needs to be relative to the root of the
# helm chart.
google_application_credentials: cloud-credentials.json
# The GCP Service Connector configuration.
authConfig:
# The GCP project ID to use. This must be set to the project ID where the
# GCP Secrets Manager service that you want to use is located.
project_id: my-gcp-project
# GCP credentials JSON to use to authenticate with the GCP Secrets
# Manager instance.
google_application_credentials: '{...}'
serviceAccount:
Expand All @@ -393,7 +401,10 @@ Unless explicitly disabled or configured otherwise, the ZenML server will use th
{% tab title="Azure" %}
#### Using the Azure Key Vault as a secrets store backend

Unless explicitly disabled or configured otherwise, the ZenML server will use the SQL database as a secrets store backend. If you want to use the Azure Key Vault service instead, you need to configure it in the Helm values. Depending on where you deploy your ZenML server and how your Kubernetes cluster is configured, you may also need to provide the Azure credentials needed to access the Azure Key Vault API:
Unless explicitly disabled or configured otherwise, the ZenML server will use the SQL database as a secrets store backend. If you want to use the Azure Key Vault service instead, you need to configure it in the Helm values. Depending on where you deploy your ZenML server and how your Kubernetes cluster is configured, you may also need to provide the Azure credentials needed to access the Azure Key Vault API.

The Azure Secrets Store uses the ZenML Azure Service Connector under the hood to authenticate with the Azure Key Vault API. This means that you can use any of the [authentication methods supported by the Azure Service Connector](https://docs.zenml.io/stacks-and-components/auth-management/azure-service-connector#authentication-methods) to authenticate with the Azure Key Vault API:


```yaml
zenml:
Expand All @@ -416,13 +427,17 @@ Unless explicitly disabled or configured otherwise, the ZenML server will use th
# Key Vault instance that you want to use.
key_vault_name:
# The Azure application service principal credentials to use to
# authenticate with the Azure Key Vault API. You can omit these if you are
# running the ZenML server hosted in Azure and are using a managed
# identity to access the Azure Key Vault service.
azure_client_id: <your Azure client ID>
azure_client_secret: <your Azure client secret>
azure_tenant_id: <your Azure tenant ID>
# The Azure Service Connector authentication method to use.
authMethod: service-principal
# The Azure Service Connector configuration.
authConfig:
# The Azure application service principal credentials to use to
# authenticate with the Azure Key Vault API.
client_id: <your Azure client ID>
client_secret: <your Azure client secret>
tenant_id: <your Azure tenant ID>
```
{% endtab %}

Expand Down
10 changes: 10 additions & 0 deletions src/zenml/zen_server/deploy/helm/templates/server-db-job.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -83,18 +83,28 @@ spec:
- name: ZENML_SECRETS_STORE_TYPE
value: {{ .Values.zenml.secretsStore.type | quote }}
{{- if eq .Values.zenml.secretsStore.type "aws" }}
- name: ZENML_SECRETS_STORE_AUTH_METHOD
value: {{ .Values.zenml.secretsStore.aws.authMethod | quote }}
{{- if .Values.zenml.secretsStore.aws.region_name }}
- name: ZENML_SECRETS_STORE_REGION_NAME
value: {{ .Values.zenml.secretsStore.aws.region_name | quote }}
{{- endif }}
- name: ZENML_SECRETS_STORE_SECRET_LIST_REFRESH_TIMEOUT
value: {{ .Values.zenml.secretsStore.aws.secret_list_refresh_timeout | quote }}
{{- else if eq .Values.zenml.secretsStore.type "gcp" }}
- name: ZENML_SECRETS_STORE_AUTH_METHOD
value: {{ .Values.zenml.secretsStore.gcp.authMethod | quote }}
{{- if .Values.zenml.secretsStore.gcp.project_id }}
- name: ZENML_SECRETS_STORE_PROJECT_ID
value: {{ .Values.zenml.secretsStore.gcp.project_id | quote }}
{{- endif }}
{{- if .Values.zenml.secretsStore.gcp.google_application_credentials }}
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /gcp-credentials/credentials.json
{{- end }}
{{- else if eq .Values.zenml.secretsStore.type "azure" }}
- name: ZENML_SECRETS_STORE_AUTH_METHOD
value: {{ .Values.zenml.secretsStore.azure.authMethod | quote }}
- name: ZENML_SECRETS_STORE_KEY_VAULT_NAME
value: {{ .Values.zenml.secretsStore.azure.key_vault_name | quote }}
{{- else if eq .Values.zenml.secretsStore.type "hashicorp" }}
Expand Down
Loading

0 comments on commit fe55e50

Please sign in to comment.