Skip to content

Commit

Permalink
Feature: use service connectors to authenticate secrets stores. (#2154)
Browse files Browse the repository at this point in the history 
* Add service connector support to AWS secrets store

* Add GCP and Azure service connectors support to secrets stores

* Add helm chart support and update docs

* Tested all secrets stores and fixed remaining bugs

* Fix code repository created and updated fields

* Apply suggestions from code review

Co-authored-by: Jayesh Sharma <wjayesh@outlook.com>

* Auto-update of E2E template

---------

Co-authored-by: Jayesh Sharma <wjayesh@outlook.com>
Co-authored-by: GitHub Actions <actions@github.com>
  • Loading branch information
@stefannica @wjayesh @actions-user
3 people authored Dec 19, 2023
1 parent 74a1329 commit 3ba010d
Show file tree
Hide file tree
Showing 26 changed files with 798 additions and 248 deletions.
37 changes: 30 additions & 7 deletions docs/book/deploying-zenml/zenml-self-hosted/deploy-with-docker.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -75,17 +75,32 @@ The SQL database is used as the default secret store. You only need to configure
These configuration options are only relevant if you're using the AWS Secrets Manager as the secrets store backend.
* **ZENML\_SECRETS\_STORE\_TYPE:** Set this to `aws` in order to set this type of secret store.
* **ZENML\_SECRETS\_STORE\_REGION\_NAME**: The AWS region to use. This must be set to the region where the AWS Secrets Manager service that you want to use is located.
* **ZENML\_SECRETS\_STORE\_AWS\_ACCESS\_KEY\_ID**: The AWS access key ID to use for authentication. This must be set to a valid AWS access key ID that has access to the AWS Secrets Manager service that you want to use. If you are using an IAM role attached to an EKS cluster to authenticate, you can omit this variable. NOTE: this is the same as setting the `AWS_ACCESS_KEY_ID` environment variable.
* **ZENML\_SECRETS\_STORE\_AWS\_SECRET\_ACCESS\_KEY**: The AWS secret access key to use for authentication. This must be set to a valid AWS secret access key that has access to the AWS Secrets Manager service that you want to use. If you are using an IAM role attached to an EKS cluster to authenticate, you can omit this variable. NOTE: this is the same as setting the `AWS_SECRET_ACCESS_KEY` environment variable.
* **ZENML\_SECRETS\_STORE\_AWS\_SESSION\_TOKEN**: Optional AWS session token to use for authentication. NOTE: this is the same as setting the `AWS_SESSION_TOKEN` environment variable.
The AWS Secrets Store uses the ZenML AWS Service Connector under the hood to authenticate with the AWS Secrets Manager API. This means that you can use any of the [authentication methods supported by the AWS Service Connector](https://docs.zenml.io/stacks-and-components/auth-management/aws-service-connector#authentication-methods) to authenticate with the AWS Secrets Manager API. The following configuration options are supported:
* **ZENML\_SECRETS\_STORE\_AUTH\_METHOD**: The AWS Service Connector authentication method to use (e.g. `secret-key` or `iam-role`).
* **ZENML\_SECRETS\_STORE\_AUTH\_CONFIG**: The AWS Service Connector configuration, in JSON format (e.g. `{"role_arn": "arn:aws:iam::123456789012:role/MyRole"}`).
* **ZENML\_SECRETS\_STORE\_SECRET\_LIST\_REFRESH\_TIMEOUT**: AWS' [Secrets Manager](https://aws.amazon.com/secrets-manager) has a known issue where it does not immediately reflect new and updated secrets in the `list_secrets` results. To work around this issue, you can set this refresh timeout value to a non-zero value to get the ZenML server to wait after creating or updating an AWS secret until the changes are reflected in the secrets returned by `list_secrets` or the number of seconds specified by this value has elapsed. Defaults to `0` (disabled). Should not be set to a high value as it may cause thread starvation in the ZenML server on high load.
> **Note:** The remaining configuration options are deprecated and may be removed in a future release. Instead, you should set the `ZENML_SECRETS_STORE_AUTH_METHOD` and `ZENML_SECRETS_STORE_AUTH_CONFIG` variables to use the AWS Service Connector authentication method.
* **ZENML\_SECRETS\_STORE\_REGION\_NAME**: The AWS region to use. This must be set to the region where the AWS Secrets Manager service that you want to use is located.
* **ZENML\_SECRETS\_STORE\_AWS\_ACCESS\_KEY\_ID**: The AWS access key ID to use for authentication. This must be set to a valid AWS access key ID that has access to the AWS Secrets Manager service that you want to use. If you are using an IAM role attached to an EKS cluster to authenticate, you can omit this variable.
* **ZENML\_SECRETS\_STORE\_AWS\_SECRET\_ACCESS\_KEY**: The AWS secret access key to use for authentication. This must be set to a valid AWS secret access key that has access to the AWS Secrets Manager service that you want to use. If you are using an IAM role attached to an EKS cluster to authenticate, you can omit this variable.
{% endtab %}
{% tab title="GCP" %}
These configuration options are only relevant if you're using the GCP Secrets Manager as the secrets store backend.
* **ZENML\_SECRETS\_STORE\_TYPE:** Set this to `gcp` in order to set this type of secret store.
The GCP Secrets Store uses the ZenML GCP Service Connector under the hood to authenticate with the GCP Secrets Manager API. This means that you can use any of the [authentication methods supported by the GCP Service Connector](https://docs.zenml.io/stacks-and-components/auth-management/gcp-service-connector#authentication-methods) to authenticate with the GCP Secrets Manager API. The following configuration options are supported:
* **ZENML\_SECRETS\_STORE\_AUTH\_METHOD**: The GCP Service Connector authentication method to use (e.g. `service-account`).
* **ZENML\_SECRETS\_STORE\_AUTH\_CONFIG**: The GCP Service Connector configuration, in JSON format (e.g. `{"project_id": "my-project", "service_account_json": { ... }}`).
> **Note:** The remaining configuration options are deprecated and may be removed in a future release. Instead, you should set the `ZENML_SECRETS_STORE_AUTH_METHOD` and `ZENML_SECRETS_STORE_AUTH_CONFIG` variables to use the GCP Service Connector authentication method.
* **ZENML\_SECRETS\_STORE\_PROJECT\_ID**: The GCP project ID to use. This must be set to the project ID where the GCP Secrets Manager service that you want to use is located.
* **GOOGLE\_APPLICATION\_CREDENTIALS**: The path to the GCP service account credentials file to use for authentication. This must be set to a valid GCP service account credentials file that has access to the GCP Secrets Manager service that you want to use. If you are using a GCP service account attached to a GKE cluster to authenticate, you can omit this variable. NOTE: the path to the credentials file must be mounted into the container.
{% endtab %}
Expand All @@ -95,9 +110,17 @@ These configuration options are only relevant if you're using Azure Key Vault as
* **ZENML\_SECRETS\_STORE\_TYPE:** Set this to `azure` in order to set this type of secret store.
* **ZENML\_SECRETS\_STORE\_KEY\_VAULT\_NAME**: The name of the Azure Key Vault. This must be set to point to the Azure Key Vault instance that you want to use.
* **ZENML\_SECRETS\_STORE\_AZURE\_CLIENT\_ID**: The Azure application service principal client ID to use to authenticate with the Azure Key Vault API. If you are running the ZenML server hosted in Azure and are using a managed identity to access the Azure Key Vault service, you can omit this variable. NOTE: this is the same as setting the `AZURE_CLIENT_ID` environment variable.
* **ZENML\_SECRETS\_STORE\_AZURE\_CLIENT\_SECRET**: The Azure application service principal client secret to use to authenticate with the Azure Key Vault API. If you are running the ZenML server hosted in Azure and are using a managed identity to access the Azure Key Vault service, you can omit this variable. NOTE: this is the same as setting the `AZURE_CLIENT_SECRET` environment variable.
* **ZENML\_SECRETS\_STORE\_AZURE\_TENANT\_ID**: The Azure application service principal tenant ID to use to authenticate with the Azure Key Vault API. If you are running the ZenML server hosted in Azure and are using a managed identity to access the Azure Key Vault service, you can omit this variable. NOTE: this is the same as setting the `AZURE_TENANT_ID` environment variable.
The Azure Secrets Store uses the ZenML Azure Service Connector under the hood to authenticate with the Azure Key Vault API. This means that you can use any of the [authentication methods supported by the Azure Service Connector](https://docs.zenml.io/stacks-and-components/auth-management/azure-service-connector#authentication-methods) to authenticate with the Azure Key Vault API. The following configuration options are supported:
* **ZENML\_SECRETS\_STORE\_AUTH\_METHOD**: The Azure Service Connector authentication method to use (e.g. `service-account`).
* **ZENML\_SECRETS\_STORE\_AUTH\_CONFIG**: The Azure Service Connector configuration, in JSON format (e.g. `{"tenant_id": "my-tenant-id", "client_id": "my-client-id", "client_secret": "my-client-secret"}`).
> **Note:** The remaining configuration options are deprecated and may be removed in a future release. Instead, you should set the `ZENML_SECRETS_STORE_AUTH_METHOD` and `ZENML_SECRETS_STORE_AUTH_CONFIG` variables to use the Azure Service Connector authentication method.
* **ZENML\_SECRETS\_STORE\_AZURE\_CLIENT\_ID**: The Azure application service principal client ID to use to authenticate with the Azure Key Vault API. If you are running the ZenML server hosted in Azure and are using a managed identity to access the Azure Key Vault service, you can omit this variable.
* **ZENML\_SECRETS\_STORE\_AZURE\_CLIENT\_SECRET**: The Azure application service principal client secret to use to authenticate with the Azure Key Vault API. If you are running the ZenML server hosted in Azure and are using a managed identity to access the Azure Key Vault service, you can omit this variable.
* **ZENML\_SECRETS\_STORE\_AZURE\_TENANT\_ID**: The Azure application service principal tenant ID to use to authenticate with the Azure Key Vault API. If you are running the ZenML server hosted in Azure and are using a managed identity to access the Azure Key Vault service, you can omit this variable.
{% endtab %}
{% tab title="Hashicorp" %}
Expand Down
93 changes: 60 additions & 33 deletions docs/book/deploying-zenml/zenml-self-hosted/deploy-with-helm.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -312,7 +312,10 @@ This method requires you to configure a DNS service like AWS Route 53 or Google
{% tab title="AWS" %}
#### Using the AWS Secrets Manager as a secrets store backend

Unless explicitly disabled or configured otherwise, the ZenML server will use the SQL database as a secrets store backend. If you want to use the AWS Secrets Manager instead, you need to configure it in the Helm values. Depending on where you deploy your ZenML server and how your Kubernetes cluster is configured, you may also need to provide the AWS credentials needed to access the AWS Secrets Manager API:
Unless explicitly disabled or configured otherwise, the ZenML server will use the SQL database as a secrets store backend. If you want to use the AWS Secrets Manager instead, you need to configure it in the Helm values. Depending on where you deploy your ZenML server and how your Kubernetes cluster is configured, you may also need to provide the AWS credentials needed to access the AWS Secrets Manager API.

The AWS Secrets Store uses the ZenML AWS Service Connector under the hood to authenticate with the AWS Secrets Manager API. This means that you can use any of the [authentication methods supported by the AWS Service Connector](https://docs.zenml.io/stacks-and-components/auth-management/aws-service-connector#authentication-methods) to authenticate with the AWS Secrets Manager API:


```yaml
zenml:
Expand All @@ -331,24 +334,28 @@ Unless explicitly disabled or configured otherwise, the ZenML server will use th
# Configuration for the AWS Secrets Manager secrets store
aws:
# The AWS region to use. This must be set to the region where the AWS
# Secrets Manager service that you want to use is located.
region_name: us-east-1
# The AWS credentials to use to authenticate with the AWS Secrets
# Manager instance. You can omit these if you are running the ZenML server
# in an AWS EKS cluster that has an IAM role attached to it that has
# permissions to access the AWS Secrets Manager instance.
aws_access_key_id: <your AWS access key ID>
aws_secret_access_key: <your AWS secret access key>
aws_session_token: <your AWS session token>
# The AWS Service Connector authentication method to use.
authMethod: secret-key
# The AWS Service Connector configuration.
authConfig:
# The AWS region to use. This must be set to the region where the AWS
# Secrets Manager service that you want to use is located.
region: us-east-1
# The AWS credentials to use to authenticate with the AWS Secrets
aws_access_key_id: <your AWS access key ID>
aws_secret_access_key: <your AWS secret access key>
```
{% endtab %}

{% tab title="GCP" %}
#### Using the GCP Secrets Manager as a secrets store backend

Unless explicitly disabled or configured otherwise, the ZenML server will use the SQL database as a secrets store backend. If you want to use the GCP Secrets Manager instead, you need to configure it in the Helm values. Depending on where you deploy your ZenML server and how your Kubernetes cluster is configured, you may also need to provide the GCP credentials needed to access the GCP Secrets Manager API:
Unless explicitly disabled or configured otherwise, the ZenML server will use the SQL database as a secrets store backend. If you want to use the GCP Secrets Manager instead, you need to configure it in the Helm values. Depending on where you deploy your ZenML server and how your Kubernetes cluster is configured, you may also need to provide the GCP credentials needed to access the GCP Secrets Manager API.

The GCP Secrets Store uses the ZenML GCP Service Connector under the hood to authenticate with the GCP Secrets Manager API. This means that you can use any of the [authentication methods supported by the GCP Service Connector](https://docs.zenml.io/stacks-and-components/auth-management/gcp-service-connector#authentication-methods) to authenticate with the GCP Secrets Manager API:


```yaml
zenml:
Expand All @@ -367,18 +374,31 @@ Unless explicitly disabled or configured otherwise, the ZenML server will use th
# Configuration for the GCP Secrets Manager secrets store
gcp:
# The GCP project ID to use. This must be set to the project ID where the
# GCP Secrets Manager service that you want to use is located.
project_id: my-gcp-project
# Path to the GCP credentials file to use to authenticate with the GCP Secrets
# Manager instance. You can omit this if you are running the ZenML server
# in a GCP GKE cluster that uses workload identity to authenticate with
# GCP services without the need for credentials.
# NOTE: the credentials file needs to be copied in the helm chart folder
# and the path configured here needs to be relative to the root of the
# helm chart.
google_application_credentials: cloud-credentials.json
# The GCP Service Connector authentication method to use.
authMethod: service-account
# The GCP Service Connector configuration.
authConfig:
# The GCP project ID to use. This must be set to the project ID where the
# GCP Secrets Manager service that you want to use is located.
project_id: my-gcp-project
# GCP credentials JSON to use to authenticate with the GCP Secrets
# Manager instance.
google_application_credentials: |
{
"type": "service_account",
"project_id": "my-project",
"private_key_id": "...",
"private_key": "-----BEGIN PRIVATE KEY-----\n...=\n-----END PRIVATE KEY-----\n",
"client_email": "...",
"client_id": "...",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "..."
}
serviceAccount:
Expand All @@ -393,7 +413,10 @@ Unless explicitly disabled or configured otherwise, the ZenML server will use th
{% tab title="Azure" %}
#### Using the Azure Key Vault as a secrets store backend

Unless explicitly disabled or configured otherwise, the ZenML server will use the SQL database as a secrets store backend. If you want to use the Azure Key Vault service instead, you need to configure it in the Helm values. Depending on where you deploy your ZenML server and how your Kubernetes cluster is configured, you may also need to provide the Azure credentials needed to access the Azure Key Vault API:
Unless explicitly disabled or configured otherwise, the ZenML server will use the SQL database as a secrets store backend. If you want to use the Azure Key Vault service instead, you need to configure it in the Helm values. Depending on where you deploy your ZenML server and how your Kubernetes cluster is configured, you may also need to provide the Azure credentials needed to access the Azure Key Vault API.

The Azure Secrets Store uses the ZenML Azure Service Connector under the hood to authenticate with the Azure Key Vault API. This means that you can use any of the [authentication methods supported by the Azure Service Connector](https://docs.zenml.io/stacks-and-components/auth-management/azure-service-connector#authentication-methods) to authenticate with the Azure Key Vault API:


```yaml
zenml:
Expand All @@ -416,13 +439,17 @@ Unless explicitly disabled or configured otherwise, the ZenML server will use th
# Key Vault instance that you want to use.
key_vault_name:
# The Azure application service principal credentials to use to
# authenticate with the Azure Key Vault API. You can omit these if you are
# running the ZenML server hosted in Azure and are using a managed
# identity to access the Azure Key Vault service.
azure_client_id: <your Azure client ID>
azure_client_secret: <your Azure client secret>
azure_tenant_id: <your Azure tenant ID>
# The Azure Service Connector authentication method to use.
authMethod: service-principal
# The Azure Service Connector configuration.
authConfig:
# The Azure application service principal credentials to use to
# authenticate with the Azure Key Vault API.
client_id: <your Azure client ID>
client_secret: <your Azure client secret>
tenant_id: <your Azure tenant ID>
```
{% endtab %}

Expand Down
4 changes: 4 additions & 0 deletions src/zenml/integrations/aws/__init__.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,10 @@
AWS_SAGEMAKER_STEP_OPERATOR_FLAVOR = "sagemaker"
AWS_SAGEMAKER_ORCHESTRATOR_FLAVOR = "sagemaker"

# Service connector constants
AWS_CONNECTOR_TYPE = "aws"
AWS_RESOURCE_TYPE = "aws-generic"
S3_RESOURCE_TYPE = "s3-bucket"

class AWSIntegration(Integration):
"""Definition of AWS integration for ZenML."""
Expand Down
10 changes: 7 additions & 3 deletions src/zenml/integrations/aws/flavors/aws_container_registry_flavor.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,15 @@

from pydantic import validator

from zenml.constants import DOCKER_REGISTRY_RESOURCE_TYPE
from zenml.container_registries.base_container_registry import (
BaseContainerRegistryConfig,
BaseContainerRegistryFlavor,
)
from zenml.integrations.aws import AWS_CONTAINER_REGISTRY_FLAVOR
from zenml.integrations.aws import (
AWS_CONNECTOR_TYPE,
AWS_CONTAINER_REGISTRY_FLAVOR,
)
from zenml.models import ServiceConnectorRequirements

if TYPE_CHECKING:
Expand Down Expand Up @@ -81,8 +85,8 @@ def service_connector_requirements(
connector is required for this flavor.
"""
return ServiceConnectorRequirements(
connector_type="aws",
resource_type="docker-registry",
connector_type=AWS_CONNECTOR_TYPE,
resource_type=DOCKER_REGISTRY_RESOURCE_TYPE,
resource_id_attr="uri",
)

Expand Down
7 changes: 5 additions & 2 deletions src/zenml/integrations/aws/flavors/sagemaker_orchestrator_flavor.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,10 @@
from typing import TYPE_CHECKING, Any, Dict, Optional, Type, Union

from zenml.config.base_settings import BaseSettings
from zenml.integrations.aws import AWS_SAGEMAKER_STEP_OPERATOR_FLAVOR
from zenml.integrations.aws import (
AWS_RESOURCE_TYPE,
AWS_SAGEMAKER_STEP_OPERATOR_FLAVOR,
)
from zenml.models import ServiceConnectorRequirements
from zenml.orchestrators import BaseOrchestratorConfig
from zenml.orchestrators.base_orchestrator import BaseOrchestratorFlavor
Expand Down Expand Up @@ -167,7 +170,7 @@ def service_connector_requirements(
Requirements for compatible service connectors, if a service
connector is required for this flavor.
"""
return ServiceConnectorRequirements(resource_type="aws-generic")
return ServiceConnectorRequirements(resource_type=AWS_RESOURCE_TYPE)

@property
def docs_url(self) -> Optional[str]:
Expand Down
Loading

0 comments on commit 3ba010d

Please sign in to comment.