Skip to content

aws-ia/terraform-aws-eks-data-addons

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

Terraform Module: πŸš€ Data & AI/ML Kubernetes Add-ons βš™οΈ

This Terraform module contains commonly used Data & AI/ML related Kubernetes add-ons that are typically included in Data on EKS blueprints. The purpose of this module is to provide users with the flexibility to select and customize the add-ons they require while leveraging the Data on EKS blueprints.

⚠️ Important Note

Users can consume this Terraform module in their projects to deploy any of the available addons. We will continue to maintain and update the existing Data/ML add-ons. However, we kindly request that you refrain from submitting Pull Requests (PRs) to add new addons at the moment, unless there is a supported blueprint available in the Data on EKS repository. The Apache and CNCF communities offer numerous open-source Data and ML add-ons, and while we appreciate their value, supporting all of them poses challenges.

Your understanding and cooperation are highly appreciated. πŸ™

Usage

Create Addon with the following example using Terraform registry. Checkout the complete example under test folder.

module "eks_data_addons" {
  source = "aws-ia/eks-data-addons/aws"
  version = "~> 1.0" # ensure to update this to the latest/desired version

  oidc_provider_arn = module.eks.oidc_provider_arn

  # Example to deploy AWS Neuron Device Plugin for Trainium and Inferentia instances
  enable_aws_efa_k8s_device_plugin = true

  # Example to deploy EFA K8s Device Plugin for GPU/Neuron instances
  enable_aws_efa_k8s_device_plugin = true

  # Example to deploy NVIDIA GPU Operator
  enable_nvidia_gpu_operator = true

  # Example to deploy Spark Operator Helm Chart
  enable_spark_opertor = true

  # Example to deploy Flink Operator Helm Chart
  enable_flink_operator = true

  # Example to deploy Apache YuniKorn Helm Chart
  enable_yunikorn = true

  # Example that uses ECR authentication for a particular registry ID
  enable_emr_spark_operator = var.enable_emr_spark_operator
  emr_spark_operator_helm_config = {
    repository_username = data.aws_ecr_authorization_token.token.user_name
    repository_password = data.aws_ecr_authorization_token.token.password
  }

  # Example to deploy Helm chart that uses IAM Role for ServiceAccounts. You can disable `create_irsa` and bring your own IAM role.
  enable_spark_history_server = var.enable_emr_spark_operator
  spark_history_server_helm_config = {
    create_irsa = true
    values = [
      templatefile("${path.module}/test/helm-values/spark-history-server-values.yaml", {
        s3_bucket_name   = module.s3_bucket.s3_bucket_id
        s3_bucket_prefix = aws_s3_object.this.key
      })
    ]
  }
}

Support & Feedback

Important

EKS Blueprints for Terraform is maintained by AWS Solution Architects. It is not part of an AWS service and support is provided as a best-effort by the EKS Blueprints community. To provide feedback, please use the issues templates provided. If you are interested in contributing to EKS Blueprints, see the Contribution guide.

Requirements

Name Version
terraform >= 1.0.0
aws >= 3.72
helm >= 2.4.1

Providers

Name Version
aws >= 3.72
helm >= 2.4.1

Modules

Name Source Version
spark_history_server_irsa ./irsa n/a

Resources

Name Type
helm_release.airflow resource
helm_release.aws_efa_k8s_device_plugin resource
helm_release.aws_neuron_device_plugin resource
helm_release.cnpg_operator resource
helm_release.dask_operator resource
helm_release.daskhub resource
helm_release.emr_flink_operator resource
helm_release.emr_spark_operator resource
helm_release.flink_operator resource
helm_release.jupyterhub resource
helm_release.karpenter_resources resource
helm_release.kubecost resource
helm_release.kuberay_operator resource
helm_release.mlflow_tracking resource
helm_release.nvidia_device_plugin resource
helm_release.nvidia_gpu_operator resource
helm_release.nvidia_triton_server resource
helm_release.pinot resource
helm_release.qdrant resource
helm_release.spark_history_server resource
helm_release.spark_operator resource
helm_release.strimzi_kafka_operator resource
helm_release.superset resource
helm_release.trino resource
helm_release.volcano resource
helm_release.yunikorn resource
aws_partition.current data source
aws_region.current data source

Inputs

Name Description Type Default Required
airflow_helm_config Airflow Helm Chart config any {} no
aws_efa_k8s_device_plugin_helm_config EFA K8s Plugin add-on Helm Chart config any {} no
aws_neuron_device_plugin_helm_config AWS Neuron Device Plugin Helm Chart config any {} no
cnpg_operator_helm_config CloudNative PG Operator Helm Chart config any {} no
dask_operator_helm_config Dask Operator add-on configurations any {} no
daskhub_helm_config DaskHub add-on configurations any {} no
emr_flink_operator_helm_config Helm configuration for Flink Operator with EMR Runtime any {} no
emr_spark_operator_helm_config Helm configuration for Spark Operator with EMR Runtime any {} no
enable_airflow Enable Airflow add-on bool false no
enable_aws_efa_k8s_device_plugin Enable EFA K8s Plugin add-on bool false no
enable_aws_neuron_device_plugin Enable AWS Neuron Device Plugin add-on bool false no
enable_cnpg_operator Enable CloudNative PG Operator add-on bool false no
enable_dask_operator Enable Dask Operator add-on bool false no
enable_daskhub Enable DaskHub bool false no
enable_emr_flink_operator Enable the Flink Operator to run Flink application with EMR Runtime bool false no
enable_emr_spark_operator Enable the Spark Operator to submit jobs with EMR Runtime bool false no
enable_flink_operator Enable Flink Operator add-on bool false no
enable_jupyterhub Enable Jupyterhub Add-On bool false no
enable_karpenter_resources Enable Karpenter Resources (NodePool and EC2NodeClass) bool false no
enable_kubecost Enable Kubecost add-on bool false no
enable_kuberay_operator Enable Kuberay Operator add-on bool false no
enable_mlflow_tracking Enable MLflow Tracking add-on bool false no
enable_nvidia_device_plugin Enable NVIDIA Device Plugin add-on bool false no
enable_nvidia_gpu_operator Enable NVIDIA GPU Operator add-on bool false no
enable_nvidia_triton_server Enable NVIDIA Triton Server add-on bool false no
enable_pinot Enable Apache Pinot Add-On bool false no
enable_qdrant Enable Qdrant Vector Database add-on bool false no
enable_spark_history_server Enable Spark History Server add-on bool false no
enable_spark_operator Enable Spark on K8s Operator add-on bool false no
enable_strimzi_kafka_operator Enable the Strimzi Kafka Operator bool false no
enable_superset Enable Apache Supeset add-on bool false no
enable_trino Enable Trino add-on bool false no
enable_volcano Enable volcano scheduler add-on bool false no
enable_yunikorn Enable Apache YuniKorn K8s scheduler add-on bool false no
flink_operator_helm_config Flink Operator Helm Chart config any {} no
jupyterhub_helm_config Helm configuration for JupyterHub any {} no
karpenter_resources_helm_config Karpenter Resources Helm Chart config any {} no
kubecost_helm_config Kubecost Helm Chart config any {} no
kuberay_operator_helm_config Helm configuration for Kuberay Operator any {} no
mlflow_tracking_helm_config MLflow Tracking add-on Helm Chart config any {} no
nvidia_device_plugin_helm_config NVIDIA Device Plugin Helm Chart config any {} no
nvidia_gpu_operator_helm_config Helm configuration for NVIDIA GPU Operator any {} no
nvidia_triton_server_helm_config Helm configuration for NVIDIA Triton Server any {} no
oidc_provider_arn The ARN of the cluster OIDC Provider string n/a yes
pinot_helm_config Apache Pinot Helm Chart config any {} no
qdrant_helm_config Helm configuration for Qdrant any {} no
spark_history_server_helm_config Helm configuration for Spark History Server any {} no
spark_operator_helm_config Helm configuration for Spark K8s Operator any {} no
strimzi_kafka_operator_helm_config Helm configuration for Strimzi Kafka Operator any {} no
superset_helm_config Apache Supeset Helm Chart config any {} no
trino_helm_config Trino Helm Chart config any {} no
volcano_helm_config Volcano scheduler add-on configurations any {} no
yunikorn_helm_config Helm configuration for Apache YuniKorn any {} no

Outputs

Name Description
airflow Airflow Helm Chart metadata
aws_efa_k8s_device_plugin AWS EFA K8s Plugin Helm Chart metadata
aws_neuron_device_plugin AWS Neuron Device Plugin Helm Chart metadata
dask_hub Dask Hub Helm Chart metadata
dask_operator Dask Operator Helm Chart metadata
emr_spark_operator EMR Spark Operator Helm Chart metadata
flink_operator Flink Operator Helm Chart metadata
jupyterhub Jupyterhub Helm Chart metadata
kubecost Kubecost Helm Chart metadata
kuberay_operator Kuberay Operator Helm Chart metadata
nvidia_gpu_operator Nvidia GPU Operator Helm Chart metadata
pinot Apache Pinot Helm Chart metadata
spark_history_server Spark History Server Helm Chart metadata
spark_operator Spark Operator Helm Chart metadata
strimzi_kafka_operator Strimzi Kafka Operator Helm Chart metadata
volcano Volcano Batch Scheduler Helm Chart metadata
yunikorn Yunikorn Helm Chart metadata