terraform-kubernetes-confluent-platform

A Terraform module for managing Confluent for Kubernetes (CFK).

Prerequisites

It is recommended that you at least install Custom Resource Definitions (CRDs) on the Kubernetes cluster before applying the Terraform.

Please see The Confluent for Kubernetes Quickstart for more information.

Override CFK Manifest Values

Similar to the values file for Helm; The variables supplied to the Confluent Platform module will be deep-merged with the default local values of the module.

Example

module "confluent_platform" {
  source  = "aidanmelen/confluent-platform/kubernetes"
  version = ">= 0.9.5"

  namespace = var.namespace

  # assumes the confluent operator was deployed in another terraform run
  confluent_operator = {
    create = false
  }

  # uncomment to override the modules default local values
  /*
  zookeeper      = yamldecode(file("${path.module}/values/zookeeper.yaml"))
  kafka          = yamldecode(file("${path.module}/values/kafka.yaml"))
  connect        = yamldecode(file("${path.module}/values/connect.yaml"))
  ksqldb         = yamldecode(file("${path.module}/values/ksqldb.yaml"))
  controlcenter  = yamldecode(file("${path.module}/values/controlcenter.yaml"))
  schemaregistry = yamldecode(file("${path.module}/values/schemaregistry.yaml"))
  kafkarestproxy = yamldecode(file("${path.module}/values/kafkarestproxy.yaml"))
  */

  create_controlcenter = var.create_controlcenter
}

Usage

Release the Confluent Platform with:

cd examples/complete
terraform init
terraform apply

Troubleshooting

Error: Failed to determine GroupVersionResource for manifest

This Terraform module uses the kubernetes_manifest resource to deploy CFK custom resources. The following excerpt is from Beta Support for CRDs in the Terraform Provider for Kubernetes:

Custom resource definitions must be applied before custom resources. As above, this is because the provider queries the Kubernetes API for the OpenAPI specification for the resource supplied in the manifest attribute. If the CRD doesn’t exist in the OpenAPI specification during plan time then Terraform can’t use it to create custom resources.

Please see Troubleshoot Confluent for Kubernetes for other troubleshooting needs.

Tests

Run Terratest using the Makefile targets:

1. make setup
2. make tests

Results

Terratest Suite (Module v0.9.5) (Terraform v1.2.6)
--- PASS: TestTerraformCompleteExample (175.48s)
--- PASS: TestTerraformConfluentOperatorExample (22.24s)
--- PASS: TestTerraformConfluentPlatformExample (236.10s)
--- PASS: TestTerraformConfluentPlatformSinglenodeExample (232.32s)
--- PASS: TestTerraformConnectorExample (222.19s)
--- PASS: TestTerraformKafkaTopicsExample (168.37s)
--- PASS: TestTerraformSchemaExample (336.87s)

Makefile Targets

help                                This help.
build                               Build docker dev container
run                                 Run docker dev container
setup                               Setup project
lint                                Lint with pre-commit
lint-all                            Lint all files with pre-commit
tests                               Tests with Terratest
test-confluent-operator             Test the confluent_operator example
test-confluent-platform             Test the confluent_platform example
test-confluent-platform-singlenode  Test the confluent_platform_singlenode example
test-complete                       Test the complete example
test-kafka-topic                    Test the kafka_topic example
test-schema                         Test the schema example
test-connector                      Test the connector example
release                             Tag remote triggering Terraform Registry release
clean                               Clean project

Requirements

Name	Version
terraform	>= 0.14.8
helm	>= 2.0.0
kubernetes	>= 2.12.1

Providers

Name	Version
kubernetes	>= 2.12.1

Modules

Name	Source	Version
confluent_operator	./modules/confluent_operator	n/a
confluent_platform_override_values	Invicton-Labs/deepmerge/null	0.1.5
confluent_role_bindings	./modules/confluent_role_binding	n/a
connectors	./modules/connector	n/a
kafka_rest_classes	./modules/kafka_rest_class	n/a
kafka_topics	./modules/kafka_topic	n/a
schemas	./modules/schema	n/a

Resources

Name	Type
kubernetes_manifest.components	resource
kubernetes_resource.components	data source

Inputs

Name	Description	Type	Default	Required
confluent_operator	Controls if the Confluent Operator resources should be created. This is required when the Confluent Operator is not already running on the kubernetes cluster.	any	{ "create": true }	no
confluent_operator_app_version	The default Confluent Operator app version. This may be overriden by component override values. This version must be compatible with the confluent_platform_version. Please see confluent docs for more information: https://docs.confluent.io/platform/current/installation/versions-interoperability.html#confluent-operator	string	"2.4.0"	no
confluent_platform_version	The default Confluent Platform app version. This may be overriden by component override values. This version must be compatible with the confluent_operator_app_version. Please see confluent docs for more information: https://docs.confluent.io/platform/current/installation/versions-interoperability.html#confluent-operator	string	"7.2.0"	no
confluent_role_bindings	A map of Confluent Role Bindings to create. The key is the confluent role binding name. The value map is the input for the confluent_role_binding submodule.	any	{}	no
connect	The Connect override values.	any	{}	no
connectors	A map of Connectors to create. The key is the connector name. The value map is the input for the connector submodule.	any	{}	no
controlcenter	The ControlCenter override values.	any	{}	no
create	Controls if the Confluent Platform and Operator resources should be created (affects all resources).	bool	true	no
create_connect	Controls if the Connect component of the Confluent Platform should be created.	bool	true	no
create_controlcenter	Controls if the ControlCenter component of the Confluent Platform should be created.	bool	true	no
create_kafka	Controls if the Kafka component of the Confluent Platform should be created.	bool	true	no
create_kafkarestproxy	Controls if the KafkaRestProxy component of the Confluent Platform should be created.	bool	true	no
create_ksqldb	Controls if the KsqlDB component of the Confluent Platform should be created.	bool	true	no
create_schemaregistry	Controls if the SchemaRegistry component of the Confluent Platform should be created.	bool	true	no
create_timeout	The create timeout for each Confluent Platform component.	string	"1h"	no
create_zookeeper	Controls if the Zookeeper component of the Confluent Platform should be created.	bool	true	no
delete_timeout	The delete timeout for each Confluent Platform component.	string	"10m"	no
kafka	The Kafka override values.	any	{}	no
kafka_rest_classes	A map of Kafka Rest Classes to create. The key is the kafka rest class name. The value map is the input for the kafka_rest_class submodule.	any	{}	no
kafka_topics	A map of Kafka Topics to create. The key is the topic name. The value map is the input for the kafka_topic submodule.	any	{}	no
kafkarestproxy	The KafkaRestProxy override values.	any	{}	no
ksqldb	The KsqlDB override values.	any	{}	no
namespace	The namespace to release Confluent Platform into. When confluent_operator is specified, this will also ensure the Confluent Operator is released into the same namespace.	string	"confluent"	no
schemaregistry	The SchemaRegistry override values.	any	{}	no
schemas	A map of Schemas to create. The key is the schema name. The value map is the input for the schema submodule.	any	{}	no
update_timeout	The update timeout for each Confluent Platform component.	string	"1h"	no
zookeeper	The Zookeeper override values.	any	{}	no

Outputs

Name	Description
confluent_operator	Map of attributes for the Confluent Operator.
confluent_role_binding_manifests	Map of attribute maps for all the ConfluentRoleBinding manifests created.
confluent_role_binding_objects	Map of attribute maps for all the ConfluentRoleBinding objects created.
confluent_role_bindings	Map of attribute maps for all ConfluentRoleBinding submodules created.
connect_manifest	The Connect manifest.
connect_object	The Connect object.
connector_manifests	Map of attribute maps for all the Connector manifests created.
connector_objects	Map of attribute maps for all the Connector objects created.
connectors	Map of attribute maps for all Connector submodules created.
controlcenter_manifest	The ControlCenter manifest.
controlcenter_object	The ControlCenter object.
kafka_manifest	The Kafka manifest.
kafka_object	The Kafka object.
kafka_rest_class_manifests	Map of attribute maps for all the KafkaRestClass manifests created.
kafka_rest_class_objects	Map of attribute maps for all the KafkaRestClass objects created.
kafka_rest_classes	Map of attribute maps for all KafkaRestClass submodules created.
kafka_topic_manifests	Map of attribute maps for all the KafkaTopic manifests created.
kafka_topic_objects	Map of attribute maps for all the KafkaTopic objects created.
kafka_topics	Map of attribute maps for all KafkaTopic submodules created.
kafkarestproxy_manifest	The KafkaRestProxy manifest.
kafkarestproxy_object	The KafkaRestProxy object.
ksqldb_manifest	The KsqlDB manifest.
ksqldb_object	The KsqlDB object.
namespace	The default namespace for the Confluent Platform.
schema_config_map	Map of attribute maps for all the Schema ConfigMap created.
schema_manifests	Map of attribute maps for all the Schema manifests created.
schema_objects	Map of attribute maps for all the Schema objects created.
schemaregistry_manifest	The SchemaRegistry manifest.
schemaregistry_object	The SchemaRegistry object.
schemas	Map of attribute maps for all Schema submodules created.
zookeeper_manifest	The Zookeeper manifest.
zookeeper_object	The Zookeeper object.

License

Apache 2 Licensed. See LICENSE for full details.