terraform-module-k3s

Terraform module which creates a k3s cluster, with multi-server and annotations/labels/taints management features.

⚠️ This module only works with k3s version greater than v1.0.0
⚠️ This module use experimental features of Terraform (validation)

Usage

module "k3s" {
  source  = "xunleii/k3s/module"

  k3s_version = "v1.0.0"

  name = "my.k3s.local"
  cidr = {
    pods = "10.0.0.0/16"
    services = "10.1.0.0/16"
  }
  drain_timeout = "30s"
  manage_fields = ["label", "taint"]  

  global_flags = [
    "--flannel-backend=none"
  ]

  additional_flags = {
    server = [
        "--flannel-backend=none",
        "--tls-san k3s.my.domain.com"
    ]
    agent = [
      "--flannel-backend=none",
    ]
  }
  
  servers = {
    # The node name will be automatically provided by
    # the module using the field name... any usage of
    # --node-name in additional_flags will be ignored
    server_one = {
      ip = "10.123.45.67" // internal node IP
      connection = {
        host = "203.123.45.67" // public node IP
        user = "ubuntu"
      }
      flags = ["--tls-san k3s.my.domain.com"]

      labels = {"node.kubernetes.io/type" = "master"}
      taints = {"node.k3s.io/type" = "server:NoSchedule"}
    }
    server_two: {
      ip = "10.123.45.68"
      connection = {
        host = "203.123.45.68" // bastion node
        user = "ubuntu"
      }
      flags = ["--tls-san k3s.my.domain.com"]

      labels = {"node.kubernetes.io/type" = "master"}
      taints = {"node.k3s.io/type" = "server:NoSchedule"}
    }
    server_three: {
      ip = "10.123.45.69"
      connection = {
        host = "203.123.45.69" // bastion node
        user = "ubuntu"
      }
      flags = ["--tls-san k3s.my.domain.com"]

      labels = {"node.kubernetes.io/type" = "master"}
      taints = {"node.k3s.io/type" = "server:NoSchedule"}
    }
  }

  agents = {
      # The node name will be automatically provided by
      # the module using the field name... any usage of
      # --node-name in additional_flags will be ignored
      agent_one = {
          ip = "10.123.45.70"
          connection = {
              user = "root"
              bastion_host = "203.123.45.67" // server_one node used as bastion
              bastion_user = "ubuntu"
          }

          labels = {"node.kubernetes.io/pool" = "service-pool"}
      },
      agent_two = {
          ip = "10.123.45.71"
          connection = {
              user = "root"
              bastion_host = "203.123.45.67"
              bastion_user = "ubuntu"
          }

          labels = {"node.kubernetes.io/pool" = "service-pool"}
      },
      agent_three = {
          name = "gpu-agent-one"
          ip = "10.123.45.72"
          connection = {
              user = "root"
              bastion_host = "203.123.45.67"
              bastion_user = "ubuntu"
          }

          labels = {"node.kubernetes.io/pool" = "gpu-pool"}
          taints = {dedicated = "gpu:NoSchedule"}
      },
  }
}

Inputs

Name	Description	Type	Default	Required
depends_on_	Like resource.depends_on, but with only one target			false
k3s_version	k3s version to be use	string	"latest"	false
name	k3s cluster name	string	"cluster.local"	false
cidr	k3s CIDR definitions	object		false
cidr.pods	Network CIDR to use for pod IPs (--cluster-cidr)	string (ip)	"10.42.0.0/16"	false
cidr.service	Network CIDR to use for services IPs (--service-cidr)	string (ip)	"10.43.0.0/16"	false
drain_timeout	Length of time to wait before giving up the node draining	string	"0s" (infinite)	false
managed_fields	List of fields which must be managed by this module (can be annotation, label and/or taint)	list(string)	["annotation", "label", "taint"]	false
global_flags	Additional installation flags used by all nodes	list(string)	[]	false
servers	k3s server nodes definition	map(NodeType)		true
agents	k3s server nodes definition	map(NodeType)	{}	false

NOTES:
   servers must have an odd number of nodes
   use the first server node to configure the cluster

Node Type

Name	Description	Type	Default	Required
name	Node name	string	""	false
ip	Node IP (used by others node to communicate)	string (ip)		true
connection	Connection block used by null_resource to provision the node	Connection block		false
flags	Installation flags to be used on this node	list(string)	[]	false
annotations	Map of annotations to be applied on this node ({<name>: <value>})	map(string)	{}	false
labels	Map of labels to be applied on this node ({<name>: <value>})	map(string)	{}	false
taints	Map of taints to be applied on this node ({<name>: <value>})	map(string)	{}	false

NOTES:
   if name is not specified, the key in the map will be used as name
   only one taint can be applied per taint name and per node

Outputs

Name	Description	Type
summary	A summary of the current cluster state (version, server and agent list with all annotations, labels, ...)	string

More information

Security warning

Because using external references on destroy provisionner is deprecated by Terraform, storing information inside each resources will be mandatory in order to manage several features like auto-draining node and fields management. So, several fields like connection block will be available in your TF state. This means that used password or private key will be clearly readable in this TF state.
Please do not use this module if you need to pass private key or password in the connection block, even if your TF state is securely stored.

Kubeconfig

This module is not in charge of generating a Kubeconfig, mainly because Terraform doesn't allow us to get file remotely. You need to get it manually (with external data for example).

Example:

resource null_resource kubeconfig {
  provisioner "local-exec" {
    command = "scp ubuntu@203.123.45.67:/etc/rancher/k3s/k3s.yaml kubeconfig"
  }
}

License

terraform-module-k3s is released under the MIT License. See the bundled LICENSE file for details.