This terraform module can be used to deploy IBM Cloud Private on any supported infrastructure vendor. Tested on Ubuntu 16.04 and RHEL 7 on SoftLayer, VMware, AWS and Azure.
If the default SSH user is not the root user, the default user must have password-less sudo access.
| Variable | Default | Required | Description |
|---|---|---|---|
| Cluster settings | |||
| icp-inception | No* | Version of ICP to provision. Not required when See below for details on using private registry | |
| icp-master | No* | IP address of ICP Masters. Required if you don't use icp-host-groups | |
| icp-worker | No* | IP addresses of ICP Worker nodes. Required if you don't use icp-host-groups | |
| icp-proxy | No* | IP addresses of ICP Proxy nodes. Required if you don't use icp-host-groups | |
| icp-management | No | IP addresses of ICP Management Nodes, if management is to be separated from master nodes. Optional | |
| icp-host-groups | No* | Map of host types and IPs. See below for details. | |
| boot-node | No* | IP Address of boot node. Needed when using icp-host-groups or when using a boot node separate from first master node. If separate it must be included in cluster_size |
|
| cluster_size | Yes | Define total clustersize. Workaround for terraform issue #10857. Normally computed | |
| ICP Configuration | |||
| icp_config_file | No | Yaml configuration file for ICP installation. | |
| icp_configuration | No | Configuration items for ICP installation. See KnowledgeCenter for reference. Note: Boolean values (true/false) must be supplied as strings | |
| config_strategy | merge | No | Strategy for original config.yaml shipped with ICP. Default is merge, everything else means override. |
| ICP Boot node to cluster communication | |||
| generate_key | True | No | Whether to generate a new ssh key for use by ICP Boot Master to communicate with other nodes |
| icp_pub_key | No | Public ssh key for ICP Boot master to connect to ICP Cluster. Only use when generate_key = false | |
| icp_priv_key | No | Private ssh key for ICP Boot master to connect to ICP Cluster. Only use when generate_key = false | |
| Terraform installation process | |||
| hooks | No | Hooks into different stages in the cluster setup process. See below for details | |
| local-hooks | No | Locally run hooks at different stages in the cluster setup process. See below for details | |
| on_hook_failure | fail | Behavior when hooks fail. Anything other than fail will continue |
|
| install-verbosity | No | Verbosity of the icp ansible installer. -v to -vvvv. See ansible documentation for verbosity information | |
| install-command | install | No | Installer command to run |
| cluster-directory | /opt/ibm/cluster | No | Location to use for the cluster directory |
| cluster_dir_owner | No | Username to own cluster directory after an install. Defaults to ssh_user |
|
| Terraform to cluster ssh configuration | |||
| ssh_user | root | No | Username for Terraform to ssh into the ICP cluster. This is typically the default user with for the relevant cloud vendor |
| ssh_key_base64 | No | base64 encoded content of private ssh key | |
| ssh_agent | True | No | Enable or disable SSH Agent. Can correct some connectivity issues. Default: true (enabled) |
| bastion_host | No | Specify hostname or IP to connect to nodes through a SSH bastion host. Assumes same SSH key and username as cluster nodes | |
| Docker and ICP Enterprise Edition Image configuration | |||
| docker_package_location | No | http or nfs location of docker installer which ships with ICP. Typically used for RHEL which does not support docker-ce | |
| docker_image_name | docker-ce | No | Name of docker image to install when installing from Ubuntu or Centos repository |
| docker_version | latest | No | Version of docker image to install from Ubuntu or Centos repository. i.e. 18.06.1 latest install latest version. |
| image_location | No | Location of image file. Start with nfs: or http to indicate protocol to download with. Starting with / indicates local file. See example below |
|
| image_locations | No | List of images in same format as image_location. Typically used for multi-arch deployments |
|
| image_location_user | No | Username if authentication required for image_location | |
| image_location_pass | No | Password if authentication required for image_location |
- icp_public_key
- The public key used for boot master to connect via ssh for cluster setup
- icp_private_key
- The public key used for boot master to connect via ssh for cluster setup
- install_complete
- Boolean value that is set to true when ICP installation process is completed
- icp_version
- The ICP version that has been installed
- cluster_ips
- List of IPs of the cluster
The icp-inception field supports the format org/repo:version. ibmcom is the default organisation and icp-inception is the default repo, so if you're installing for example version 2.1.0.2 from Docker Hub it's sufficient to specify 2.1.0.2 as the version number.
It is also supported to install from private docker registries.
In this case the format is:
username:password@private_registry_server/org/repo:version.
So for exmaple
myuser:SomeLongPassword@myprivateregistry.local/ibmcom/icp-inception:2.1.0.2
It is possible to execute arbitrary commands between various phases of the cluster setup and installation process. Currently, the following hooks are defined. Each hook must be a list of commands to run.
| Hook name | Where executed | When executed |
|---|---|---|
| cluster-preconfig | all nodes | Before any of the module scripts |
| cluster-postconfig | all nodes | After prerequisites are installed |
| boot-preconfig | boot master | Before any module scripts on boot master |
| preinstall | boot master | After configuration image load and configuration generation |
| postinstall | boot master | After successful ICP installation |
It is possible to execute arbitrary commands between various phases of the cluster setup and installation process. Currently, the following hooks are defined. Each hook must be a single command to run.
| Hook name | When executed |
|---|---|
| local-preinstall | After configuration and preinstall remote hook |
| local-postinstall | After successful ICP installation |
These hooks support the execution of a single command or a local script. While this is a local-exec command, passing additional interpreter/environment parameters are not supported and therefore everything will be treated as a BASH script.
In ICP version 2.1.0.2 the concept of host groups were introduced. This allows users to define groups of hosts by an arbritrary name that will be labelled such that they can be dedicated to particular workloads. You can read more about host groups on the KnowledgeCenter
To support this an input map called icp-host-groups were introduced, and this can be used to generate the relevant hosts file for the ICP installer. When using this field it should be used instead of the icp-master, icp-worker, etc fields.
module "icpprovision" {
source = "github.com/ibm-cloud-architecture/terraform-module-icp-deploy?ref=3.0.0"
icp-master = ["${softlayer_virtual_guest.icpmaster.ipv4_address}"]
icp-worker = ["${softlayer_virtual_guest.icpworker.*.ipv4_address}"]
icp-proxy = ["${softlayer_virtual_guest.icpproxy.*.ipv4_address}"]
icp-inception = "3.1.2"
cluster_size = "${var.master["nodes"] + var.worker["nodes"] + var.proxy["nodes"]}"
icp_configuration = {
"network_cidr" = "192.168.0.0/16"
"service_cluster_ip_range" = "172.16.0.1/24"
"default_admin_password" = "My0wnPassw0rd"
}
generate_key = true
ssh_user = "ubuntu"
ssh_key_base64 = "${base64encode(file("~/.ssh/id_rsa"))}"
hooks = {
"cluster-preconfig" = [
"echo This will run on all nodes",
"echo And I can run as many commands",
"echo as I want",
"echo ....they will run in order"
]
"postinstall" = [
"echo Performing some post install backup",
"${ var.postinstallbackup != "true" ? "" : "sudo chmod a+x /tmp/icp_backup.sh ; sudo /tmp/icp_backup.sh" }"
]
}
}module "icpprovision" {
source = "github.com/ibm-cloud-architecture/terraform-module-icp-deploy?ref=3.0.0"
# We will define master, management, worker, proxy and va (Vulnerability Assistant) as well as a custom db2 group
icp-host-groups = {
master = "${openstack_compute_instance_v2.icpmaster.*.access_ip_v4}"
management = "${openstack_compute_instance_v2.icpmanagement.*.access_ip_v4}"
worker = "${openstack_compute_instance_v2.icpworker.*.access_ip_v4}"
proxy = "${openstack_compute_instance_v2.icpproxy.*.access_ip_v4}"
va = "${openstack_compute_instance_v2.icpva.*.access_ip_v4}"
hostgroup-db2 = "${openstack_compute_instance_v2.icpdb2.*.access_ip_v4}"
}
# We always have to specify a node to bootstrap the cluster. It can be any of the cluster nodes, or a separate node that has network access to the cluster.
# We will use the first master node as boot node to run the ansible installer from
boot-node = "${openstack_compute_instance_v2.icpmaster.0.access_ip_v4}"
icp-inception = "3.1.2"
cluster_size = "${var.master["nodes"] + var.worker["nodes"] + var.proxy["nodes"]}"
icp_configuration = {
"network_cidr" = "192.168.0.0/16"
"service_cluster_ip_range" = "172.16.0.1/24"
"default_admin_password" = "My0wnPassw0rd"
}
}module "icpprovision" {
source = "github.com/ibm-cloud-architecture/terraform-module-icp-deploy?ref=3.0.0"
icp-master = ["${softlayer_virtual_guest.icpmaster.ipv4_address}"]
icp-worker = ["${softlayer_virtual_guest.icpworker.*.ipv4_address}"]
icp-proxy = ["${softlayer_virtual_guest.icpproxy.*.ipv4_address}"]
icp-inception = "3.1.2"
cluster_size = "${var.master["nodes"] + var.worker["nodes"] + var.proxy["nodes"]}"
icp_configuration = {
"network_cidr" = "192.168.0.0/16"
"service_cluster_ip_range" = "172.16.0.1/24"
"default_admin_password" = "My0wnPassw0rd"
}
generate_key = true
ssh_user = "ubuntu"
ssh_key_base64 = "${base64encode(file("~/.ssh/id_rsa"))}"
}From NFS location
module "icpprovision" {
source = "github.com/ibm-cloud-architecture/terraform-module-icp-deploy?ref=3.0.0"
icp-master = ["${softlayer_virtual_guest.icpmaster.ipv4_address}"]
icp-worker = ["${softlayer_virtual_guest.icpworker.*.ipv4_address}"]
icp-proxy = ["${softlayer_virtual_guest.icpproxy.*.ipv4_address}"]
image_location = "nfs:fsf-lon0601b-fz.adn.networklayer.com:/IBM02S6275/data01/ibm-cloud-private-x86_64-2.1.0.1.tar.gz"
cluster_size = "${var.master["nodes"] + var.worker["nodes"] + var.proxy["nodes"]}"
icp_configuration = {
"network_cidr" = "192.168.0.0/16"
"service_cluster_ip_range" = "172.16.0.1/24"
"default_admin_password" = "My0wnPassw0rd"
}
generate_key = true
ssh_user = "ubuntu"
ssh_key_base64 = "${base64encode(file("~/.ssh/id_rsa"))}"
}From HTTP location
module "icpprovision" {
source = "github.com/ibm-cloud-architecture/terraform-module-icp-deploy?ref=3.0.0"
icp-master = ["${softlayer_virtual_guest.icpmaster.ipv4_address}"]
icp-worker = ["${softlayer_virtual_guest.icpworker.*.ipv4_address}"]
icp-proxy = ["${softlayer_virtual_guest.icpproxy.*.ipv4_address}"]
image_location = "https://myserver.host/myfiles/ibm-cloud-private-x86_64-2.1.0.1.tar.gz"
cluster_size = "${var.master["nodes"] + var.worker["nodes"] + var.proxy["nodes"]}"
icp_configuration = {
"network_cidr" = "192.168.0.0/16"
"service_cluster_ip_range" = "172.16.0.1/24"
"default_admin_password" = "My0wnPassw0rd"
}
generate_key = true
ssh_user = "ubuntu"
ssh_key_base64 = "${base64encode(file("~/.ssh/id_rsa"))}"
}Load tarball already existing on the boot node.
module "icpprovision" {
source = "github.com/ibm-cloud-architecture/terraform-module-icp-deploy?ref=3.1.0"
icp-master = ["${softlayer_virtual_guest.icpmaster.ipv4_address}"]
icp-worker = ["${softlayer_virtual_guest.icpworker.*.ipv4_address}"]
icp-proxy = ["${softlayer_virtual_guest.icpproxy.*.ipv4_address}"]
image_location = "/opt/ibm/cluster/images/ibm-cloud-private-x86_64-3.1.2.tar.gz"
cluster_size = "${var.master["nodes"] + var.worker["nodes"] + var.proxy["nodes"]}"
icp_configuration = {
"network_cidr" = "192.168.0.0/16"
"service_cluster_ip_range" = "172.16.0.1/24"
"default_admin_password" = "My0wnPassw0rd"
}
generate_key = true
ssh_user = "ubuntu"
ssh_key_base64 = "${base64encode(file("~/.ssh/id_rsa"))}"
}There are several examples for different providers available from IBM Cloud Architecture Solutions Group github page
Configuration file is generated from items in the following order
- config.yaml shipped with ICP (if config_strategy = merge, else blank)
- config.yaml specified in
icp_config_file - key: value items specified in
icp_configuration
Details on configuration items on ICP KnowledgeCenter
The module supports automatic scaling of worker nodes.
To scale simply add more nodes in the root resource supplying the icp-worker variable.
You can see working examples for softlayer in the icp-softlayer repository
Please note, because of how terraform handles module dependencies and triggers, it is currently necessary to retrigger the scaling resource after scaling down nodes. If you don't do this ICP will continue to report inactive nodes until the next scaling event. To manually trigger the removal of deleted node, run these commands:
terraform taint --module icpprovision null_resource.icp-worker-scalerterraform apply
As new use cases and best practices emerge code will be added to and changed the in module. Any changes in the code leads to a new release version. The module versions follow a semantic versioning scheme.
To avoid breaking existing templates which depends on the module it is recommended to add a version tag to the module source when pulling directly from git.
- Fix issues with offline install
- Fix issue with config generation
- Allow cluster directory to be specified
- Allow other targets to be called from
icp-inception - Fix issues when owner of cluster files are something other than
ssh_user - Allow the cluster directory to be owned by arbitrary user after install
- Accept local files as valid location for
image_location
- Fix docker install from yum repo for non-root user on RHEL
- Fix password length to comply with ICP 3.1.2 password rules
- Fix verbose installation option
- Fix docker install on ppc64le and s390x
- Fix default_admin_password output when provided as empty string in icp_config
- Fix blank icp_configuration["default_admin_password"] not generating random password
- Fix remote hook issue
- Fix local-hook issue
- Retire parallel image pull
- Retire unused variables (enterprise-edition, image_file, ssh_key, ssh_key_file)
- Default to generate strong default admin password if no password is specified
- Detect inception image when installing from offline tarball
- Rename
icp-versionvariable to more descriptiveicp-inception - Overhaul of scaler function
- Add support for automatic installation of docker on RHEL and Centos
- Add support for downloading multi-arch images
- Fix wget output spamming during wget image download
- Include BATS test for testing scripts locally
- Rewrite image-load to take username and password separately when downloading from HTTP source
- Add support for local hooks
- Support specifying docker version when installing docker with apt (Ubuntu only)
- Ensure /opt/ibm is present before copying cluster skeleton
- Add retry logic to apt-get when installing prerequisites. Sometimes cloud-init or some other startup process can hold a lock on apt.
- Retry ssh from boot to cluster nodes when generating /etc/hosts entries. Fixes issues when some cluster nodes are provisioned substantially slower.
- Report exit code from docker when running ansible installer, rather than the last command in the pipelist (tee)
- Skip blanks when generating config.yaml as yaml.safe_dump exports them as '' which ansible installer doesn't like
- Create backup copy of original config.yaml to keep options and comments
- Support nested dictionaries when parsing
icp_configurationto convert true/false strings to booleans
- Fix empty icp-master list issue when using icp-host-groups
- Fix issue with docker package install from nfs source
- Make docker check silent when docker is not installed
- Fix issues with terraform formatting of boolean values in config.yaml
- Fix issue with non-hostgroups installations not generating hosts files
- Fix boot-node not being optional in non-hostgroups installations
- Fix issue with boot node trying to ssh itself
- Install docker from repository if no other method selected (ubuntu only)
- Fix apt install issue for prerequisites
- Add full support for separate boot node
- Save icp install log output to /tmp/icp-install-log.txt
- Add option for verbosity on icp install log output
- Fix issues with email usernames when using private registry
- Fix passwords containing ':' when using private registry
- Fix scaler error when using hostgroups
- Added support for hostgroups
- Updated preprequisites scripts to avoid emediate failure in airgapped installations
- Include module outputs
- Added support for install hooks
- Added support for converged proxy nodes (combined master/proxy)
- Added support for private docker registry
- Fixed problem with worker scaler
- Added support for ssh bastion host
- Added support for dedicated management hosts
- Split up null_resource provisioners to increase granularity
- Added support for parallel load of EE images
- Various fixes
- Initial release