Setting Up a Kubernetes Cluster with VMware NSX

Introduction

Setting up a Kubernetes cluster powered by VMware NSX demonstrates the power of Kubernetes as a service with virtual networking.

Deploying a Kubernetes cluster involves two main steps: creating resources for a tenant and its project and then spinning them up into a cluster with the photon service create command. On Photon Platform with NSX, the following primitives are the building blocks of clusters:

• Images
• Virtual machines
• Disks
• Quotas
• Virtual routers, switches, and subnets

Requirements

A Kubernetes cluster powered by NSX carries several requirements:

• NSX is installed and configured to work with Photon Controller.
• The Kubernetes image for Photon Controller.
• Enough floating IP addresses for the master and etcd nodes.

The instructions assume that you are using a Linux workstation with the Photon Controller CLI connected to a deployment of Photon Controller on ESXi with VMware NSX. You can also use a Mac or Windows workstation with the Photon Controller CLI installed.

For information about Photon commands, subcommands, and options, see the help for the Photon Controller CLi on your Linux workstation; examples:

photon --help
photon service create -h

For more information about creating a Kubernetes cluster, or for instructions on creating a cluster without using NSX, see Creating a Kubernetes Cluster.

Obtaining, Uploading, and Enabling the Kubernetes Image

Download the Kubernetes disk image for Photon Controller from the following URL to the workstation on which you are running the Photon Controller command-line utility:

https://github.com/vmware/photon-controller/releases

The Kubernetes image is packaged as an OVA file; it has a file name that looks like this, though the version, build, and other numbers might be slightly different:

kubernetes-1.6.0-pc-1.2-5caa892.ova

Upload the the Kubernetes image to Photon Controller by running the following commands as the system administrator, replacing the variables with the IP address of the load balancer for the Photon Controller management plane:

photon target set https://mgmt-ip-address:443
photon target login -u administrator@photon.local.v -p 'MySecret1!'
photon image create kubernetes-1.6.0-pc-1.2-5caa892.ova -n kube1 -i EAGER

The upload takes a few minutes. When it finishes, enable the image by obtaining its ID with the photon image list command and using it to replace the variable in the enable-cluster-type command. (Another option is to specify the image at the time of creating the cluster by using the --image-id option; see the help output for the photon service create command.)

photon image list
photon -n deployment enable-cluster-type default -k KUBERNETES -i <Kubernetes_image_ID>

As the Photon Controller system administrator, you typically need to enable a cluster type only once. Enabling the cluster type sets Photon Controller to use the given image to deploy each of your Kubernetes clusters. You should, however, disable the cluster type for a given image before you delete the image.

Creating a Tenant, a Project, and a Quota for the Cluster

A tenant is a unit of administrative control allocated a quota for projects, such as a Kubernetes cluster. The command creates a tenant and gives it quota of resources with the limits option. To provide the tenant with resources for NSX and Kubernetes clusters, the quota includes persistent disks, VMware vSAN storage, and floating IP addresses.

(This example code block requires that vSAN is installed and set up to work with Photon Controller. However, vSAN is not required for the Kubernetes cluster with NSX, and you can provision a different set of resources in the quota. See Creating a Tenant, a Project, and a Quota.)

photon tenant create demoTenant 
--limits 'vm 10000 COUNT, vm.cost 20000 COUNT, vm.cpu 10000 COUNT, 
vm.memory 10000 GB, ephemeral-disk.count 20000 COUNT, 
ephemeral-disk.capacity 20000 GB, ephemeral-disk.cost 20000 GB, 
ephemeral-disk 9999 COUNT, persistent-disk.count 20000 COUNT, 
persistent-disk.capacity 20000 GB, persistent-disk.cost 20000 GB, 
storage.LOCAL_VMFS 20000 COUNT, storage.VSAN 20000 COUNT, 
sdn.floatingip.size 20000 COUNT'

The following command creates a project that uses 50 percent of the tenant's quota:

photon project create demoProject --tenant demoTenant --percent 50  -c 192.168.192.0/24

Creating Flavors for the Cluster

Photon Controller includes the following default flavors for clusters:

• cluster-master-vm. Photon Controller supplies this flavor as the default for a Kubernetes master node.
• cluster-other-vm, for etcd nodes.
• cluster-vm-disk, the default ephemeral disk for a Kubernetes cluster.

When you create a Kubernetes cluster, Photon Controller taps the default flavors unless you specify other flavors.

The next two commands specify custom flavors. These custom flavors control the size of the resources that the cluster will use.

photon flavor create --name "small-vm" --kind "vm" 
--cost "vm 1.0 COUNT, vm.cpu 3.0 COUNT, vm.memory 2.0 GB"
photon flavor create --name "small-disk" 
--kind "ephemeral-disk" --cost "ephemeral-disk 1.0 COUNT"

Create a Subnet for the Cluster

Photon Platform combines Photon Controller with NSX to let you create infrastructure as a service---in this case, a virtual subnet for a Kubernetes cluster.

Before creating the subnet, however, you will need to identify an NSX Tier-1 router for the subnet by listing the available routers:

photon router list

From the list, copy the ID of the router that you want to use for the subnet and add it as the argument of the router option (-r) in the following command, which creates the subnet for the cluster:

photon subnet create -n myTestSubnet -d description -i 192.168.192.0/24 -r <router-ID>

Example:

photon subnet create -n myTestSubnet -d description 
-i 192.168.192.0/24 -r 28db58b854cecc8dbb4e1

Finally, list the subnets so that you can obtain the ID of the subnet you just created for the cluster. You will need the subnet's ID when you create the cluster in the next step:

photon subnet list

Creating the Kubernetes Cluster

You are now ready to create a Kubernetes cluster by running the photon service create command. First, however, take a moment to look at the command's help and compare the differences in required options between clusters that use NSX networking and those that do not.

photon service create --help

When you create a Kubernetes cluster that uses NSX virtual networking, for example, you do not specify the IP addresses for the Kubernetes master, the load balancer, the etcd nodes, DNS, the gateway, or the netmask. The master node, load balancer, and etcd nodes automatically receive floating IP addresses. After the cluster is deployed, you can run the photon service show command to obtain information about the cluster, including its IP addresses.

Some information, however, must be supplied when you create the cluster:

• The number of masters, which must, at present, equal 1 or 2.
• The number of etcd nodes, which must equal either 1 or, for high availability, 3.

Here, then, is the command to create a Kubernetes cluster that uses NSX for networking. Note that, for the argument of the w option, the command uses the example ID of the subnet from earlier. Replace the example ID with your own.

photon service create --name k8-service --type KUBERNETES 
--subnet_id 28db58b854ced1c9521e8 --worker_count 1 
--number-of-etcds 3 --number-of-masters 1 
--container-network 10.10.0.0/16
--master-vm-flavor "small-vm"
--worker-vm-flavor "small-vm"
--disk_flavor "small-disk"

Keep in mind that the flavors are not required but are added to constrain the size of the resources; if you do not specify the flavors, the defaults are used.

If the service create command prompts you for several inputs, you can press Enter to accept the defaults, or you can specify the values that you want. For example, you are prompted for the SSH key for the VMs. You can leave it empty or supply your own.

Getting Information About the New Cluster

You can get information about the cluster by running the following commands.

First, get the ID of the cluster:

photon service list

Next,run the following command, replacing the example ID with that of your newly created cluster:

photon service show 5b879631-3c28-4f54-af09-7cda100d9e42
Service ID:             5b879631-3c28-4f54-af09-7cda100d9e42
  Name:                 k8-service
  State:                READY
  Type:                 KUBERNETES
  Image ID:             de8e2b9d-0a82-4973-badc-3447fe4a4ed8
  Kubernetes IP:        172.22.0.11
  Worker count:         1
  Extended Properties:  map[master_ips:172.22.0.10 
  etcd_ips:172.22.0.8 
  service_ui_url:https://172.22.0.11:6443/ui 
  pc_external_uri:https://172.20.0.55 client_linux_386_
...

The photon service show command's output contains key information:

• The Kubernetes IP address is the IP address that is assigned to the load balancer---the IP address that you connect to manage the cluster.
• The IP address to connect to the Kubernetes Dashboard.

Creating the Cluster with an API Call

Instead of using the photon service create command to set up a cluster, you can use the Photon Controller API. The following payload of an API call, for example, establishes an NSX-powered Kubernetes cluster. An API call assumes that you have obtained an access token to work with the UI; see Using the API.

{
  "name": "test",
  "workerCount": 1,
  "type": "KUBERNETES",
  "extendedProperties": {
    "number_of_masters": 1,
    "number_of_etcds": 1,
    "container_network": "10.0.2.0/16"
  },
  "subnetId": "28db58b854ced1c9521e8",
  "workerBatchExpansionSize": 1
}

Checking the Cluster's Resources and Status

After provisioning the cluster and its resources, you can log in to the Photon Controller web interface to check them out:

https://<ip-address-of-photon-load-balancer>:4343

In the web interface, locate your tenant and the project that contains the NSX-powered Kubernetes cluster. Once you've reached the project, you can click Clusters and then click the name of a cluster to see its settings.

Opening the Kubernetes Dashboard

Now you're ready to open the Kubernetes Dashboard to work with your cluster. For instructions on how to obtain authentication information to open the dashboard, see Opening the Kubernetes Dashboard.

After you connect to the dashboard, you can run an application in your Kubernetes cluster; see Deploying an nginx Web Server on a Kubernetes Cluster.