Kubernetes Cloud Controller Manager implementation for Brightbox Cloud
This implementation uses the cloud-controller framework from Kubernetes described in the following documents
- https://kubernetes.io/docs/tasks/administer-cluster/running-cloud-controller/
- https://kubernetes.io/docs/concepts/architecture/cloud-controller/
- https://kubernetes.io/docs/tasks/administer-cluster/developing-cloud-controller-manager/
main.go is a copy of the controller manager
with alterations made to accommodate this implementation.
The controller uses the Kubernetes versioning system with magic strings injected into the binary via Loader flags given to the Go compiler.
The script hack/version.sh generates the required loader flags from
the commits and tags in the git repository.
Vendored dependencies are managed with dep with the restrictions
described in Gopkg.toml. Run dep ensure to check vendor is up to date.
The implementation in the brightbox package satisifes the
cloudprovider.Interface pluggable interface.
The provider implements:
-
the node controller - annotating nodes with zone, region and image type information allowing the k8s scheduler to better place containers in a resilient manner
-
the service controller - building Brightbox Cloud load balancers on demand.
It treats the cloud-controller framework and the wider Kubernetes system as a supervisor system - along the lines of Erlang. The approach taken in each implemented function is to be completely idempotent and to crash out as soon as any problem is detected - returning a meaningful error. Kubernetes will then log the error message in the event list for the resource and schedule a retry.
The main advantage is that you don't need to wait for anything in the code. Just stop and let the retry process handle it. Avoid making unnecessary API calls to reduce the load on Brightbox Cloud API servers.
Currently each load balancer created generates a server group and firewall policy specifically for that load balancer with a single firewall rule pointing at the exposed node port. This avoids any race conditions within the cloud-controller trying to insert and update rules in a single firewall policy and group.
The Controller avoids any additional Goroutines. The Interfaces
implemented are described in brightbox/cloud-controller-interface.go,
with a separate file in the package for each of the interfaces
implemented.
The load balancer supports annotations that allow you to use the range of facilities available on Brightbox Cloud Load Balancers. See the manifest examples for details.
We're very happy to receive PR requests on this controller. If you can improve things or make suggestions then please do so.
make test runs the Go test files against the implementation. The
Brightbox Cloud API is mocked out so you can run it without credentials.
make compile creates the stripped binary cloud controller (including
magic version information) that can be copied to a cluster.
make build creates a docker image of the cloud controller.
To test on a cluster shutdown any running cloud controller containers
using kubectl, copy the binary to the cluster and then run in
a shell. (This assumes the cluster was built with kubeadm and the
admin.conf kubeconfig is available).
Obtain an api client from the Brightbox Manager and run the controller using the following command
sudo BRIGHTBOX_CLIENT=cli-xxxxx \
BRIGHTBOX_CLIENT_SECRET=my_secret \
BRIGHTBOX_API_URL=https://api.gb1.brightbox.com \
./brightbox-cloud-controller-manager \
--cloud-provider=brightbox \
--bind-address=::1 \
--port=0 \
--secure-port=10253 \
--configure-cloud-routes=false \
--kubeconfig=/etc/kubernetes/admin.conf \
--cluster-name=My_Cluster_Name \
--leader-elect=false \
-v=4 \
--use-service-account-credentials=false
The logs from the cloud controller will come out on the terminal.
Exercise the controller by using kubectl in a separate terminal by
creating, updating and deleting nodes and loadbalancers.