pupernetes - p8s

Table of Contents

• Goals
• Requirements
  • Runtime
    • Executables
    • Systemd
    • Resources
    • DNS
  • Development
    • Build
• Getting started
  • Download
  • Run
  • Stop
  • Systemd as job type
  • Command line docs
• Metrics
• Current limitations

Goals

Run a managed Kubernetes setup.

This project's purpose is to provide a simple Kubernetes setup to validate any software on top of it.

You can use it to validate a software dependency on Kubernetes itself or just to run some classic app workflows with argo.

As pupernetes runs in travis and circle-ci, it becomes very easy to integrate this tool in any Kubernetes project.

This project has been initially designed to perform the end to end testing of the datadog-agent.

Provides:

• etcd v3
• kubectl
• kubelet
• kube-apiserver
• kube-scheduler
• kube-controller-manager
• kube-proxy
• coredns

The default setup is secured with:

• Valid x509 certificates provided by an embedded vault PKI
  • Able to use the Kubernetes CSR and the service account root-ca
• HTTPS webhook to provide token lookups for the kubelet API
• RBAC

Requirements

Runtime

Executables

• tar
• unzip
• systemctl
• systemd-resolve (or a non-systemd managed /etc/resolv.conf)
• mount

Additionally any implicit requirements needed by the kubelet, like the container runtime and more. Currently only reporting docker, please see the current limitations.

Systemd

A recent systemd version is better to gain:

• systemd-resolve
• journalctl --since
• more convenient dbus API

Resources

• 4GB of memory is recommended
• 5GB of free disk space for the binaries and the container images

DNS

Ensure your hostname is discoverable:

dig $(hostname) -short

Development

Pupernetes must be run on linux (or linux VM).

Please see our ubuntu 18.04 notes about it.

To compile pupernetes, you need the following binaries:

• go 1.10
• make

Build

go get -u github.com/DataDog/pupernetes
cd ${GOPATH}/src/github.com/DataDog/pupernetes
make

Getting started

Download

You need to download the last version:

VERSION=0.6.1
curl -LOf https://github.com/DataDog/pupernetes/releases/download/v${VERSION}/pupernetes
chmod +x ./pupernetes
./pupernetes --help

Run

sudo ./pupernetes daemon run /opt/sandbox/

Note:

kubectl can be automatically installed by pupernetes.

You need to run the following command to add kubectl to the $PATH:

sudo ./pupernetes daemon run /opt/sandbox/ --kubectl-link /usr/local/bin/kubectl

$ kubectl get svc,ds,deploy,job,po --all-namespaces

NAMESPACE     NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
default       kubernetes   ClusterIP   192.168.254.1   <none>        443/TCP         3m
kube-system   coredns      ClusterIP   192.168.254.2   <none>        53/UDP,53/TCP   3m

NAMESPACE     NAME             DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
kube-system   kube-proxy       1         1         1         1            1           <none>          3m
kube-system   kube-scheduler   1         1         1         1            1           <none>          3m

NAMESPACE     NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
kube-system   coredns   1         1         1            1           3m

NAMESPACE     NAME                       READY     STATUS    RESTARTS   AGE
kube-system   coredns-747dbcf5df-p2lhq   1/1       Running   0          3m
kube-system   kube-controller-manager    1/1       Running   0          3m
kube-system   kube-proxy-wggdn           1/1       Running   0          3m
kube-system   kube-scheduler-92zrj       1/1       Running   0          3m

Stop

Gracefully stop it with:

• SIGINT
• SIGTERM
• --timeout
• curl -XPOST 127.0.0.1:8989/stop

Hyperkube versions

pupernetes can start a specific Kubernetes version with the flag --hyperkube-version=1.9.3.

These are the current supported versions:

• 1.11
• 1.10
• 1.9
• 1.8
• 1.7
• 1.6
• 1.5
• 1.4
• 1.3

Systemd as job type

It's possible to run pupernetes as a systemd service directly with the command line. In this case, pupernetes asks to systemd-dbus to be daemonised with the given arguments. See more info about it in the run command.

This command line is very convenient to run pupernetes in SaaS CI:

• travis
• circle-ci

Command line docs

The full documentation is available here.

Metrics

Pupernetes exposes prometheus metrics to improve the observability.

You can have a look at which metrics are available here.

Current limitations

• Container runtime
  • You need docker already up and running
  • You cannot use cri-containerd / crio without changing manually the systemd unit /run/systemd/system/p8s-kubelet.service
• Systemd
  • Currently working with systemd only
  • Could be containerized with extensive mounts
    • binaries
    • dbus
• Networking
  • The CNI bridge cannot be used yet
  • Kubernetes cluster IP range is statically set
• Secrets
  • IP SAN
    • Statically configured with the given Kubernetes cluster IP range
• Support for Custom Metrics
  • You can register an API Service for an External Metrics Provider. This is only supported for 1.10.x and 1.11.x.