Contents
- Try Kubernetes on separated machines
- Install
- Initialization of kube-master
- Initialization of a k8s node
- Retrive the way to add a node
- Enable metrics-server for HPA
- Integrate standalone-cinder of the external cloud-provider-openstack for Dynamic Volume Provisioning
- Enable MetricsGrabber
- Enable StorageObjectInUseProtection plugin on admission controller
- How to see REST API operation on kubectl command
- How to know resource workload for each node
- Run e2e test
- Helm & Spinnaker
- Operate something
- Sort instances with --sort-by
- Create a pod
- Create a deployment
- Create a snapshot of etcd
- Make a node tainted and pods go away from the node
- Create a secret and use it from a pod
- Rolling-upgrade for a deployment
- Rolling-back of a deployment
- Verify DNS works for Services and Pods
- Use init-containers
- Create a DaemonSet
- Troubleshooting
Based on http://tracpath.com/works/devops/how-to-install-the-kubernetes-kubeadm/
- Distro: ubuntu 18.04 LTS
(VirtualBox) Add an internal network between VMs:
http://qiita.com/areaz_/items/c9075f7a0b3e147e92f2#%E3%82%B2%E3%82%B9%E3%83%88os%E3%81%AE%E5%8B%95%E4%BD%9C%E7%A2%BA%E8%AA%8D Shutdown a VM Setting -> Network -> Adapter 2 Check "Enable Network Adapter" Attached to: "Internal Network" Reboot the VM SSH into the VM (kube-host01 should be 172.168.0.2) $ sudo vi /etc/network/interfaces + auto enp0s8 + iface enp0s8 inet static + address 172.168.0.1 + netmask 255.255.255.0 Reboot the VM
Operate the following installation on both kube-master and kube-host01 (nfs-common is for Subpath):
$ sudo su - # apt-get update && apt-get install -y apt-transport-https # curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - # echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" > /etc/apt/sources.list.d/kubernetes.list # add-apt-repository ppa:gluster/glusterfs-4.1 # apt-get update # apt-get install -y docker.io nfs-common glusterfs-client
To install the latest packages of Kubernetes:
# apt-get install -y kubelet kubeadm kubectl
If you want to install previous release of Kubernetes, check available releases with:
# apt-cache policy kubelet
kubelet:
Installed: (none)
Candidate: 1.10.0-00
Version table:
1.10.0-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
1.9.6-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
1.9.5-00 500
[..]
#
Then you can specify the release like:
# apt-get install -y kubelet=1.9.6-00 kubeadm=1.9.6-00 kubectl=1.9.6-00
(Flannel) Operate the following commands:
# kubeadm init --pod-network-cidr=10.244.0.0/16 [..] Your Kubernetes master has initialized successfully! [..] You can now join any number of machines by running the following on each node as root: kubeadm join --token 22ac74.4d061109507a992b 172.27.138.55:6443
10.244.0.0/16 comes from kube-flannel.yml which contains:
"Network": "10.244.0.0/16",
(Other) Operate the following commands:
# kubeadm init [..] Your Kubernetes master has initialized successfully! [..] You can now join any number of machines by running the following on each node as root: kubeadm join --token 22ac74.4d061109507a992b 172.27.138.55:6443 #
The above output needs to be operated on kube-host01 to join into the cluster.
If using VirtualBox, need to specify the internal ip address like:
# kubeadm init --apiserver-advertise-address 172.168.0.1
Operate the following commands:
$ sudo cp /etc/kubernetes/admin.conf $HOME/ $ sudo chown $(id -u):$(id -g) $HOME/admin.conf $ export KUBECONFIG=$HOME/admin.conf $ echo "export KUBECONFIG=$HOME/admin.conf" >> $HOME/.bashrc
(Flannel) Configure network setting for pod2pod communication:
$ wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml $ kubectl apply -f kube-flannel.yml
(Weave) Configure network setting for pod2pod communication:
$ kubectl apply -f https://git.io/weave-kube-1.6
Check the valid installation:
$ kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system etcd-kube-manager 1/1 Running 0 1h kube-system kube-apiserver-kube-manager 1/1 Running 0 1h kube-system kube-controller-manager-kube-manager 1/1 Running 0 1h kube-system kube-dns-692378583-3gbgp 0/3 ContainerCreating 0 1h kube-system kube-proxy-4rbvg 1/1 Running 0 1h kube-system kube-scheduler-kube-manager 1/1 Running 0 1h kube-system weave-net-cjf25 2/2 Running 0 51s $
Confirm the STATUS becomes Ready:
$ kubectl get nodes NAME STATUS AGE VERSION k8s-master Ready 1m v1.7.3
Make the manager schedulable:
$ kubectl describe nodes | grep Tain Taints: node-role.kubernetes.io/master:NoSchedule $ kubectl taint nodes <master nodename: k8s-master> node-role.kubernetes.io/master:NoSchedule- node "k8s-master" untainted $ kubectl describe nodes | grep Tain Taints: <none> $
To make the mount propagation work proerly, edit /etc/systemd/system/multi-user.target.wants/docker.service like:
[Service] - MountFlags=slave + MountFlags=shared
This is required to pass e2e test "[sig-storage] CSI Volumes CSI plugin test using CSI driver: hostPath".
To add a node into k8s cluster, operate the following command on a node (not manager):
# kubeadm join --token 22ac74.4d061109507a992b 172.27.138.55:6443
Check the node joins into the cluster with the command on the manager:
$ kubectl get nodes NAME STATUS AGE VERSION kube-host01 Ready 51s v1.6.6 kube-manager Ready 1h v1.6.6 $
Get a kubeadm token on k8s-master:
$ TOKEN=`sudo kubeadm token list | grep authentication | awk '{print $1}'`
$ echo $TOKEN
c3cf19.89e62945a88d7a91
If you cannot get a token, need to recreate with:
$ sudo kubeadm token create
Get a discovery token on k8s-master:
$ DISCOVERY_TOKEN=`openssl x509 -pubkey \ -in /etc/kubernetes/pki/ca.crt | openssl rsa \ -pubin -outform der 2>/dev/null | openssl dgst \ -sha256 -hex | sed 's/^.* //'` $ echo $DISCOVERY_TOKEN b3bb83c24673649bf1909e9144929a64569b1a7988df97323a9a3449c3b4c1e6
Get an endpoint on k8s-master:
$ ENDPOINT=`cat admin.conf | grep server | sed s@" server: https://"@@` $ echo $ENDPOINT 192.168.1.105:6443
Use the token and the discovery token on k8s-node to add a new node on the node:
# TOKEN=c3cf19.89e62945a88d7a91
# DISCOVERY_TOKEN=b3bb83c24673649bf1909e9144929a64569b1a7988df97323a9a3449c3b4c1e6
# ENDPOINT=192.168.1.105:6443
#
# kubeadm join --token ${TOKEN} ${ENDPOINT} \
--discovery-token-ca-cert-hash sha256:${DISCOVERY_TOKEN}
Install metrics-server on k8s-master:
$ git clone https://github.com/kubernetes-incubator/metrics-server $ cd metrics-server/ $ kubectl create -f deploy/kubernetes/
Integrate standalone-cinder of the external cloud-provider-openstack for Dynamic Volume Provisioning
NOTE: It is not necessary to add options (--cloud-provider, --cloud-config) to kube-controller-manager and other processes at all.
Use manifests as samples from https://github.com/oomichi/try-kubernetes/tree/master/manifests/standalone-cinder-external
Add RBAC for standalone-cinder deployment:
$ kubectl create -f rbac.yaml
Change hostAliases, OS_AUTH_URL and other OS_*** env values of deployment.yaml for your environment.
Deploy standalone-cinder:
$ kubectl create -f deployment.yaml
Add default StorageClass:
$ kubectl create -f storage-class.yaml
Verify Dynamic Volume Provisioning works fine:
$ kubectl create -f pvc.yaml $ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE cinder-claim Bound pvc-af01ada4-9cf4-11e8-a146-fa163e420595 1Gi RWO gold 31s $
Change the listening address of kube-scheduler to 0.0.0.0:
$ sudo vi /etc/kubernetes/manifests/kube-scheduler.yaml
--- ./kube-scheduler.yaml.orig 2018-08-22 02:30:16.060204589 +0000
+++ /etc/kubernetes/manifests/kube-scheduler.yaml 2018-08-22 02:30:38.160555932 +0000
@@ -13,7 +13,7 @@
containers:
- command:
- kube-scheduler
- - --address=127.0.0.1
+ - --address=0.0.0.0
- --kubeconfig=/etc/kubernetes/scheduler.conf
- --leader-elect=true
image: k8s.gcr.io/kube-scheduler-amd64:v1.11.1
Change the listening address of kube-controller-manager to 0.0.0.0:
$ sudo vi /etc/kubernetes/manifests/kube-controller-manager.yaml
--- kube-controller-manager.yaml 2019-01-08 23:39:48.721525219 +0000
+++ /etc/kubernetes/manifests/kube-controller-manager.yaml 2019-01-08 23:40:09.266003213 +0000
@@ -13,7 +13,7 @@
containers:
- command:
- kube-controller-manager
- - --address=127.0.0.1
+ - --address=0.0.0.0
- --allocate-node-cidrs=true
- --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
- --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
StorageObjectInUseProtection sets the PV protection finalizer flag before a PV object appears on k8s API. "PV protection" e2e test requires this behavior.
Add StorageObjectInUseProtection to --enable-admission-plugins option:
$ sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml - --advertise-address=192.168.1.102 - --allow-privileged=true - --client-ca-file=/etc/kubernetes/pki/ca.crt - - --enable-admission-plugins=NodeRestriction + - --enable-admission-plugins=NodeRestriction,StorageObjectInUseProtection - --enable-bootstrap-token-auth=true - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
Just specify '--v=8' option on kubectl command like:
$ kubectl --v=8 get nodes
[..] GET https://172.27.138.55:6443/api/v1/nodes
[..] Request Headers:
[..] Accept: application/json
[..] User-Agent: kubectl/v1.6.6 (linux/amd64) kubernetes/7fa1c17
[..] Response Status: 200 OK in 21 milliseconds
[..] Response Headers:
[..] Content-Type: application/json
[..] Date: Wed, 28 Jun 2017 00:33:39 GMT
[..] Response Body: {"kind":"NodeList","apiVersion":"v1",
"metadata":{"selfLink":"/api/v1/nodes","resourceVersion":"7254"},
"items":[{"metadata":{"name":"kube-host01","selfLink":"/api/v1/nodeskube-host01",
"uid":"a354969d-5b98-11e7-9e55-1866da463eb0",
"resourceVersion":"7244","creationTimestamp":"2017-06-28T00:27:59Z",
"labels":{"beta.kubernetes.io/arch":"amd64",
"beta.kubernetes.io/os":"linux",
"kubernetes.io/hostname":"kube-host01"} ..
This requires metrics-server is enabled. How to know resource work load for each node is:
$ kubectl top node NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% k8s-master 113m 5% 1271Mi 33% k8s-node01 116m 5% 902Mi 23%
Need to install golang 1.13.7 which is not provided from ubuntu 18.04 as the default. So we need to do the following process for that:
$ wget https://go.dev/dl/go1.18.2.linux-amd64.tar.gz $ sudo tar -C /usr/local/ -xzf go1.18.2.linux-amd64.tar.gz $ export PATH=$PATH:/usr/local/go/bin $ echo "export PATH=$PATH:/usr/local/go/bin" >> $HOME/.bashrc $ sudo ln -s /usr/local/go/bin/go /usr/local/bin/go
Set GOPATH as parmanent setting:
$ mkdir ${HOME}/go
$ echo "export GOPATH=${HOME}/go" >> ${HOME}/.bashrc
Install some building packages:
$ sudo apt-get install -y docker.io gcc make
Download k8s source code:
$ go get k8s.io/test-infra $ go get k8s.io/kubernetes
Check out the same version as the target k8s cluster:
$ cd $GOPATH/src/k8s.io/kubernetes
$ kubectl version
Client Version: version.Info{
Major:"1", Minor:"11", GitVersion:"v1.11.1",
GitCommit:"b1b29978270dc22fecc592ac55d903350454310a",
GitTreeState:"clean", BuildDate:"2018-07-17T18:53:20Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1",
GitCommit:"b1b29978270dc22fecc592ac55d903350454310a",
GitTreeState:"clean", BuildDate:"2018-07-17T18:43:26Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
$
$ git tag -l
v0.10.0
..
v1.11.1
..
$
$ git checkout refs/tags/v1.11.1
$ git checkout -b tag-v1.11.1
Build e2e test binary. (NOTE: When changing the e2e code, we need to build the binary again to apply the changes):
# The docker daemon runs as root user, not docker user. So it is necessary to specify `sudo`
$ cd ${HOME}/go/src/k8s.io/test-infra/kubetest
$ GO111MODULE=on go install ./kubetest
$ export PATH=${PATH}:${HOME}/go/bin
$ cd ${HOME}/go/src/k8s.io/kubernetes
$ sudo ${HOME}/go/bin/kubetest --build
Run e2e test:
$ export KUBECONFIG=$HOME/admin.conf $ export KUBERNETES_CONFORMANCE_TEST=true $ kubetest --provider=skeleton --test --test_args="--ginkgo.focus=\[Conformance\]"
Confirm which tests will run without actual tests (Need to specify provider to avoid an error):
$ kubetest --provider=skeleton --test --test_args="--ginkgo.dryRun=true --ginkgo.focus=\[Conformance\]" [..] [k8s.io] Docker Containers should use the image defaults if command and args are blank [Conformance] /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/docker_containers.go:35 ~SS ------------------------------ [k8s.io] EmptyDir volumes should support (non-root,0644,tmpfs) [Conformance] [sig-storage] /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/empty_dir.go:85 ~SS ------------------------------ [sig-apps] ReplicaSet should serve a basic image on each replica with a public image [Conformance] /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/replica_set.go:82 ~S ------------------------------ [sig-network] Services should provide secure master service [Conformance] /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/service.go:71 ~ Ran 149 of 652 Specs in 0.072 seconds SUCCESS! -- 0 Passed | 0 Failed | 0 Pending | 503 Skipped PASS Ginkgo ran 1 suite in 519.123083ms Test Suite Passed 2017/08/09 15:38:12 util.go:133: Step './hack/ginkgo-e2e.sh --ginkgo.dryRun=true --ginkgo.focus=\[Conformance\]' finished in 937.615925ms 2017/08/09 15:38:12 e2e.go:80: Done $
Specify a single test with regex:
$ kubetest --provider=skeleton --test --test_args="--ginkgo.focus=1\spod\sto\s2\spods"
If changing e2e code, we need to specify --check-version-skew=false to skip checking versions of both server and e2e client:
$ kubetest --provider=skeleton --test --test_args="--ginkgo.focus=from\s3\sto\s5$" --check-version-skew=false
Install bazel:
$ sudo apt-get install pkg-config zip g++ zlib1g-dev unzip python $ wget https://github.com/bazelbuild/bazel/releases/download/0.21.0/bazel-0.21.0-installer-linux-x86_64.sh $ chmod 755 bazel-0.21.0-installer-linux-x86_64.sh $ ./bazel-0.21.0-installer-linux-x86_64.sh --user $ $ export PATH="$PATH:$HOME/bin" $ echo "export PATH=$PATH:$HOME/bin" >> $HOME/.bashrc
Run unit tests on kubernetes/test-infra:
$ bazel test //..
- https://github.com/kubernetes/test-infra#building-and-testing-the-test-infra
- http://qiita.com/lucy/items/e4f21c507d3fd2c0ffe9
with make:
$ make test
with bazel:
$ bazel test //...
As https://github.com/kubernetes/helm#install
$ wget https://storage.googleapis.com/kubernetes-helm/helm-v2.9.1-linux-amd64.tar.gz $ tar -zxvf helm-v2.9.1-linux-amd64.tar.gz $ sudo mv linux-amd64/helm /usr/local/bin/ $ helm init
Verify helm:
$ helm version
Client: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}
$
Add permission to deploy tiller:
$ kubectl create serviceaccount --namespace kube-system tiller
$ kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
$ kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'
Install Spinnaker:
$ wget https://raw.githubusercontent.com/kubernetes/charts/master/stable/spinnaker/values.yaml $ helm install -n kubelive -f values.yaml stable/spinnaker Error: timed out waiting for the condition $ $ helm ls --all kubelive NAME REVISION UPDATED STATUS CHART NAMESPACE kubelive 1 Tue May 15 21:36:52 2018 FAILED spinnaker-0.4.1 default $ $ kubectl get pods NAME READY STATUS RESTARTS AGE kubelive-create-bucket-j97wn 0/1 CrashLoopBackOff 5 10m kubelive-jenkins-86bcb6c4b5-h4bqx 0/1 Pending 0 10m kubelive-minio-5d78b95d9c-pkpss 0/1 Pending 0 10m kubelive-redis-5667b84965-k4nmz 0/1 Pending 0 10m kubelive-spinnaker-clouddriver-85997f4b64-q97qq 0/1 Running 0 10m kubelive-spinnaker-deck-86c48f7594-vxmnt 1/1 Running 0 10m kubelive-spinnaker-echo-8ccc9956c-prk58 1/1 Running 0 10m kubelive-spinnaker-front50-6859bf64bb-cn9bd 0/1 CrashLoopBackOff 6 10m kubelive-spinnaker-gate-5468cccbc7-n2ncw 0/1 CrashLoopBackOff 6 10m $ $ kubectl logs kubelive-create-bucket-j97wn mc: <ERROR> Unable to initialize new config from the provided credentials. Get http://kubelive-minio:9000/probe-bucket-sign/?location=: dial tcp: lookup kubelive-minio on 10.96.0.10:53: no such host $
Easy one:
$ kubectl get pods -n=default NAME READY STATUS RESTARTS AGE pod-00 1/1 Running 0 51s pod-01 1/1 Running 0 1m pod-name 1/1 Running 0 18m $ $ kubectl get pods --sort-by=.status.startTime -n=default NAME READY STATUS RESTARTS AGE pod-name 1/1 Running 0 18m pod-01 1/1 Running 0 55s pod-00 1/1 Running 0 42s $ $ kubectl get pods --sort-by=.metadata.name -n=default NAME READY STATUS RESTARTS AGE pod-00 1/1 Running 0 2m pod-01 1/1 Running 0 2m pod-name 1/1 Running 0 20m $
Easy one:
$ kubectl create -f manifests/pod-01.yaml
Create a pod with some changes by edit without any chages of the original manifest file:
$ kubectl create -f manifests/pod-01.yaml --edit -o json
Create a deployment with external network access:
$ kubectl run nginx --image nginx --replicas=3 $ kubectl expose deployment nginx --port=80 --target-port=80 $ kubectl create -f manifests/ingress-nginx.yaml $ kubectl describe ingress Name: test-ingress Namespace: default Address: Default backend: nginx:80 (10.244.0.25:80,10.244.0.26:80,10.244.0.27:80) Rules: Host Path Backends ---- ---- -------- * * nginx:80 (10.244.0.25:80,10.244.0.26:80,10.244.0.27:80) Annotations: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal CREATE 17s nginx-ingress-controller Ingress default/test-ingress $
On this environment, ingress-nginx-controller is used and the setting is:
$ kubectl get services -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default-http-backend ClusterIP 10.102.0.178 <none> 80/TCP 2h ingress-nginx NodePort 10.101.145.191 <none> 80:31454/TCP,443:31839/TCP 2h $
So NodePort is configured and the host's 31454/TCP is proxied to 80/TCP of the ingress. You can get nginx page like:
$ curl http://localhost:31454 <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> ..
On this environment, etcd is running as a pod on kube-system namespace:
$ kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE etcd-k8s-v109-flannel-master 1/1 Running 0 1d .. $
The manifest is /etc/kubernetes/manifests/etcd.yaml and we can see the endpoint (http://127.0.0.1:2379) in this case:
$ sudo cat /etc/kubernetes/manifests/etcd.yaml
..
- command:
- etcd
- --data-dir=/var/lib/etcd
- --listen-client-urls=http://127.0.0.1:2379
- --advertise-client-urls=http://127.0.0.1:2379
..
Install etcdctl command (The ubuntu package is too old and doesn't support the snapshot feature):
$ mkdir foo $ cd foo $ wget https://github.com/coreos/etcd/releases/download/v3.2.18/etcd-v3.2.18-linux-amd64.tar.gz $ tar -zxvf etcd-v3.2.18-linux-amd64.tar.gz $ cd etcd-v3.2.18-linux-amd64
Create a snapshot:
$ ETCDCTL_API=3 ./etcdctl --endpoints http://127.0.0.1:2379 snapshot save snapshot.db Snapshot saved at snapshot.db $
Check pods where live and the node:
$ kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE nginx-foo-74cd78d68f-4jwsq 1/1 Running 0 1m 10.244.0.30 k8s-v109-flannel-master nginx-foo-74cd78d68f-5jl55 1/1 Running 0 1m 10.244.1.7 k8s-v109-flannel-worker nginx-foo-74cd78d68f-9cts2 1/1 Running 0 1m 10.244.1.5 k8s-v109-flannel-worker nginx-foo-74cd78d68f-9gtwx 1/1 Running 0 1m 10.244.1.6 k8s-v109-flannel-worker nginx-foo-74cd78d68f-b7zmx 1/1 Running 0 1m 10.244.1.4 k8s-v109-flannel-worker nginx-foo-74cd78d68f-d97pw 1/1 Running 0 1m 10.244.0.29 k8s-v109-flannel-master nginx-foo-74cd78d68f-j27qf 1/1 Running 0 1m 10.244.0.28 k8s-v109-flannel-master nginx-foo-74cd78d68f-j45c8 1/1 Running 0 1m 10.244.1.2 k8s-v109-flannel-worker nginx-foo-74cd78d68f-l4mwq 1/1 Running 0 1m 10.244.0.31 k8s-v109-flannel-master nginx-foo-74cd78d68f-wnb4c 1/1 Running 0 1m 10.244.1.3 k8s-v109-flannel-worker $ $ kubectl describe node k8s-v109-flannel-worker | grep Taints Taints: <none> $
Even if making the node tainted with NoSchedule, the pods still exist in the node:
$ kubectl taint nodes k8s-v109-flannel-worker key=value:NoSchedule node "k8s-v109-flannel-worker" tainted $ kubectl describe node k8s-v109-flannel-worker | grep Taints Taints: key=value:NoSchedule $ $ kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE nginx-foo-74cd78d68f-4jwsq 1/1 Running 0 5m 10.244.0.30 k8s-v109-flannel-master nginx-foo-74cd78d68f-5jl55 1/1 Running 0 5m 10.244.1.7 k8s-v109-flannel-worker nginx-foo-74cd78d68f-9cts2 1/1 Running 0 5m 10.244.1.5 k8s-v109-flannel-worker nginx-foo-74cd78d68f-9gtwx 1/1 Running 0 5m 10.244.1.6 k8s-v109-flannel-worker nginx-foo-74cd78d68f-b7zmx 1/1 Running 0 5m 10.244.1.4 k8s-v109-flannel-worker nginx-foo-74cd78d68f-d97pw 1/1 Running 0 5m 10.244.0.29 k8s-v109-flannel-master nginx-foo-74cd78d68f-j27qf 1/1 Running 0 5m 10.244.0.28 k8s-v109-flannel-master nginx-foo-74cd78d68f-j45c8 1/1 Running 0 5m 10.244.1.2 k8s-v109-flannel-worker nginx-foo-74cd78d68f-l4mwq 1/1 Running 0 5m 10.244.0.31 k8s-v109-flannel-master nginx-foo-74cd78d68f-wnb4c 1/1 Running 0 5m 10.244.1.3 k8s-v109-flannel-worker $
After making the node tainted with NoExecute, the pods go away from the node:
$ kubectl taint nodes k8s-v109-flannel-worker key=value:NoExecute node "k8s-v109-flannel-worker" tainted $ kubectl describe node k8s-v109-flannel-worker | grep Taints Taints: key=value:NoExecute $ kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE nginx-foo-74cd78d68f-48q4p 1/1 Running 0 17s 10.244.0.37 k8s-v109-flannel-master nginx-foo-74cd78d68f-4jwsq 1/1 Running 0 8m 10.244.0.30 k8s-v109-flannel-master nginx-foo-74cd78d68f-9q6f8 1/1 Running 0 17s 10.244.0.34 k8s-v109-flannel-master nginx-foo-74cd78d68f-d97pw 1/1 Running 0 8m 10.244.0.29 k8s-v109-flannel-master nginx-foo-74cd78d68f-j27qf 1/1 Running 0 8m 10.244.0.28 k8s-v109-flannel-master nginx-foo-74cd78d68f-jlxng 1/1 Running 0 17s 10.244.0.36 k8s-v109-flannel-master nginx-foo-74cd78d68f-k5rl9 1/1 Running 0 17s 10.244.0.32 k8s-v109-flannel-master nginx-foo-74cd78d68f-l4mwq 1/1 Running 0 8m 10.244.0.31 k8s-v109-flannel-master nginx-foo-74cd78d68f-sg52l 1/1 Running 0 17s 10.244.0.33 k8s-v109-flannel-master nginx-foo-74cd78d68f-vzspf 1/1 Running 0 17s 10.244.0.35 k8s-v109-flannel-master $
Remove the taint after this try:
$ kubectl taint nodes k8s-v109-flannel-worker key-
Encode a plain password with base64:
$ echo -n "mypassword" | base64 bXlwYXNzd29yZA== $
Create a secret:
$ cat manifests/secret-01.yaml apiVersion: v1 kind: Secret metadata: name: secret-01 type: Opaque data: password: bXlwYXNzd29yZA== $ $ kubectl create -f manifests/secret-01.yaml
Create a pod with the secret as a file:
$ kubectl create -f manifests/pod-using-secret-as-file.yaml
Confirm the password in the pod:
$ kubectl exec -it pod-using-secret-as-file /bin/bash (login the pod) # # ls /etc/foo/ password # cat /etc/foo/password mypassword
Create a pod with the secret as a variable:
$ kubectl create -f manifests/pod-using-secret-as-variable.yaml
Confirm the password in the pod:
$ kubectl exec -it pod-using-secret-as-variable /bin/bash (login the pod) # # echo $SECRET_PASSWORD mypassword
Create a deployment with a little old nginx (v1.7.9):
$ kubectl create -f manifests/nginx-deployment.yaml
$ kubectl describe deployment/nginx-deployment | grep Image
Image: nginx:1.7.9
$
Check the strategy (in this case (the default), that is RollingUpdate and the upgrade happens immediately just after setting the image):
$ kubectl describe deployment/nginx-deployment | grep StrategyType StrategyType: RollingUpdate $
Check the ReplicaSet name and the pod names:
$ kubectl get rs NAME DESIRED CURRENT READY AGE nginx-deployment-75675f5897 3 3 3 6s $ $ kubectl get pods NAME READY STATUS RESTARTS AGE nginx-deployment-75675f5897-9mhmv 1/1 Running 0 36s nginx-deployment-75675f5897-kpgtr 1/1 Running 0 36s nginx-deployment-75675f5897-plq92 1/1 Running 0 36s $
Set a newer nginx image (v1.9.1):
$ kubectl set image deployment/nginx-deployment nginx=nginx:1.9.1
$ kubectl describe deployment/nginx-deployment | grep Image
Image: nginx:1.9.1
$
Then check the status of the upgrade:
$ kubectl rollout status deployment/nginx-deployment Waiting for rollout to finish: 1 out of 3 new replicas have been updated... Waiting for rollout to finish: 1 out of 3 new replicas have been updated... Waiting for rollout to finish: 1 out of 3 new replicas have been updated... Waiting for rollout to finish: 2 out of 3 new replicas have been updated... Waiting for rollout to finish: 2 out of 3 new replicas have been updated... Waiting for rollout to finish: 2 old replicas are pending termination... Waiting for rollout to finish: 1 old replicas are pending termination... Waiting for rollout to finish: 1 old replicas are pending termination... deployment "nginx-deployment" successfully rolled out $
Conform new created ReplicaSet and pods. The old ReplicaSet doesn't have any pods now and new pods only exist:
$ kubectl get rs NAME DESIRED CURRENT READY AGE nginx-deployment-75675f5897 0 0 0 3m nginx-deployment-c4747d96c 3 3 3 1m $ $ kubectl get pods NAME READY STATUS RESTARTS AGE nginx-deployment-c4747d96c-fbsw6 1/1 Running 0 2m nginx-deployment-c4747d96c-gvqg2 1/1 Running 0 1m nginx-deployment-c4747d96c-jfvvl 1/1 Running 0 1m $
Check the history of a deployment:
$ kubectl rollout history deployment/nginx-deployment deployments "nginx-deployment" REVISION CHANGE-CAUSE 1 <none> 2 <none> $
Show the detail of each revision:
$ kubectl rollout history deployment/nginx-deployment --revision=2
deployments "nginx-deployment" with revision #2
Pod Template:
Labels: app=nginx
pod-template-hash=1520898311
Containers:
nginx:
Image: nginx:1.9.1
Port: 80/TCP
Environment: <none>
Mounts: <none>
Volumes: <none>
$
$ kubectl rollout history deployment/nginx-deployment --revision=1
deployments "nginx-deployment" with revision #1
Pod Template:
Labels: app=nginx
pod-template-hash=2710681425
Containers:
nginx:
Image: nginx:1.7.9
Port: 80/TCP
Environment: <none>
Mounts: <none>
Volumes: <none>
$
Rolling-back the deployment:
$ kubectl rollout undo deployment/nginx-deployment
Confirm the rolling-back succeeded:
$ kubectl rollout history deployment/nginx-deployment
deployments "nginx-deployment"
REVISION CHANGE-CAUSE
2 <none>
3 <none>
$ kubectl rollout history deployment/nginx-deployment --revision=3
deployments "nginx-deployment" with revision #3
Pod Template:
Labels: app=nginx
pod-template-hash=2710681425
Containers:
nginx:
Image: nginx:1.7.9
Port: 80/TCP
Environment: <none>
Mounts: <none>
Volumes: <none>
$
$ kubectl describe deployment/nginx-deployment | grep Image
Image: nginx:1.7.9
$
https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/
Check what service works on the cluster:
$ kubectl get services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 2d nginx-deployment ClusterIP 10.99.52.90 <none> 80/TCP 24s $
Create a pod for verifying DNS works:
$ kubectl create -f manifests/pod-busybox.yaml $ kubectl exec -it pod-busybox sh (login the pod) wget http://nginx-deployment Connecting to nginx-deployment (10.99.52.90:80) index.html 100% |********************************************************************************************************************************************| 612 0:00:00 ETA / # cat index.html <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> .. #
As the above, DNS works fine and the service nginx-deployment can be looked up from a pod as the same name.
A pod also can be looked up by "pod-ip-address.my-namespace.pod.cluster.local" like:
$ kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE pod-01 1/1 Running 0 19m 10.244.0.48 k8s-v109-flannel-master $ kubectl exec -it pod-busybox sh / # / # ping 10-244-0-48.default.pod.cluster.local PING 10-244-0-48.default.pod.cluster.local (10.244.0.48): 56 data bytes 64 bytes from 10.244.0.48: seq=0 ttl=64 time=0.033 ms 64 bytes from 10.244.0.48: seq=1 ttl=64 time=0.064 ms
https://kubernetes.io/docs/concepts/workloads/pods/init-containers/
Create a pod with init-containers:
$ kubectl create -f manifests/pod-init-container.yaml
Check the pod status, it waits for end of init process:
$ kubectl get pods NAME READY STATUS RESTARTS AGE pod-init-container 0/1 Init:0/2 0 30s $
Check logs of each containers, init-containers start on the order of the manifest. That means 2nd init-container also wait for 1st one's finishes:
$ kubectl logs pod-init-container -c myapp-container Error from server (BadRequest): container "myapp-container" in pod "pod-init-container" is waiting to start: PodInitializing $ $ kubectl logs pod-init-container -c init-myservice waiting for myservice nslookup: can't resolve 'myservice' Server: 10.96.0.10 Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local waiting for myservice nslookup: can't resolve 'myservice' Server: 10.96.0.10 Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local waiting for myservice $ $ kubectl logs pod-init-container -c init-mydb Error from server (BadRequest): container "init-mydb" in pod "pod-init-container" is waiting to start: PodInitializing $
Create services for making end of init process:
$ kubectl create -f manifests/services-for-init-containers.yaml service "myservice" created service "mydb" created $ $ kubectl get pods NAME READY STATUS RESTARTS AGE pod-init-container 0/1 PodInitializing 0 4m $ $ kubectl get pods NAME READY STATUS RESTARTS AGE pod-init-container 1/1 Running 0 5m $
Then the pod outputs the message to show the end as its command in the manifest:
$ kubectl logs pod-init-container The app is running! $
Create a daemonset:
$ kubectl create -f manifests/daemonset.yaml
Check the existence:
$ kubectl get ds -n kube-system NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE fluentd-elasticsearch 1 1 1 1 1 <none> 1m .. $
(Non-recommended way) Enforce kubelet boot on an environment with swap:
$ sudo diff -u /etc/systemd/system/kubelet.service.d/10-kubeadm.conf.orig /etc/systemd/system/kubelet.service.d/10-kubeadm.conf sudo: unable to resolve host k8s-v109-flannel-worker --- /etc/systemd/system/kubelet.service.d/10-kubeadm.conf.orig 2018-04-05 21:28:10.278748887 +0000 +++ /etc/systemd/system/kubelet.service.d/10-kubeadm.conf 2018-04-05 21:32:14.191449307 +0000 @@ -6,5 +6,6 @@ Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt" Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0" Environment="KUBELET_CERTIFICATE_ARGS=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki" +Environment="KUBELET_SWAP_ARGS=--fail-swap-on=false" ExecStart= -ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS +ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS $KUBELET_SWAP_ARGS $ $ sudo reboot
Swapoff on lxcfs (lxcfs is a simple file system to implement nest-cgroup for systemd environments which are defact init of Linux kernel today):
$ diff -u /usr/share/lxcfs/lxc.mount.hook.orig /usr/share/lxcfs/lxc.mount.hook
--- /usr/share/lxcfs/lxc.mount.hook.orig 2018-04-05 21:55:21.626302043 +0000
+++ /usr/share/lxcfs/lxc.mount.hook 2018-04-05 21:57:05.956673664 +0000
@@ -7,6 +7,7 @@
if [ -d /var/lib/lxcfs/proc/ ]; then
for entry in /var/lib/lxcfs/proc/*; do
[ -e "${LXC_ROOTFS_MOUNT}/proc/$(basename $entry)" ] || continue
+ [ $entry != "swap" ] || continue
mount -n --bind $entry ${LXC_ROOTFS_MOUNT}/proc/$(basename $entry)
done
fi