Kubernetes Basics.txt

Kubernetes is an open source system for automating deployment, scaling, and management of containerized applications.

At heart it is a "container orchestration" tool

With containers we can run a variety of software components across a "cluster" of generic servers; instead of running a component on (1) server and if it dies we lose everything we can have a cluster and we can distribute the components (High Availabity)

We can also scale based on resource needs

Server		              Server
Pod (Container)     Pod (Container)  
Pod (Container)  	  Pod (Container) 
Pod (Container)     Pod (Container)  


Kube Master	
------------
Docker  // container runtime
Kubeadm // automates cluster setup
Kubelet // agent that manages processes
Kubectl // CLI tool to interact w/ the cluster
Control Plane // cluster controller

Kube Node (n)
--------------
Docker
Kubeadm
Kubelet
Kubectl

"Pods" are the smallest atomic unit (like cells) in the Kubernetes ecosystem. They consist of 
one or more containers, storage resources, and have a unique IP in the cluster network 

Node
  Pod (10.244.0.1)
     Container1

Node
  Pod (10.244.0.2)
     Container1
     Container2


Control Node		       
------------------
kube-api-server
other core components

Worker Node
-----------------
kubelet
kube-proxy
Pod 
Pod


KIND 3-Node Cluster Setup (DEV)
--------------------------------------------
kind create cluster --name lowk8s --wait 5m --config kind-multi-node.yaml (in ~/.kube)
kind delete cluster --name lowk8s

kubectl create deployment autodor --image=lwooden/testrun:firstry
kubectl get deployments
kubectl get pods
kubectl describe pod <POD NAME>
kubeclt describe service <deployment name>

kubectl expose deployment autodor  --type=Loadbalancer --port 8080 // does not work

To test connectivity to a application I have to:
 - find out what node the deployment was assigend to
 - ssh to that node
 - curl http://<NODE-IP>:<PORT>/ping
 
 
 
 Kubeadm 3-Node Cluster Setup
 --------------------------------------------
 
 
 
 
AWS EKS 1-Node Cluster Setup (DEV)
----------------------------------------------
eksctl create cluster --name=lowk8 --nodes=1 --node-type=m5.large --zones=us-east-1a,us-east-1b
aws eks update-kubeconfig --name <CLUSTER_NAME> // updates kube-config w/ credentials
eksctl delete cluster --region=us-east-1 --name=lowk8


// Accessing AWS Services Via Pods

IRSA - IAM Roles for Service Accounts (Available on High-Side)
- require OIDC Provider setup
- would need to be setup per cluster

eksctl utils associate-iam-oidc-provider --region=us-east-1 --cluster=lowk8 --approve

Pod Identity (Not Available on High-Side)
- does not require OIDC Provider setup
- require pod identity add-on to be running in the cluster
- require association via the aws-cli/eksctl (aws eks create-pod-identity-association --cluster-name my-cluster --role-arn arn:aws:iam::111122223333:role/my-role --namespace default --service-account my-service-account)

eksctl create addon --name eks-pod-identity-agent --cluster lowk8
eksctl create addon --name aws-ebs-csi-driver --cluster lowk8
eksctl update addon -f ebsPodIdentityConfig.yaml // with podIdentity setup
eksctl get podidentityassociation --cluster lowk8 --output json // check pod identity association



Azure AKS 1-Node Cluster Setup (DEV)
----------------------------------------------
deploy resource group, vnet, subnets, etc using pulumi
az aks create --resource-group lowsandbox-dev-rg --name lowk8-aks --node-count 1 --enable-addons monitoring --generate-ssh-keys

az aks create -n lowk8-aks -g lowsandbox-dev-rg --network-plugin azure --enable-managed-identity -a ingress-appgw --appgw-name aks-ingress-gateway --appgw-subnet-cidr "10.225.0.0/16" --node-count 1 --generate-ssh-keys

az aks get-credentials --resource-group lowsandbox-dev-rg --name lowk8-aks // updates kube-config w/ credentials
az aks delete --name lowk8-aks --resource-group lowsandbox-dev-rg


Google GKS 1-Node Cluster Setup (DEV)
----------------------------------------------
deploy vpc, subnets, etc using pulumi
gcloud services enable container // enable GKE api 
gcloud container clusters create lowk8-gke --cluster-version=1.27 --num-nodes=1 --network="low-sandbox-dev-vpc" --subnetwork="dev-private" --zone=us-east1-b --addons=HttpLoadBalancing,HorizontalPodAutoscaling

gcloud container clusters create-auto CLUSTER_NAME \ # creates auto-pilot cluster
    --region REGION \
    --project=PROJECT_ID
    
gcloud container clusters get-credentials lowk8-gke --region=us-east1-b
gcloud container clusters delete lowk8-gke --region=us-east1-b


Static Pod Location (On Node)
--------------------------
/etc/kubernestes/manifest 
**If you move one of these files to /tmp temporarily, the static pod/component will cease to run; after the file is restored, the pod/component will be rebuilt


CNI Config Locaiton (On Node)
--------------------------
find /etc/cni/net.d/ ## find CNI info here


Upgrading Cluster Components
----------------------------------------
After upgrading kubeadm, kubelet, and trying to execute the upgrade command, if there is no kube.conf on the server then that means it hasn't been initialized
Head over to the master node and get the join token command -> kubeadm token create --print-join-command
Go back to the node and perform the join; the ca.crt and kubelet.conf will be generated at that time

Back-up Etcd (Snapshot)
----------------------------------
Inspect the etcd pod to get cert details (Node)
    k describe pod/etcd -n kube-system
    
Perform the snapshot (Node)
    ETCDCTL_API=3 etcdctl --endpoints $ENDPOINT snapshot save <FILENAME>
    
Verify the snapshot (Node)
    ETCDCTL_API=3 etcdctl --write-out=table snapshot status <FILENAME>


Restore Etcd From Snapshot
---------------------------------------
Stop Etcd (Node)
    if service, stop it; if static-pod, move it out from /etc/kubernetes/manifest

Delete /var/lib/etcd (Node)
    rm -rf /var/lib/etcd

Restore from snapshot (Node)
    ETCDCTL_API=3 etcdctl --data-dir /var/lib/etcd snapshot restore <FILENAME>

Start Etcd (Node)
    if service, stop it; if static-pod, move it back into /etc/kubernetes/manifest)
    
Verify etcd is up and running (kubectl/client)
    k describe pod/etcd -n kube-system
    
    
Using Helm for Manifest File Management
--------------------------------------------------------
- Helm is a package/release manager aimed to help make deploying container based applications to Kubernetes simpler

helm create example-chart // scaffolds a new helm project
helm repo add <name> <url>
helm repo update
helm search repo vweb --versions
helm show values kiamol/vweb --version 1.0.0
helm install --set namespace=test --set image.tag=e3e3140ad30ac965c5aef04e811fb629651d9fbb graphql-api alaffia-charts/graphql-api
helm install --upgrade --set namespace=test --set image.tag=e3e3140ad30ac965c5aef04e811fb629651d9fbb graphql-api alaffia-charts/graphql-api
helm ls [--all-namespaces] // list charts that have been deployed/released
helm -n <namespace> get values <release-name> // check the values you passed to a previous deployment
helm pull <REPO> --version <string> // pulls down a chart in tar.gz format
helm uninstall graphql-api // uninstall a release

// pull down a helm chart and transform it into a ready to deploy kubernetes manifest file
helm template ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx --version ${CHART_VERSION} --namespace ingress-nginx > nginx-ingress.${APP_VERSION}.yaml 

helm dep up // downloads all dependent sub-charts referenced in Chart.yml into the current project


Using Helmfile to Manage Helm Charts
--------------------------------------------------------
- Helmfile is a higher level abstraction built on top of Helm. It makes managing Helm charts within your project simpler as well as offering synchornization options between your project and a cluster

helmfile sync // references helmfile.yaml in the working directory
helmfile delete


Using Werf to Perform Deployments
------------------------------------------------
// Build
- Define a werf.yaml file in the root of the project

project: samplenodeapi
configVersion: 1
---
image: samplenodeapi
dockerfile: Dockerfile


- Specify the werf.yaml image specification in the Helm chart

spec:
      containers:
        {{- if eq .Release.Namespace "default"}}
        - name: sample-node-service
        {{ else }}
        - name: "sample-node-service-{{ .Release.Namespace }}"
        {{- end }}
          image: {{ .Values.werf.image.samplenodeapi }}
          
werf build --repo public.ecr.aws/mock-node-api/lowsandbox
werf build --repo public.ecr.aws/mock-node-api/lowsandbox --platform linux/amd64 || linux/arm64

// Deploy
werf converge --repo public.ecr.aws/mock-node-api/lowsandbox

// Clean-Up


Common Commands
----------------------------

// Working with Contexts
kubectl config view
kubectl config current-context 
kubectl config use-context arn:aws:eks:us-east-1:548883107094:cluster/alaffia-eks-cluster-prod

kubectl config get-clusters || delete-cluster <CLUSTER_NAME>
kubectl config get-contexts || delete-context <CLUSTER_NAME>
kubectl config get-users || delete-user <CLUSTER_NAME>

kubectl config unset contexts.low-aks-cluster
kubectl config unset users.clusterUser_Low-Playground_low-aks-cluster
kubectl config unset clusters.low-aks-cluster


// Getting Help
kubectl api-resources // list all resources currently available to you
kubectl explain <resource> // list information about fields associated with the resource


// Get All
kubectl get all -n <NAMESPACE>
kubectl get all --all-namespaces
kubectl get namespaces
kubectl get [ pods | svc | sa | cm | secrets | roles | rolebinding | clusterrole | clusterrolebindings | ingress ] --all-namespaces


// Exporting Manifests from Existing Resources
kubectl get service hellworldexample-helloworld -o yaml > service.yaml 


// Nodes
kubectl get nodes
kubectl get nodes --show-labels
kubectl describe nodes <NODE>
kubectl label nodes <NODE> purpose: reports-services
kubectl drain <NODE> --ignore-daemonsets // drains a node of it's running pods/containers and any daemonsets that are running on it and prevents anymore pods from being scheduled
kubectl uncordon <NODE> // makes the node available to accept pod scheduling again


// Namespaces
kubectl create namespace <namespace>
kubectl get namespaces // gets all namespaces


// Pods
kubectl get pods  -n <NAMESPACE>
kubectl get pods --all-namespaces
kubectl get pods -o wide // show ip address and assinged node for each pod
kubectl get pods --all-namepsaces
kubectl get pods -A --sort-by=.metadata.name
kubectl get pod -l app=fluentbit-logging -o jsonpath='{.items[0].status.containerStatuses[*].name}' // get container names in a pod

kubectl run <NAME> --image <IMAGE> --dry-run=client -o yaml > pod.yml


// Service Accounts
kubectl get sa  -n <NAMESPACE> 
kubectl create sa tom -n <NAMESPACE>


// Persistent Volumes
kubectl get pv // gets list of persistent volumes


// Persistent Volume Claims
kubectl get pvc // gets list of persistent volume claims
kubectl edit pvc my-pvc --record


// Services
kubectl get svc // get list of services
kubectl get endponts <service-name> // get list of backend pods the service is currently routing to


// Ingress
kubectl get ingress [ -n <NAMESPACE> ] // displayed ingress rules that exists in the namespace


// ConfigMap
kubectl get cm [ -n <NAMESPACE> ]
kubectl  get cm --all-namespaces
kkubectl describe cm <NAME>

kubectl create cm sleep-config-literal --from-literal=kiamol.section='4.1' // create a config map in your cluster
kubectl create cm graphql-env --from-env-file=./graphql.env  // create a config based on the contents of another .env file 


// Deployments
kubectl get deployments || deploy
kubectl create deployment my-deployment --image=nginx --dry-run -o yaml > deployment.yml // generates a sample manifest imperatively
kubectl apply -f deployment.yml
kubectl scale deployment my-deployment --replicas 5 --record // scales replicaset to 5
kubectl rollout restart deploy graphql-api // does a rolling restart on containers in deployment

// Service Accounts

kubectl get [ serviceaccount | sa ] iam-test
kubectl describe sa iam-test


// Daemon Sets
kubectl get ds


// Stateful Sets
kubectl get sts


// Roles
kubectl get roles [ -n <NAMESPACE> ]
kubectl get clusterroles

kubectl create role testrole --verbs list,create ----resource=secrets,configmap [ -n <NAMESPACE> ]


// Rolebindings
kubectl get rolebindings [ -n <NAMESPACE> ]
kubectl get clustetrolebindings [ -n <NAMESPACE> ]

kubectl create rolebinding testrolebinding --role testrole --serviceaccount <NAMESPACE>:test -n project-hamster // shorthand for creating role-bindings

kubectl auth can-i create secret --as system:serviceaccount:project-hamster:processor -n project-hamster // command to do authorization checks


// Port-Forwarding
kubectl port-forward pod/hello-kiamol 8080:80 // port forward to pod resource
kubectl port-forward deploy/hello-kiamol-2 8080:80 // port forward to deployment resource

kubectl apply -f pod.yml // send local file to the K8 API for deployment
kubectl apply -f https://github.com/pod.yaml  // send remote file to the K8 API for deployment


// Remote Execution
kubectl exec -it hello-kiamol-2-7f6dd54b9b-j6zvl (pod name) -- sh // get interactive shell to your container
kubectl exec -it hello-kiamol-2-7f6dd54b9b-j6zvl --namespace test -- sh // specific namespace
kubectl exec deploy/sleep -- printenv // get container environment variables
kubectl exec deploy/graphql-api -- sh -c 'nslookup processing-api' // perform an nslookup on a clusterIP service inside a container to make sure it resolves
kubectl exec deploy/graphql-api -- ping -c  1 processing-api // perform a ping from inside a container


// Logging
kubectl logs --tail=2 hello-kiamol-2-7f6dd54b9b-j6zvl // get logs from your container
kubectl logs -l app=graphql-container --follow // trail logs as they come in w/ label app=graphql-container


// Events + Cluster Information
kubectl get events -A --sort-by=.metadata.creationTimestamp // get events from all namespaces
kubectl cluster-info dump


// Resource Monitoring
kubectl top pod // gets raw resource usage metrics for pods
kubectl top pod --sort-by cpu
kubectl top pod -l app=graphql-container 

kubectl top node


// Copying Files
kubectl cp hello-kiamol-2-7f6dd54b9b-j6zvl:/tmp/low.txt /Users/low/low.txt // move a file from pod to your local computer

**NOTE**: By default Kubernetes does not PULL images if they are already in the local cache. I may need to add "imagePullPolicy: Always" in the manifest to force this to happen in the event I have a container update that has the same tag name. This will ensure I get the latest image with the latest code

aws sts get-caller-identity // get your current user



<------ DEEP DIVE -------->


Services
-------------
- provides a way to expose an application runnings as a set of Pods
- clients do need to be concerned with how many pods are running or which pod they are communicating with

Client Request --> Service --> Backend Pods

[ Service Types ]
- ClusterIP: expose applications INSIDE the cluster network (needs to be consumed by other pods)
- NodePort: expose applications OUTSIDE the cluster (needs to be consumed by external clients)
    **NOTE** ports exposed using this type are opened up ALL NODES and the CONTROL PLANE
- Loadbalancer: expose applications OUTSIDE the cluster (needs to be consumed by external clients) by using the cloud platforms load balancer functionality
- ExternalName

[ DNS for Services ]
- a services FULLY QUALIFIED DOMAIN NAME can be used to reach the service from ANY NAMESPACE
- a services SHORT NAME can be used to reach the service from within the SAME NAMESPACE it was created in

service-name.namespace.svc.cluster.local -> Structure
my-service.default.svc.cluster.local -> FQDN (external namespace reference)
my-service -> SHORT NAME (internal namespace reference)

Deployments
-----------------


Volumes
-------------
- file systems within containers are ephemeral (they exist only for the lifetime of the container)
- volumes provide a way to allow data to exist outside of the lifecycle of a container
- each class of storage has a particular type

Common volume types:
- hostPath: stores data in a specific dir on the host worker node
- emptyDir: stores data in dynamically created location on the host worker node; tied to the lifespan of the pod

(2) classes of storage:

Volumes 

spec:
  containers:
  - name: buzybox
    image: buzybox
    volumeMounts:
    - name: my-volume
      mountPath: /output
  volumes:
  - name: my-volume
     hostPath:
      path: /data


Persistent Volumes - allows you to treat storage as an abstract resource to be consumed by pods (EBS, EFS, NFS, etc)

Persistent Volume Claim - allows you to carve out storage that is provided by the associated Persistent Volume
**PVCs are scoped to a namespace**


// Concept
Storage Class -> cluster resource
  PV -> cluster resource
    PVC -> namespace specific resource (consumed by pods)
    
    
// Create the Storage Class
apiVersion: storage.k8s.io/v1 
kind: StorageClass
metadata:
  name: gp3
provisioner: kubernetes.io/aws-ebs # AWS
parameters:
  type: gp3
  fsType: ext4
  
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast
provisioner: kubernetes.io/gce-pd # GKE
parameters:
  type: pd-ssd
  replication-type: none

// Create the persistent volume
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv
spec:
  storageClassName: localdisk // different types of storage services available on a particular platform
  persistentVolumeReclaimPolicy: Recycle || Retain || Delete   // what happens when no one is using the this storage
  capacity:
      storage: 1Gi
  accessModes:
      - ReadWriteOnce
  hostPath:
      path: /var/output
      
      
 // Create the claim for the persistent volume  claim    
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
spec:
  storageClassName: localdisk
  accessModes:
      - ReadWriteOnce
  resources:
      requests:
          storage: 100Mi
     
          
// Reference the claim in the Pod Spec
spec:
  containers:
  - name: buzybox
    image: buzybox
    volumeMounts:
    - name: pv-storage
      mountPath: /output
  volumes:
  - name: pv-storage
    persistentVolumeClaim:
        claimName: my-pvc   // reference claim created above
        
        
Scheduling
---------------
- process of assigning pods to a node so kubelet can run them
- scheduler is a core k8 service running in "kube-system"
- checks against each nodes resource requests and available resources before placing a pod

// nodeSelector
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: redis
    image: redis
  nodeSelector:
      purpose: reports-services // assigns node based on label
  nodeName: worker-1 // assings nodes based on explicit node name
  
  
Taints, Tolerations, and Node Selectors
----------------------------------------------------
- a mechanism for flagging a node to say it isn’t suitable for general work
- have a key-value pair just like a label
- also have an effect, which tells the scheduler how to treat the node
- use taints to identify nodes that are different from the rest

kubectl taint nodes --all diskType=hdd:NoSchedule

spec:                     
 containers:
   - name: sleep
     image: kiamol/ch03-sleep      
 tolerations:                  # Lists taints this Pod is happy with
     - key: "diskType"      # The key, value, and effect all need 
       operator: "Equal"       # to match the taint on the node.
       value: "hdd"
       effect: "NoSchedule"


- to guarantee that Pods only run on certain nodes, use a NodeSelector

spec:
 containers:
   - name: sleep
     image: kiamol/ch03-sleep      
 nodeSelector:                           # The Pod will run only on nodes
   kubernetes.io/arch: zxSpectrum        # that match this CPU type.


Affinity and Anti-Affinity
------------------------------
- provides a rich way of expressing preferences or requirements to the scheduler
- can claim an affinity to certain nodes to ensure Pods land on those nodes
- uses a node selector but with a match expression rather than a simple equality check. Match expressions support multiple clauses, so you can build much more complex requirements

affinity:                           # Affinity expresses a requirement
 nodeAffinity:                     # or a preference for nodes.
   requiredDuringSchedulingIgnoredDuringExecution:
     nodeSelectorTerms:
       - matchExpressions:               # Match expressions work on
         - key: kubernetes.io/arch       # labels, and you can supply
           operator: In                  # a list of values that 
           values:                       # should match or not match.
             - amd64
       - matchExpressions:               # Multiple match expressions
         - key: beta.kubernetes.io/arch  # work as a logical OR.
           operator: In
           values:
             - amd64

  
  

Daemon-Sets
-------------------
- automatically runs a copy of a Pod on each node
- launches Pods on new nodes that are added to cluster immediately
- respect normal scheduling rules; if node is in violation of any of the constraints, it will not schedule the pod there

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: my-daemonset
spec:
  selector:
    matchLabels:
      app: fluentbit // all pods with this label will be managed by my-daemonset
  template:
    metadata:
      labels:
        app: fluentbit // ensure container has this label so it can be picked up by my-daemonset
    spec:
      containers:
      - name: nginx
        image: nginix:1.19.0
        
        
Static-Pods
---------------
- a pod that is managed directly by the nodes kubelet process and not the k8 API
- when a static-pod is created k8 creates a mirror-pod which is a representaiton of the static-pod
- a mirror-pod can be viewed from the k8 API but you cannot perform any actions against it

  
Networking (Plugins)
-----------------------------

CNI Plugins - plugins that provide network connectivity between pods according to standards set by the Kubernetes network model; 
Calico is one of the most popular general purpose CNI plugins

Nodes will remain in a NOTREADY state untill a network plugin is installed

[ Network Model ]

Node
- Pod 192.168.100.2 (unique IP address in the entire cluster)
- Pod 192.168.100.3

Any pod can reach any other pod simply using the Pod's IP address (over the virtual network) regardless of what node they are running on


Network Policy
---------------------

- an object that allows you to control the flow of network comms to and from pods
- BY DEFAULT pods are considered "non-isolated" and OPEN to all traffic;
- when a network policy is applied, the pod is deemed "isolated" and ONLY traffic that is permitted by the policy is allowed

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: my-network-policy
  namespace: np-test
spec:
  podSelector:
    matchLabels:
      app: db
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - ipBlock:
        cidr: 172.17.0.0/16
        except:
        - 172.17.1.0/24
    - namespaceSelector: # traffic from pods in the same namespace that have the defined label
        matchLabels:
          app: nginix
    - podSelector: # traffic from pods with the specified label
        matchLabels:
          app: nginix
    ports:
    - protocol: TCP 
      port: 80
  egress:
  - to:
    - ipBlock:
        cidr: 10.0.0.0/24
    ports:
    - protocol: TCP
      port: 5978




DNS
------
- handled by the coreDNS service located in the kube-system namespace
- queries for services within the same namepsace work using short and fully qualified names
- queries for services in a different namespace fail using short name
- queries for services in a different namespace work using fully qualified names

// Structure
pod-ip-address.namespace-name.pod.cluster.local
service-name.namespace-name.svc.cluster.local

192-168-10-100 # short name
192-168-10-100.default.pod.cluster.local # fqdn

user-db # short 
user-db.default.svc.cluster.local #fqdn





Managing RBAC
---------------------

There (4) role-based access related objects are:

Role - defines permissions within a particular namespace (e.g viewing logs for a pod in the default namespace)

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: pod-reader
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["pods"]
  verbs: ["get", "watch", "list"]
  
RoleBinding - links users to roles

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods // name of the binding
  namespace: default
subjects:
  // you can specify more than one "subject"
- kind: User
  name: jane
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role || ClusterRole
  name: pod-reader // role to bind to
  apiGroup: rbac.authorization.k8s.io

Clusterrole - defines permissions regardless of namespace (e.g  viewing logs for any pod in the cluster)

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: secret-reader
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "watch", "list"]
  
ClusterRoleBinding - links users to clusterroles


Service Accounts
-----------------------
- service accounts are accounts used by container processes inside of Pods to authenticate and use the k8 API
- they can be bound to clusterRoles or ClusterRoleBindings to control access to the API
- scoped to a particular namespace
- this also extends to AWS where pods can use IAM roles as service accounts to access other AWS services


Managing Container Resources
-------------------------------------------

Requests - amount of resources that we think a container may use; this value is only considered during scheduling in order to determine if a node has enough resources prior to pod placement; pods can use more than what is defined here

Limits - amount of resources a pod cannot exceed, otherwise the container runtime will kill it

apiVersion: v1
kind: Pod
metadata:
  name: frontend
spec:
  containers:
  - name: app
    image: images.my-company.example/app:v4
    resources:
      requests:
        cpu: "250m" # 1/1000th of 1 CPU (e.g. 1000m = 1 CPU)
        memory: "64Mi"
      limits:
        cpu: "500m" 
        memory: "128Mi"
        
        
        
Container Probing
----------------------

[ LivenessProde ] 
- a task that is run periodically in order to determine whether your app is healthy or not

spec:
  containers:
  - name: busybox
    image: busybox
    command: ['sh', '-c', 'while true; do sleep 3600; done']
    livenessProbe:
      exec:
        command: ["echo", "Hello World"]
      initialDelaySeconds: 5 // wait 5 secs after container starts before executing
      periodSeconds: 5 // run every 5 secs
      

[ StartupProbe ]  
- a task that is run only at startup time in order to determine whether your app is healthy or not

spec:
  containers:
  - name: nginx
    image: nginx:1.19.1
    startupProbe:
        httpGet:
            path: /
            port: 80
        failureThreshold: 30 // has to fail 30 times before deemed unhealthy
        periodSeconds: 10
        

[ ReadinessProbe ]  
- a task that is run only at startup in order to determine whether your app is ready to recevie requests or not; ideal when your container is dependent on a downstream container being availble (e.g. a databse); the pod will not enter the READY state until the readinessProbe has completed successfully

spec:
  containers:
  - name: nginx
    image: nginx:1.19.1
    readinessProbe:
        httpGet:
            path: /
            port: 80
        failureThreshold: 5
        periodSeconds: 5
        
        
        
Multi-Container Pods
---------------------------
- pods that contains more than (1) container
- best practice is to have each container run in its own pod
- containers share the same network and storage (volume in a pod)

apiVersion: v1
kind: Pod
metadata:
  name: sidecar-pod
spec:
  containers:
   // 1st container
  - name: busybox1
    image: busybox
    command: ['sh', '-c', 'while true; do echo log data > /output/output.log; sleep 3600; done']
    volumeMounts:
    - name: sharedvol
       mountPath: /output  # same volume; different location in the container
  // 2nd container
  - name: sidecar
    image: busybox
    command: ['sh', '-c', 'tail -f /input/output.log']
    volumeMounts:
    - name: sharedvol
       mountPath: /input  # same volume; different location in the container
    // shared volumes created in the pod but accessible by all containers
    volumes:
    - name: sharedvol
       emptyDir: {}
       
$kubectl logs sidecar-pod -c sidecar
 log data



Init Containers
-------------------
- run once to completion during the startup process of a pod
- they run once and in the order they are defined



Troubleshooting the Cluster
------------------------------------

Check node status
  kubectl get nodes
  kubectl describe node <nodename>

Check status of k8 servies on a node
  ssh into worker node
  systemctl status kubelet
  systemctl status docker
  
Check key system pods in kube-system namespace
  kubectl get pods -n kube-system
  kubectl describe pod <podname> -n kube-system
  

Check logs for key k8 services
  journalctl -u kubelet | docker
  less /var/log/kube-apiserver.log
  less /var/log/kube-scheduler.log
  less /var/log/kube-controller-manager.log
  
  
Troubleshooting Apps/Containers
------------------------------------------

Get Pod status
  kubectl get pods
  kubectl desctibe pod <podname>

Remote into a pod and run commands 
  kubectl exec <pod> -- sh // opens a shell inside of the container running in this pod
  kubectl exec <pod> -c <container-name> -- sh // if a pod has more than one container running
  
  
Check container logs
  kubectl logs <pod-name>
  kubectl logs -l app=graphql-container --follow // by label
  
  
Cluster Logging
---------------------
By default kubernetes only keeps (2) instances of logs in /var/log/pods:
1. logs for the current running instance of a container/pod
2. logs for the previously restarted instance of a container/pod

They looks as follows:
0.log - previous instance
1.log - current instance
  
  
Troubleshooting Networking Inside the Cluster
-----------------------------------------------------------

Deploy a container inside of your cluster to run commands from
  deploy a contianer using image nicolaka/netshoot
  kubectl exec netshoot -- sh
  
  








Restart Policies
--------------------

Always - default
OnFailure - only restart on failure
Never - never restart


Add-Ons
------------
- provide additional functionality

Examples include:
- Kubernetes Metrics Server
- Calico (Networking)



Backing Up etcd
----------------------
- etcd is the backend data storage solution k8 cluster
- k8 objects, applications and configs are stored in etcd

etcdctl --endpoints $ENDPOINT snapshot save <file-name>
etcdctl snapshot restore <file-name> // creates another cluster and restores based on the snapshot



High Availability Control Plane
--------------------------------------

Stacked etcd - etcd runs on the same server as the other components on the control plane nodes

Control Plane Node 1
- control plane components 
- kube-api
- etcd

Control Plane Node 2
- control plane components 
- kube-api-server
- etcd

External etcd - etcd runs on seperate servers and the control plane nodes talk to them over the network

Control Plane Node 1
- control plane components 
- kube-api-server

Control Plane Node 2
- control plane components 
- kube-api-server

etcd Server 1
- etcd

etcd Server 2
- etcd

etcd Server 3
- etcd





Resource Hierarchy
-------------------------

Services
     - LoadBalancer: integrates w/ an external loadbalancer which sends traffic to the cluster; traffic is associate with the back end pod based on port and label selector
     - ClusterIP: cluster wide ip that any pod can access; only works WITHIN the cluster; best used for comms between pods (internal)
     - NodePort: every node in the cluster listens on the port specified and sends traffic to the target port on the pod; can route external traffic into pods as well (like a loadbalancer)
     - ExternalName: creates a local alias in kube-dns that pods can use in to resolve external endpoints; the local alias will be used in the application as opposed to the true external domain name
  Deployments
            Pods
            
          
            

Commands
--------------------

kubeadm version 
sudo kubeadm init --pod-network-cidr=10.244.0.0/16

kubectl version // check Kubernetes version ($Major: and $Minor:)
kubectl get nodes
kubectl get pods // displays default namespace
kubectl get pods -A // displays all namespaces
kubectl get pods -n kube-system // kube-system namespace
kubectl get pods -o wide // more detail
kubectl get deployments // displays list of deployments and their state
kubectl get service // displays list of services
kubectl get all

kubectl describe node $node_name
kubectl describe pods $pod_name
kubectl describe deployment $deployment_name

kubectl delete node $node_name
kubectl delete pod $pod_name
kubectl delete deployment $deployment_name

kubectl create namespace $name


Base Components
--------------------

[Kubernetes Control Plane]
etcd - provides distributed, synchronized data storage for the cluster state
kube-apiserver - serves the Kubernetes API, interface for the cluster
kube-controller-manager - bundles several components into one package
kube-scheduler - schedules pods to run on nodes; 

[Worker Nodes]
kubelet - agent that runs on each node and interfaces with the Kubernetes API and docker; runs as a systemd service (systemctl status kubelet
kube-proxy - handles the network layer inter/intra node traffic; runs on each node


Deployments
-------------------

"Deployments" are "objects" that gives us a way to organize and maintain our pods
- Scaling: specify the number of replicas we want and that number of pods will be created
- Rolling Updates: gradually replace existing containers with a new update without taking them all out (no down-time)
- Self-Healing: if a pod happens to fail or get destroy, the deployment will immediately spin up another one to replace it in order to achieve the desired state (replica set)


# Deployment File Example

cat <<EOF | kubectl create -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx // tag name in Kubernetes
spec:
  replicas: 2 // desired state (number of Pods to run at all times)
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx    
        image: nginx:1.15.4   // container to pull from Docker Repo
        ports:
        - containerPort: 80   // port to expose
EOF


Services
---------------

"Services" provide an abstraction layer between upstream services and the underlying pods in order to get some information. 
The upstream service will send a request to the "service layer" and the service layer will distribute the request to the backend pods. 
This way no matter what is happening at the Pod Level, upstream services will be able to get their requests processed

Pod 1 <- [Service Layer] -> Resource

# Service File Example

cat << EOF | kubectl create -f -
kind: Service
apiVersion: v1
metadata:
  name: nginx-service
spec:
  selector:
    app: nginx // determines which pods to route traffic to
  ports:
  - protocol: TCP
    port: 80  // port to route traffic to
    targetPort: 80
    nodePort: 30080 // listening port of the service layer
  type: NodePort
EOF


Install
--------
1. Install docker - All Nodes

// Get GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - 

// Add Repo
sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"

sudo apt-get update

sudo apt-get install -y docker-ce=18.06.1~ce~3-0~ubuntu

// Prevent package from being updated
sudo apt-mark hold docker-ce

// Verify
sudo docker version

2. Install Kubernetes Components - All Nodes

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

cat << EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF

sudo apt-get update

sudo apt-get install -y kubelet=1.12.7-00 kubeadm=1.12.7-00 kubectl=1.12.7-00

sudo apt-mark hold kubelet kubeadm kubectl

3. Verify - All Nodes

kubeadm version 

4. Bootstrap Cluster - Master

sudo kubeadm init --pod-network-cidr=10.244.0.0/16

//Setup Local Kubectl
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

//Verify
kubectl version # check client and server response

5. Join Worker Nodes to Cluster

sudo kubeadm join $some_ip:6443 --token $some_token --discovery-token-ca-cert-hash $some_hash

6. Verify - Master

kubectl get nodes (won't be in READY STATE yet; networking not setup yet)

7. Set up Networking - Master (Flannel)

//Execute on all nodes
echo "net.bridge.bridge-nf-call-iptables=1" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

//Grab config file from internet to apply to cluster
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/bc79dd1505b0c8681ece4de4c0d86c5cd2643275/Documentation/kube-flannel.yml

//Verify Nodes are in READY State
kubectl get nodes

//Verify Back-End Services in Kube-System Namespace
kubectl get pods -n kube-system


kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/bc79dd1505b0c8681ece4de4c0d86c5cd2643275/Documentation/kube-flannel.yml