The instructions in this repo will:
- Provide
terraform
to create an EKS cluster - Deploy K8s Metrics Server
- Deploy K8s Dashboard
- Install Splunk Operator for K8s
- Deploy a Standalone deployment of Splunk Enterprise on EKS
Before you begin, you will need the following:
- AWS Account
- AWS IAM Permissions Example Here
- AWS CLI Installed
- Kubernetes CLI
- wget installed
- Terraform 0.14.11
Clone the repo
git clone git@github.com:anthonygrees/eks_splunk_k8s_demo.git
cd eks_splunk_k8s_demo
Initiate the Terraform
terraform init -update
Apply the terraform to create the EKS cluster
terraform apply
This process should take approximately 10 minutes. Upon successful application, your terminal prints the outputs
Apply complete! Resources: 51 added, 0 changed, 0 destroyed.
Outputs:
cluster_endpoint = "https://80CD543ECDB40DCF1AC9xxxxxxx9710F.sk1.eu-north-1.eks.amazonaws.com"
cluster_id = "rees-eks-kXpc1xxxx"
cluster_name = "rees-eks-kXpcxxxx"
cluster_security_group_id = "sg-03614b30a2xxxxxd7"
config_map_aws_auth = [
{
"binary_data" = tomap(null) /* of string */
....
....
Now that you've provisioned your EKS cluster, you need to configure kubectl.
Run the following command to retrieve the access credentials for your cluster and automatically configure kubectl.
aws eks --region $(terraform output -raw region) update-kubeconfig --name $(terraform output -raw cluster_name)
The Kubernetes Metrics Server, used to gather metrics such as cluster CPU and memory usage over time, is not deployed by default in EKS clusters.
Download and unzip the metrics server by running the following command.
wget -O v0.3.6.tar.gz https://codeload.github.com/kubernetes-sigs/metrics-server/tar.gz/v0.3.6 && tar -xzf v0.3.6.tar.gz
Deploy the metrics server to the cluster by running the following command.
kubectl apply -f metrics-server-0.3.6/deploy/1.8+/
Verify that the metrics server has been deployed. If successful, you should see something like this.
kubectl get deployment metrics-server -n kube-system
The following command will schedule the resources necessary for the dashboard.
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta8/aio/deploy/recommended.yaml
Your output will look like this
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created
Now, create a proxy server that will allow you to navigate to the dashboard from the browser on your local machine. This will continue running until you stop the process by pressing CTRL + C
.
kubectl proxy
Your output will be:
Starting to serve on 127.0.0.1:8001
You can reach the Kubernetes dashboard here - http://127.0.0.1:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/
To use the Kubernetes dashboard, you need to create a ClusterRoleBinding and provide an authorization token.
In another terminal (do not close the kubectl proxy process), create the ClusterRoleBinding resource.
kubectl apply -f https://raw.githubusercontent.com/anthonygrees/eks_splunk_k8s_demo/master/kubernetes-dashboard-admin.rbac.yaml
Then, generate the authorization token.
kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep service-controller-token | awk '{print $1}')
Select "Token" on the Dashboard UI then copy and paste the entire token you receive into the dashboard authentication screen to sign in. You are now signed in to the dashboard for your Kubernetes cluster.
Navigate to the "Cluster" page by clicking on "Cluster" in the left navigation bar. You should see a list of nodes in your cluster.
A Kubernetes cluster administrator can install and start the Splunk Operator by running:
kubectl apply -f https://github.com/splunk/splunk-operator/releases/download/1.0.1/splunk-operator-install.yaml
After the Splunk Operator starts, you'll see a single pod running within your current namespace:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
splunk-operator-75f5d4d85b-8pshn 1/1 Running 0 5s
Let’s ask our operator pod to build us a standalone demo instance to play with!
kubectl -n default apply -f s1.yaml
You will now see the sevices running:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 172.20.0.1 <none> 443/TCP 115m
splunk-operator-metrics ClusterIP 172.20.51.109 <none> 8383/TCP,8686/TCP 85m
splunk-s1-standalone-headless ClusterIP None <none> 8000/TCP,8088/TCP,8089/TCP,9997/TCP 67s
splunk-s1-standalone-service ClusterIP 172.20.91.91 <none> 8000/TCP,8088/TCP,8089/TCP,9997/TCP 67s
Get your Pod details
kubectl get pods
NAME READY STATUS RESTARTS AGE
splunk-default-monitoring-console-0 1/1 Running 0 46m
splunk-operator-5845f6d45c-m4stq 1/1 Running 0 133m
splunk-s1-standalone-0 1/1 Running 0 48m
You can use a simple network port forward to open port 8000 for Splunk Web access:
kubectl port-forward splunk-s1-standalone-0 8000
To access our Spunk Operator for Kubernetes built instance, we will need to grab the secret which contains the HEC token and password, among other secrets the Operator syncs to the Splunk instance.
kubectl -n default get secret splunk-default-secret -o yaml
Your output will be:
apiVersion: v1
data:
hec_token: QUI4NUFBREMtRDkyNy0yNzJBLUFDNDAtNjRCQ0M2QzQ4RjI2
idxc_secret: Mm51S3BTWDBveE43WklHQm9pZnF5VE5s
pass4SymmKey: eWlvRUc5Nzk0cGJEWkF2UFJ5VEtVV1VY
password: UXJiRG5IUVJud1hDYTdlR0pTc2x6YWt3
shc_secret: ZXRYQzNZV3VYOHRrbHBuWUFOeVpMZDBn
kind: Secret
Note
: The output is encoded. The above is for demonstration purposes only and the secrets are not real !
To Decode
your passwords use the following:
kubectl get secret splunk-default-secret -o go-template=' {{range $k,$v := .data}}{{printf "%s: " $k}}{{if not $v}}{{$v}}{{else}}{{$v | base64decode}}{{end}}{{"\n"}}{{end}}'
Your Output will be decoded like this:
kubectl get secret splunk-default-secret -o go-template=' {{range $k,$v := .data}}{{printf "%s: " $k}}{{if not $v}}{{$v}}{{else}}{{$v | base64decode}}{{end}}{{"\n"}}{{end}}'
hec_token: AB85AADC-D927-272A-AC40-64BCC6C48F26
idxc_secret: 2nuKpSX0oxN7ZIGBoifqyTNl
pass4SymmKey: yioEG9794pbDZAvPRyTKUWUX
password: QrbDnHQRnwXCa7eGJSslzakw
shc_secret: etXC3YWuX8tklpnYANyZLd0g
Note
: The output is now decoded. The above is for demonstration purposes only and the secrets are not real ! You need to retrieve your own !
Log into Splunk Enterprise at http://localhost:8000 using the admin
account with the password.
To delete your standalone deployment, run:
kubectl delete standalone s1
Watch as the deployment is terminated:
kubectl -n default get pods -w
Destroy the EKS Cluster
terraform destroy
If you get the error:
Error: Get "http://localhost/api/v1/namespaces/kube-system/configmaps/aws-auth": dial tcp 127.0.0.1:80: connect: connection refused
then run the command
terraform state rm module.eks.kubernetes_config_map.aws_auth