This repository is an example how to use custom metrics in GKE. If you want yo read more about it:
Stackdriver Adapter is an implementation that uses Custom Metrics API and External Metrics API using Cloud Monitoring as a backend. Its purpose is to enable pod autoscaling based on Cloud Monitoring custom metrics.
These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project.
- git
- make
- python
- pip
- virtualenv
- vscode
- skaffold
- terraform
- docker
- minikube
- kubectl
- kubectx
- gcloud
To deloy this project you will need a Google Cloud account, create here.
This tutorial uses the following billable components of Google Cloud:
To generate a cost estimate based on your projected usage, use the pricing calculator.
When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Clean up.
First of all you need to clone this repository:
git clone https://github.com/claick-oliveira/k8s-custom-metrics
First let's check the architecture that we will create.
#TODO: ## Add the architecture diagram
Now that we know about the architecture and resources, let's create. First we need to connect our shell to the gcloud
:
gcloud auth login
Now that we connected:
cd terraform
terraform apply
This project you need to fill some variables:
- gcp_project_name: The GCP project ID
- gcp_region: The GCP region
- gcp_zone: The GCP availability zone
First we need to connect in our GKE:
gcloud container clusters get-credentials <CLUSTER_NAME> --region <REGION> --project <PROJECT_ID>
To be easy and skaffold use the correct environment, let's configure kubectx
:
Get the name of the environment:
kubectx
Now change the name for staging:
kubectx staging=<YOUR ENVIRONMENT>
Now we need to create a new namespace:
kubectl create namespace custom-metrics
To allow the stackdriver adapter connect on the monitoring api, wee need to create an IAM service account and give some permissions:
gcloud iam service-accounts create stackdriver-adapter-sa --project=PROJECT_ID
gcloud projects add-iam-policy-binding PROJECT_ID \
--member "serviceAccount:stackdriver-adapter-sa@PROJECT_ID.iam.gserviceaccount.com" \
--role "roles/monitoring.viewer"
gcloud iam service-accounts add-iam-policy-binding stackdriver-adapter-sa@PROJECT_ID.iam.gserviceaccount.com \
--role roles/iam.workloadIdentityUser \
--member "serviceAccount:PROJECT_ID.svc.id.goog[custom-metrics/custom-metrics-stackdriver-adapter]"
Remeber to replace PROJECT_ID for your project id, example: my-project-111111
Now we need to create a kubernetes service account and create an annotation to link the kubernetes service account with the IAM service account:
kubectl create serviceaccount custom-metrics-stackdriver-adapter --namespace custom-metrics
kubectl annotate serviceaccount custom-metrics-stackdriver-adapter \
--namespace custom-metrics \
iam.gke.io/gcp-service-account=stackdriver-adapter-sa@PROJECT_ID.iam.gserviceaccount.com
Remeber to replace PROJECT_ID for your project id, example: my-project-111111
Now we will deploy the stackdriver adapter:
kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/master/custom-metrics-stackdriver-adapter/deploy/production/adapter_new_resource_model.yaml
Ok nice! So, now wee need to create a IAM service account and kubernetes service account to our applications:
gcloud iam service-accounts create pubsub-sa --project=PROJECT_ID
gcloud projects add-iam-policy-binding PROJECT_ID \
--member "serviceAccount:pubsub-sa@PROJECT_ID.iam.gserviceaccount.com" \
--role "roles/pubsub.subscriber" \
--role "roles/pubsub.publisher"
gcloud iam service-accounts add-iam-policy-binding pubsub-sa@PROJECT_ID.iam.gserviceaccount.com \
--role roles/iam.workloadIdentityUser \
--member "serviceAccount:PROJECT_ID.svc.id.goog[custom-metrics/pubsub-sa]"
kubectl create serviceaccount pubsub-sa --namespace custom-metrics
kubectl annotate serviceaccount pubsub-sa \
--namespace custom-metrics \
iam.gke.io/gcp-service-account=pubsub-sa@PROJECT_ID.iam.gserviceaccount.com
Remeber to replace PROJECT_ID for your project id, example: my-project-111111
Nice, we finished the setup.
To test the setup we will create a deployment:
kubectl apply -f ./kubernetes/wi-test.yaml
Remeber to be in the root path of the project
Now let's connect in this pod and check if the setup it's working:
kubectl exec -it workload-identity-test \
-c workload-identity-test \
--namespace custom-metrics \
-- /bin/bash
Now execute this command:
curl -H "Metadata-Flavor: Google" http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/email
If the service accounts are correctly configured, the IAM service account email address is listed as the active (and only) identity. This demonstrates that by default, the Pod acts as the IAM service account's authority when calling Google Cloud APIs.
Now you can exit the container and delete the deployment:
exit
kubectl delete -f ./kubernetes/wi-test.yaml
To run the applications you need to execute, but you need to be in the root folder:
skaffold run --default-repo us-central1-docker.pkg.dev/<PROJECT_ID>/custom-metrics
Before you run this command remeber to create a file deployment.yaml based on the file deployment.yaml.template In this file you need to replace the variable <PROJECT_ID> to your project id
Now the application publisher will send messages to the Pub/Sub Topic and the application subscriber will read the messages. But the publisher is faster then the publisher and the messages will increse fast. Wait some minutes and create the HPA to increase the number of subscribers:
kubectl apply -f kubernetes/subscriber-hpa.yaml
And reduce to 0 the number of publishers:
kubectl scale deploy -n custom-metrics --replicas=0 publisher
If you want to check how many subscribers do you have:
kubectl get deployment subscriber --namespace custom-metrics
In this project we'll use PEP 8 as style guide.
To clean the files generated as coverage, builds, env you can use:
make cleanfull
skaffold delete
To clean the infrastructure:
cd terraform
terraform destroy
Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.
- Claick Oliveira - Initial work - claick-oliveira
See also the list of contributors who participated in this project.
This project is licensed under the GNU General Public License - see the LICENSE file for details