Skip to content

Latest commit

 

History

History
306 lines (232 loc) · 10.5 KB

README.md

File metadata and controls

306 lines (232 loc) · 10.5 KB
Raw

Autoscaler tool for Cloud Spanner

Autoscaler

Set up the Autoscaler in a per-project deployment using Terraform
Home · Scaler function · Poller function · Forwarder function · Terraform configuration
Per-Project · Centralized · Distributed

Table of Contents

Overview

This directory contains Terraform configuration files to quickly set up the infrastructure for your Autoscaler with a per-project deployment.

In this deployment option, all the components of the Autoscaler reside in the same project as your Spanner instances.

This deployment is ideal for independent teams who want to self-manage the infrastructure and configuration of their own Autoscalers. It is also a good entry point for testing the Autoscaler capabilities.

Architecture

architecture-per-project

For an explanation of the components of the Autoscaler and the interaction flow, please read the main Architecture section.

The per-project deployment has the following pros and cons:

Pros

  • Design: this option has the simplest design.
  • Configuration: The control over scheduler parameters belongs to the team that owns the Spanner instance, therefore the team has the highest degree of freedom to adapt the Autoscaler to its needs.
  • Infrastructure: This design establishes a clear boundary of responsibility and security over the Autoscaler infrastructure because the team owner of the Spanner instances is also the owner of the Autoscaler infrastructure.

Cons

  • Maintenance: with each team being responsible for the Autoscaler configuration and infrastructure it may become difficult to make sure that all Autoscalers across the company follow the same update guidelines.
  • Audit: because of the high level of control by each team, a centralized audit may become more complex.

Before you begin

In this section you prepare your project for deployment.

  1. Open the Cloud Console

  2. Activate Cloud Shell
    At the bottom of the Cloud Console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Cloud SDK already installed, including the gcloud command-line tool, and with values already set for your current project. It can take a few seconds for the session to initialize.

  3. In Cloud Shell, clone this repository

    git clone https://github.com/cloudspannerecosystem/autoscaler.git

  4. Export variables for the working directories

    export AUTOSCALER_DIR="$(pwd)/autoscaler/terraform/per-project"

Preparing the Autoscaler Project

In this section you prepare your project for deployment.

  1. Go to the project selector page in the Cloud Console. Select or create a Cloud project.

  2. Make sure that billing is enabled for your Google Cloud project. Learn how to confirm billing is enabled for your project.

  3. In Cloud Shell, set environment variables with the ID of your autoscaler project:

    export PROJECT_ID=<INSERT_YOUR_PROJECT_ID>
gcloud config set project "${PROJECT_ID}"

  4. Choose the region and zone and App Engine Location where the Autoscaler infrastructure will be located.

    export REGION=us-central1
export ZONE=us-central1-c
export APP_ENGINE_LOCATION=us-central

  5. Enable the required Cloud APIs

    gcloud services enable iam.googleapis.com \
  cloudresourcemanager.googleapis.com \
  appengine.googleapis.com \
  firestore.googleapis.com \
  spanner.googleapis.com \
  pubsub.googleapis.com \
  cloudfunctions.googleapis.com \
  cloudbuild.googleapis.com \
  cloudscheduler.googleapis.com

  6. Create a service account that will be used by Terraform to create all the resources in your infrastructure.

    gcloud iam service-accounts create terraformer --display-name "Terraform service account"

  7. Give the project owner role to the service account

    gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
  --member "serviceAccount:terraformer@${PROJECT_ID}.iam.gserviceaccount.com" \
  --role roles/owner

  8. Create a service account key file

    gcloud iam service-accounts keys create \
  --iam-account "terraformer@${PROJECT_ID}.iam.gserviceaccount.com" "${AUTOSCALER_DIR}/key.json"

  9. If your project does not have a Firestore instance yet, create one

    gcloud app create --region="${APP_ENGINE_LOCATION}"
gcloud alpha firestore databases create --region="${APP_ENGINE_LOCATION}"

Deploying the Autoscaler

  1. Set the project ID, region and zone in the corresponding Terraform environment variables

    export TF_VAR_project_id="${PROJECT_ID}"
export TF_VAR_region="${REGION}"
export TF_VAR_zone="${ZONE}"

  2. If you want to create a new Spanner instance for testing the Autoscaler, set the following variable. The spanner instance that Terraform creates is named autoscale-test.

    export TF_VAR_terraform_spanner=true

    On the other hand, if you do not want to create a new Spanner instance because you already have an instance for the Autoscaler to monitor, set the name name of your instance in the following variable

    export TF_VAR_spanner_name=<INSERT_YOUR_SPANNER_INSTANCE_NAME>

    For more information on how to make your Spanner instance to be managed by Terraform, see Import your Spanner instances

  3. Change directory into the Terraform per-project directory and initialize it.

    cd "${AUTOSCALER_DIR}"
terraform init

  4. Import the existing AppEngine application into Terraform state

    terraform import module.scheduler.google_app_engine_application.app "${PROJECT_ID}"

  5. Create the Autoscaler infrastructure. Answer yes when prompted, after reviewing the resources that Terraform intends to create.

    terraform apply -parallelism=2

If you are running this command in Cloud Shell and encounter errors of the form "Error: cannot assign requested address", this is a known issue in the Terraform Google provider, please retry with -parallelism=1

Importing your Spanner instances

If you have existing Spanner instances that you want to import to be managed by Terraform, follow the instructions in this section.

  1. List your spanner instances

    gcloud spanner instances list

  2. Set the following variable with the instance name to import

    SPANNER_INSTANCE_NAME=<YOUR_SPANNER_INSTANCE_NAME>

  3. Create a Terraform config file with an empty google_spanner_instance resource

    echo "resource \"google_spanner_instance\" \"${SPANNER_INSTANCE_NAME}\" {}" > "${SPANNER_INSTANCE_NAME}.tf"

  4. Import the Spanner instance into the Terraform state.

    terraform import "google_spanner_instance.${SPANNER_INSTANCE_NAME}" "${SPANNER_INSTANCE_NAME}"

  5. After the import succeeds, update the Terraform config file for your instance with the actual instance attributes

    terraform state show -no-color "google_spanner_instance.${SPANNER_INSTANCE_NAME}" \
  | grep -vE "(id|num_nodes|state|timeouts).*(=|{)" \
  > "${SPANNER_INSTANCE_NAME}.tf"

If you have additional Spanner instances to import, repeat this process.

Importing Spanner databases is also possible using the google_spanner_database resource and following a similar process.

Next steps

Your Autoscaler infrastructure is ready, follow the instructions in the main page to configure your Autoscaler.