Skip to content

Latest commit

 

History

History
166 lines (129 loc) · 12.1 KB

README.md

File metadata and controls

166 lines (129 loc) · 12.1 KB
Raw

Release CI License Registry Source

terraform-aws-snowflake-loader-setup

A Terraform module for preparing snowflake for loading snowplow data. Should be used after the terraform-snowflake-target.

Prerequisites

Authentication for the service user is required for the snowflake terraform provider - follow this tutorial to obtain snowflake connection details:

Parameter Description
account The account name.
username A snowflake user to perform resource creation.
region Region for the snowflake deployment.
role Needs to be ACCOUNTADMIN or similar.
private_key_path Path the private key.

Usage

Applying module directly

  1. Fill variables in terraform.tfvars.tmpl and copy it to terraform.tfvars.
  2. Using environment variables for authentication as described in here. Fill the template in snowflake_provider_vars.sh and run source ./snowflake_provider_vars.sh to set up your local environment.
  3. Run terraform init
  4. Run terraform apply

Using the module from another module

Pass authentication parameters directly to Snowflake provider.

provider "snowflake" {
  username         = "my_user"
  account          = "my_account"
  region           = "us-west-2"
  role             = "ACCOUNTADMIN"
  private_key_path = "/path/to/private/key"
}

module "snowflake_target" {
   source = "snowplow-devops/target/snowflake"
   
   name               = "snowplow"
   snowflake_password = "example_password"
}

module "snowflake_setup" {
   source = "snowplow-devops/snowflake-loader-setup/aws"

   name                  = "snowplow"
   stage_bucket          = "snowplow-oss-bucket"
   aws_account_id        = "0000000000"
   snowflake_database    = module.snowflake_target.snowflake_database
   snowflake_event_table = module.snowflake_target.snowflake_event_table
   snowflake_file_format = module.snowflake_target.snowflake_file_format
   snowflake_schema      = module.snowflake_target.snowflake_schema
   snowflake_user        = module.snowflake_target.snowflake_user
}

Requirements

Name Version
terraform >= 1.0.0
snowflake 0.45.0

Providers

Name Version
snowflake 0.45.0

Modules

No modules.

Resources

Name Type
snowflake_database_grant.loader resource
snowflake_file_format_grant.loader resource
snowflake_integration_grant.loader resource
snowflake_role.loader resource
snowflake_role_grants.loader resource
snowflake_schema_grant.loader resource
snowflake_stage.folder_monitoring resource
snowflake_stage.transformed resource
snowflake_stage_grant.folder_monitoring resource
snowflake_stage_grant.transformed resource
snowflake_storage_integration.integration resource
snowflake_table_grant.loader resource
snowflake_warehouse.loader resource
snowflake_warehouse_grant.loader resource

Inputs

Name Description Type Default Required
aws_account_id AWS account id for the role to be linked with storage integration. Role itself won't be created by this module string n/a yes
name A name which will be prepended to the created resources string n/a yes
snowflake_database Snowflake database name string n/a yes
snowflake_event_table Snowflake events table name string n/a yes
snowflake_file_format Snowflake file format name string n/a yes
snowflake_schema Snowflake schema name string n/a yes
snowflake_user Snowflake username string n/a yes
stage_bucket Name of the S3 bucket which will be used as stage by Snowflake string n/a yes
folder_monitoring_enabled Folder monitoring loading bool false no
override_folder_monitoring_stage_url Override monitoring stage url, if not set it will be defaulted "s3://${var.stage_bucket_name}/${var.name}/shredded/v1" string "" no
override_iam_loader_role_name Override integration iam role name, if not set it will be var.name with -snowflakedb-load-role suffix string "" no
override_snowflake_loader_role Override loader role name in snowflake, if not set it will be uppercase var.name with "_LOADER_ROLE" suffix string "" no
override_snowflake_loader_user Override loader user name in snowflake, if not set it will be uppercase var.name with _LOADER_USER suffix string "" no
override_snowflake_wh_name Override warehouse name, if not set it will be defaulted to uppercase var.name with "_WAREHOUSE" suffix string "" no
override_transformed_stage_url Override transformed stage url, if not set it will be defaulted "s3://${var.stage_bucket_name}/${var.name}/monitoring" string "" no
snowflake_wh_auto_resume Whether to enable auto resume which makes automatically resume the warehouse when any statement that requires a warehouse is submitted bool true no
snowflake_wh_auto_suspend Time period to wait before suspending warehouse number 60 no
snowflake_wh_size Size of the Snowflake warehouse to connect to string "XSMALL" no

Outputs

Name Description
aws_iam_storage_integration_role_arn AWS storage integration IAM role ARN
aws_iam_storage_integration_role_name AWS storage integration IAM role name
aws_iam_storage_integration_user_arn AWS storage integration IAM user ARN
aws_s3_bucket_name Name of the S3 bucket which will be used as stage by Snowflake
aws_s3_folder_monitoring_stage_url AWS bucket url of folder monitoring stage
aws_s3_transformed_stage_url AWS bucket url of transformed stage
aws_storage_external_id AWS external id
snowflake_loader_role Snowflake role for loading snowplow data
snowflake_monitoring_stage_name Name of monitoring stage
snowflake_transformed_stage_name Name of transformed stage
snowflake_warehouse Snowflake warehouse name

Copyright and license

The Terraform Snowflake Loader Setup project is Copyright 2022-2022 Snowplow Analytics Ltd.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this software except in compliance with the License.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.