-
Notifications
You must be signed in to change notification settings - Fork 0
/
template.yaml
132 lines (120 loc) · 5.04 KB
/
template.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
#-------------------------------------------------------------------
# This is the SAM template for deploying a Cloud Formation Stack to AWS.
#
# -----> NOTE: The stack name is specified when you do "sam deploy --guided"
#
# The sam deploy command converts it over to a cloud formation template.
# You can find the files that will be deployed to AWS in the .aws-sam
# folder
#
# For info on the sam template schema, see:
# https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
#
# For details on the AWS::Serverless::Function properties, see:
# https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-function.html
#-------------------------------------------------------------------
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
This template deployes an AWS stack that contains the following resources:
1) A Lambda function to run one or more tsdat ingest pipelines
2) An S3 input bucket where incoming raw files should be placed.
All files placed in the input bucket will trigger the pipeline.
3) An S3 output bucket where the tsdat processed data will be stored.
######################################################################
# PARAMETERS (modify this section for general use)
#
# For basic usage, you should only need to modify 3 parameters when you
# create a new template:
# 1) the name of the lambda function
# 2) the name of the input S3 bucket
# 3) the name of the output S3 bucket
#
# SAM CLI will prompt for these parameters when you do a "sam deploy --guided"
# so that you can override them.
# These parameters will be passed to the lambda function via environment
# variables, which will be used to specify the storage configuration for
# tsdat pipelines.
#
# -----> NOTE: Search for #<==== in this document to identify these parameters.
#
# For more advanced changes (such as changing the input event triggers),
# see an AWS specialist.
######################################################################
Parameters:
# The source/input bucket where raw files will be placed
SourceBucketName:
Type: String
Default: "tsdat-pipeline-inputs" #<==== Change this if you want your input bucket to be named differently
# The destination bucket where tsdat pipeline outputs will be placed
DestinationBucketName:
Type: String
Default: "tsdat-pipeline-outputs" #<==== Change this if you want your output bucket to be named differently
LambdaFunctionName:
Type: String
Default: "tsdat-pipeline-lambda" #<==== Change this if you want your lambda function to be named differently
# The logging level to use
LoggingLevel:
Type: String
Default: INFO
######################################################################
# STACK RESOURCES (advanced users only)
######################################################################
# This section describes the AWS resources that will be created from this template.
Resources:
# 1) Lambda Function
LambdaFunction:
Type: AWS::Serverless::Function
Properties:
PackageType: Image
FunctionName: !Ref LambdaFunctionName
Description: Lambda function to run tsdat pipelines from s3 trigger
Environment:
Variables:
STORAGE_BUCKET: !Ref DestinationBucketName
LOG_LEVEL: !Ref LoggingLevel
Timeout: 900 # Longest allowable lambda timeout is 15 min or 900 s
MemorySize: 10240 # Largest allowable memory is 10GB (specified in MB)
Policies:
- S3CrudPolicy:
BucketName: !Sub "${SourceBucketName}"
- S3CrudPolicy:
BucketName: !Sub "${DestinationBucketName}"
- AWSLambdaBasicExecutionRole
Events:
FileUploaded:
Type: S3
Properties:
Bucket: !Ref SourceS3Bucket
Events: s3:ObjectCreated:*
Metadata:
Dockerfile: Dockerfile
DockerContext: ./lambda_function
DockerTag: python3.8-v1
# 2) Input S3 Bucket
SourceS3Bucket:
Type: 'AWS::S3::Bucket'
Properties:
BucketName: !Ref SourceBucketName
# 3) Output S3 Bucket
DestinationS3Bucket:
Type: 'AWS::S3::Bucket'
Properties:
BucketName: !Ref DestinationBucketName
# 4) Explicit permission for input bucket to trigger our lambda function
LambdaInvokePermission:
Type: 'AWS::Lambda::Permission'
Properties:
FunctionName: !GetAtt LambdaFunction.Arn
Action: 'lambda:InvokeFunction'
Principal: 's3.amazonaws.com'
SourceAccount: !Sub ${AWS::AccountId}
SourceArn: !GetAtt SourceS3Bucket.Arn
Outputs:
# The optional Outputs section is used to specify the properties to return whenever you view your stack's properties
# via the AWS CLI: "aws cloudformation describe-stacks"
# In this case, we make sure that we show the Amazon Resource Name (ARN) for the generated Lambda Function so it
# is easy to find. ARNs are used by Amazon to uniquely identify each resource
LambdaFunction:
Description: "Lambda Function ARN"
Value: !GetAtt LambdaFunction.Arn