Skip to content

Latest commit

 

History

History
100 lines (74 loc) · 3.21 KB

README.md

File metadata and controls

100 lines (74 loc) · 3.21 KB
Raw

AWS Step Functions Fargate Batch Example

  • Terraform Fargate batch job example using Step Functions
  • based on the official sample Manage a Container Task (ECS, SNS)
  • plus adding ECR image and CloudWatch Logs to check the batch output
  • demo using execution input to set environment variable for container
  • also using SNS to notify the result of the task

How to run

Run the official project

  • When you follow the official doc, it will run CloudFormation
  • and you can find the details in CloudFormation > Stacks
  • official-sample will create same as the sample project by Terraform
  • you can run this official-sample independently from terraform/main.tf
  • This will run echo command in the image from docker hub
cd official-sample
terraform apply

Run this example

  • first create Fargate cluster/task and Step Functions state machine
cd terraform

# after check and update variables.tf or locals
terraform apply

export BATCH_REPO_NAME=$(terraform output batch_repository_name)
export BATCH_REPO_URL=$(terraform output batch_repository_url)
export BATCH_REPO_NAME2=$(terraform output batch_repository_name2)
export BATCH_REPO_URL2=$(terraform output batch_repository_url2)
  • next build and push simple docker image for Fargate batch job
cd batch-job                        # success rate 50%
docker build -t my-python-batch .   # build docker image
docker run --rm my-python-batch     # check if it is working
docker run --rm --env NAME=foo my-python-batch  # hello foo!

cd batch-job2                        # success rate 80%
docker build -t my-python-batch2 .   # build docker image
docker run --rm my-python-batch2     # check if it is working

# push to ECR
$(aws ecr get-login --region ap-northeast-1 --no-include-email)
docker tag my-python-batch ${BATCH_REPO_URL}:latest
docker push ${BATCH_REPO_URL}:latest

docker tag my-python-batch2 ${BATCH_REPO_URL2}:latest
docker push ${BATCH_REPO_URL2}:latest

# check the result
aws ecr list-images --repository-name ${BATCH_REPO_NAME}
aws ecr list-images --repository-name ${BATCH_REPO_NAME2}
  • go to Step Functions > State Machines and click Start Execution
  • in New execution, the Input will be set the environment variable
{
    "Comment": "Insert your JSON here"
}
  • go to CloudWatch > Logs > Filter: /ecs/${BATCH_REPO_NAME}
  • click the Log Streams to see the batch job output

Notes

  • how to check Fargate cluster arn
aws ecs describe-clusters --clusters tf-example-app

Reference