How are we supposed to use pre_deployments with db_migrations?

[jetaggart]

Hey, thanks for releasing pre/post deployment support. Unfortunately, I'm having a hard time understanding exactly how these are supposed to be used to run migrations. Is there any documentation/examples that show how to use this feature? Right now, I only see that it says "point to a buildspec", but I have no idea what is supposed to be in that buildspec.

I've come up with this as a buildspec:

version: 0.2

phases:
  install:
    commands:
      - echo "cd into $CODEBUILD_SRC_DIR"
      - cd $CODEBUILD_SRC_DIR
      - wget -q https://ecs-cli-v2-release.s3.amazonaws.com/copilot-linux-v1.32.1 -O copilot-linux
      - chmod +x ./copilot-linux
  build:
    commands:

  post_build:
    commands:
      - command=$(./copilot-linux task run --generate-cmd redacted/staging/db-migrations 2>&1 >/dev/null)
      - command="${command//copilot/./copilot-linux}"
      - command="$command -n db-migrations --follow"
      - sh -c "$command"

However, this relies on a job being deployed that's schedule to none. The issue I'm having is the job is not updated as apart of the pipeline so the job is just running whatever I deployed originally, at least for the pre_deployments phase. It also feels bad to have a job deployed with no schedule just to run these. Is there anyway to just get the dockerfile and run it directly with vpc/secrets set up?

What is the recommended way to accomplish this? Thanks!

[iamhopaul123]

Hello @jetaggart.

It also feels bad to have a job deployed with no schedule just to run these. Is there anyway to just get the dockerfile and run it directly with vpc/secrets set up?

This is very true. I think the problem right now is we don't have a manifest for one-off task and we don't have feature parity as job. However, I think your setup is the one I would have right now for DB migration as well.

The issue I'm having is the job is not updated as apart of the pipeline so the job is just running whatever I deployed originally, at least for the pre_deployments phase.

I don't understand why you think it has this issue: if your job is in the same workspace, it should be rebuilt and the image would've been pushed to the ECR at the build stage. Then, the latest image for the job should be used in the pre_deployments phase. Did I miss anything important 🤔

[jetaggart]

Hmm, okay I'll keep playing with it, I see what you're saying.

[jetaggart]

I figured out the problem with my code. Because I'm using the --generate-cmd directly in the pipeline (which I'd prefer to pick up all the complicated parts dynamically, especially if the job definition changes) it outputs the latest deployed job's image tag as the --image argument, which is always the previously deployed job since the pipeline has not deployed yet in pre_deployments. In your example, you're explicitly looking for latest which mitigates this. I ended up with this code:

version: 0.2

phases:
  install:
    commands:
      - echo "cd into $CODEBUILD_SRC_DIR"
      - cd $CODEBUILD_SRC_DIR
      - wget -q https://ecs-cli-v2-release.s3.amazonaws.com/copilot-linux-v1.32.1 -O copilot-linux
      - chmod +x ./copilot-linux
  post_build:
    commands:
      - command=$(./copilot-linux task run --generate-cmd APP_NAME_CHANGE_ME/staging/db-migrations 2>&1 >/dev/null)
      - command="${command//copilot/./copilot-linux}"
      - command=$(echo "$command" | sed 's/--image AWS_ACCOUNT_ID_CHANGE_ME.dkr.ecr.us-east-1.amazonaws.com\/APP_NAME_CHANGE_ME\/db-migrations:[^ ]*/--image AWS_ACCOUNT_ID_CHANGE_ME.dkr.ecr.us-east-1.amazonaws.com\/APP_NAME_CHANGE_ME\/db-migrations:latest/')
      - command="$command -n db-migrations --follow"
      - echo $command
      - sh -c "$command"

That way I gsub out the tag with latest and it seems to work. All that said, it's not super obvious how to do this as a newcomer approaching copilot. A couple thoughts that would make this way simpler:

1. adding an example buildspec for the pre/post migrations would be awesome
2. being able to run a task with latest tag, or with one simple command copilot task run --from-job db-migrations --follow
3. allowing a multistage build in the same env or maybe a job as a deployments? It'd be awesome to do this:

  - name: staging
    deployments:
      db-migrations:
      gateway:
        depends_on: [db-migrations]

where db-migrations is a deployed job you want to run one off and complete before deploying gateway, similar to an init container in k8s.

Thanks for the quick responses!

[jetaggart]

@iamhopaul123 We're running into another issue with this method. If I'm pulling the latest tag from ECR, there's a problem with race conditions or stale deploys. For example, I have two pipelines, one against the main branch for staging and one against the production branch for production. We merge into production and push to deploy. The issue is if we start a production pipeline and main also triggers, we end up building multiple images as expected, however if the staging image gets built and tagged as latest after the production image has been built but BEFORE the production db_migrations task has been started, we'll get the image latest built by main/staging which is wrong. This is very bad because we'd be migrating tables that are not ready/verified to go to prod.

Another way this may manifest is if we want have main way ahead of prod and we want to deploy a specific commit back in time, there's a potential that it pulls the latest image due to the same race condition from concurrent builds as described above.

Overall this feels very hacky, is there a way to pull the specific tag that we just built in the pipeline instead of relying on latest?

Or, is there a way to tag each image build as latest-staging or latest-production?

[jetaggart]

I figured out how to solve for this, although still feels hacky. In the pipeline build phase you can change it to go off the SHA of the commit when tagging the image, so change this line in the pipeline buildspec

tag=$(echo ${CODEBUILD_BUILD_ID##*:}-$env | sed 's/:/-/g' | rev | cut -c 1-128 | rev)

to

tag=$(echo ${CODEBUILD_RESOLVED_SOURCE_VERSION##*:}-$env | sed 's/:/-/g' | rev | cut -c 1-128 | rev)

and in your migrations buildspec, use that image instead of latest:

version: 0.2

phases:
  install:
    commands:
      - echo "cd into $CODEBUILD_SRC_DIR"
      - cd $CODEBUILD_SRC_DIR
      - wget -q https://ecs-cli-v2-release.s3.amazonaws.com/copilot-linux-v1.32.1 -O copilot-linux
      - chmod +x ./copilot-linux
  post_build:
    commands:
      - tag=$(echo ${CODEBUILD_RESOLVED_SOURCE_VERSION##*:}-staging | sed 's/:/-/g' | rev | cut -c 1-128 | rev)
      - command=$(./copilot-linux task run --generate-cmd redacted/staging/db-migrations 2>&1 >/dev/null)
      - command="${command//copilot/./copilot-linux}"
      - command=$(echo "$command" | sed "s/--image AWS_ACCOUNT_ID_CHANGE_ME.dkr.ecr.us-east-1.amazonaws.com\/redacted_change_me\/db-migrations[^ ]*/--image AWS_ACCOUNT_ID_CHANGE_ME.dkr.ecr.us-east-1.amazonaws.com\/redacted_change_me\/db-migrations:$tag/")
      - command="$command -n db-migrations --follow"
      - echo $command
      - sh -c "$command"

Is there any downside to this approach that you see @iamhopaul123 considering the default using CODEBUILD_BUILD_ID as the tag?

[Lou1415926]

Hey @jetaggart! Agree that using latest to reference an image is dangerous - glad you end up using the SHA as the tag! To answer your question -

Is there any downside to this approach that you see considering the default using CODEBUILD_BUILD_ID as the tag?

As long as you don't and won't need to identify the images associated with a particular build run, I don't think there is a downside. Tagging images with CODEBUILD_BUILD_ID can be helpful when you try to debug something retrospectively from an old pipeline execution; but tagging with Git SHA is as useful I think, if not more.

Copilot doesn't try to retrieve an image using the CODEBUILD_BUILD_ID tag, so you don't need to worry about anything on that front either.

A P.S. - for the sole purpose of differentiating staging image from production image (which may or may not built on the same commit), I'd image ${CODEBUILD_BUILD_ID##*:}-$env as the tag should be good enough 💭

[aburgel]

Thanks @jetaggart for putting this together. It was super helpful! I would love if some of this was baked into copilot since there are plenty of sharp edges. I figured I'd share my learnings and tweaks in case anyone else is following this pattern.

version: 0.2
phases:
  install:
    commands:
      - echo "cd into $CODEBUILD_SRC_DIR"
      - cd $CODEBUILD_SRC_DIR
      # Download the copilot linux binary.
      - wget -q https://ecs-cli-v2-release.s3.amazonaws.com/copilot-linux-v1.33.4 -O copilot-linux
      - chmod +x ./copilot-linux
  build:
    commands:
      - export COLOR="false"
      - export CI="true"
      - job="db-migration"
      - tag=$(echo ${CODEBUILD_RESOLVED_SOURCE_VERSION##*:}-$COPILOT_ENVIRONMENT_NAME | sed 's/:/-/g' | rev | cut -c 1-128 | rev)
      - command=$(./copilot-linux task run --generate-cmd $COPILOT_APPLICATION_NAME/$COPILOT_ENVIRONMENT_NAME/$job 2>&1 >/dev/null)
      - command=$(echo "$command" | sed "s/^copilot/.\/copilot-linux/")
      - command=$(echo "$command" | sed "s/--image \([A-Za-z0-9.-]*\).amazonaws.com\/$COPILOT_APPLICATION_NAME\/$job[^ ]*/--image \1.amazonaws.com\/$COPILOT_APPLICATION_NAME\/$job:$tag/")
      - command="$command --task-group-name $job --follow --platform-arch arm64 --platform-os linux --cpu 256 --memory 512"
      - echo $command
      - sh -c "$command"

1. I also used CODEBUILD_RESOLVED_SOURCE_VERSION. You can't use CODEBUILD_BUILD_ID since the image is built in a different job than this one, so the IDs will be different.
2. I changed the line that replaces copilot with ./copilot-linux to only change the first instance, since if you use secrets, the secret names have the word copilot in them.
3. I changed the line that swaps out the image to be a little more general. It uses env vars for the app name and a fancier regex for the ECR host name. It should be easier to just drop this in as is in most projects.
4. I added --task-group-name to give the task a nice name. Otherwise it just uses the current directory name which is meaningless and a bit confusing to see when you're clicking around the AWS console. You'll see something like copilot-src.
5. I added platform arch and os. Unfortunately, generate-cmd doesn't add this, so if you use arm64 and leave this off, it'll attempt to use x86_64 and fail.

One other surprising thing, which I haven't figured out yet, is that the task will run in a public subnet, even if the job is configured to use a private one. I may be doing something wrong. Will post an update if I figure this out.

[neanias]

Hi, as a follow-up to this question, how do we actually specify the command to be run (e.g. bin/rails db:migrate)? As far as I can tell, this answer is just running a task but not what should be running within the container?

[jetaggart]

@neanias I built a job that is scheduled to zero that is just a docker container that runs the migration. Here is my job manifest:

name: db-migrations
type: Scheduled Job

on:
  schedule: "none"

image:
  build: platform/gateway/Dockerfile.migrations

command: ["/app/gateway", "migrate"]
cpu: 256
memory: 512
platform: linux/x86_64

network:
  connect: true
  vpc:
    security_groups:
      - from_cfn: ${COPILOT_APPLICATION_NAME}-${COPILOT_ENVIRONMENT_NAME}-platformdbclusterSecurityGroup

secrets:
  DB_SECRET:
    from_cfn: ${COPILOT_APPLICATION_NAME}-${COPILOT_ENVIRONMENT_NAME}-platformdbclusterAuroraSecret

[neanias]

Ah, fantastic! Thank you for your help!