[Issue]: Terraform should only build+deploy Lambdas with changes

[TylerHendrickson]

Why is this issue important?

Building, uploading, and publishing all Lambda functions on every terraform apply makes iterating on Terraform changes slower than it needs to be. As a (much) secondary concern, it also uploads/publishes more Lambda artifacts than are necessary.

Current State

When terraform apply runs, every Lambda function is re-deployed regardless of whether they have changed.

The terraform-aws-modules/lambda/aws Terraform module that we use to facilitate Lambda deployments has facilities for comparing checksums of local builds against previous local builds (cached in terraform/builds), and should also be able to compare against remote (S3-hosted) artifacts. However, that doesn't seem to be having any effect in our current setup, which for the most part is customized by the source_path input to each Lambda module and looks like this:

source_path = [{
  path = var.lambda_code_path
  commands = [
    "task build-DownloadGrantsGovDB",
    "cd bin/DownloadGrantsGovDB",
    ":zip",
  ],
}]

Additionally, our Taskfile configuration (which is used by Terraform in order to build Lambdas) also short-circuits builds when relevant source files are unmodified, which is worth being aware of when exploring solutions to this issue.

Expected State

Lambda functions only get deployed during terraform apply when the source code and/or local build artifacts are modified (at the very least in local (localstack) environments).

Implementation Plan

Explore the documentation for the terraform-aws-modules/lambda/aws module and/or explore other alternatives to optimize this aspect of the build process.

Relevant Code Snippets

No response

[jakekreider]

A related issue discussed in Slack:
If the lambdas fail to build, at least in Localstack, it will still try to deploy them and can get stuck during the zip phase trying to package the parent directory. This probably wouldn't affect the AWS environments since the failed build would fail the CI check, but it can snarl local environments in the right circumstances. Ideally, if any steps in the commands fail, it would stop there & fail the tf apply.

[TylerHendrickson]

Builds currently generate inconsistent checksums in different VCS scenarios (git state is clean/dirty, different checked out commits, etc.). Reopening to address.