feat!: Terraform MSV increased to 1.0, consolidate security group rules under one generic resource, add support for managed master password

[bryantbiggs]

Description

• With RDS now supporting the integration with SecretsManager to manage the master user password, the ability to generate a random password has been removed from this module. This is the preferred and strongly recommended route to mange the password since it keeps the password out of the Terraform state and allows for the password to be rotated automatically.
• Support for generating a random snapshot identifier has been removed. The AWS provider has been updated to enforce a conflict between snapshot_identifier and global_cluster_identifier which makes this feature challenging to support; which it has already been challenging to support in the past and often catching users off guard. Instead, the module now defaults to null for both of these values and puts the control back in user's hands if they wish to set a value for one of these arguments.
• The default value for create_db_subnet_group has changed from true to false - typically, a common/shared DB subnet group is utilized since there are no real tangible benefits to creating a new one for each DB cluster
• allowed_security_groups, allowed_cidr_blocks, and security_group_egress_rules have been removed and replaced with a more generic security_group_rules variable which supports both ingress and egress rules to/from all supported resources/destinations (e.g. security groups, CIDR blocks, prefix lists, etc.)
• Minimum supported Terraform version is now 1.0
• Update GitHub action versions to use latest. This remove warnings related to https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
• Ensure pre-commit config is aligned with latest
• Add lock.yml workflow to automatically lock issues and PRs after 30 days. Theres a lot "Me too" or "I have this issue, when will this get fixed" on really old/stale issues and in order to properly triage, users need to supply their configurations. This workflow is pulled from the AWS provider's repo to force users to fill out a proper issue ticket and keep chatter out of merged PRs or old issues
• Update examples to:
  • Use data source for availability zones and compute subnet CIDR blocks using the cidrsubnet() function
  • Use latest supported engine versions
  • Remove standalone resources for parameter groups since module now supports this resource management

See UPGRADE-8.0.md for full details

Motivation and Context

• Patch warnings on CI checks to keep output clean
• Resolves Security Group Updates are Broken. #338
• Resolves "snapshot_identifier": conflicts with global_cluster_identifier #358
• Resolves feat: Allow override of protocol in egress rule #359
• Resolves Add newly released manage_master_user_password feature in Terraform (v4.61.0)  #364

Breaking Changes

• Yes; see UPGRADE-8.0.md

How Has This Been Tested?

• I have updated at least one of the examples/* to demonstrate and validate my change(s)
• I have tested and validated these changes using one or more of the provided examples/* projects
• I have executed pre-commit run -a on my pull request

[bryantbiggs]

The fix here is a version bump so I will catch that up with other TODOs slated for a breaking change to roll it up into one

[bryantbiggs]

we should wait for hashicorp/terraform-provider-aws#28538 to see what that implementation is before merging this, but in the meantime I'll prep the other changes

[mad-it]

Since this is a breaking change anyways, maybe also good to set the default password length to something like 20 characters. 10 characters is not that long and since not using special characters it makes it fairly weak.

[bryantbiggs]

Since this is a breaking change anyways, maybe also good to set the default password length to something like 20 characters. 10 characters is not that long and since not using special characters it makes it fairly weak.

I am waiting for hashicorp/terraform-provider-aws#28538 which deals with the password management so we'll see what that API looks like and make changes as necessary here. Ideally, there are no more passwords generated/managed by this module and therefore outside of the Terraform state

[antonbabenko]

Zero comments! Looks perfect right away :)

[antonbabenko]

Not related to the PR, but I would like to know how to avoid GitHub token rate limit throttling since we have the same cron expression in all repositories, and every night I receive a lot of emails when this workflow fails.

There is an open issue already, but it doesn't seem to be moving anywhere - dessant/lock-threads#35

[antonbabenko]

This PR is included in version 8.0.0 🎉

[nikolay]

@antonbabenko What is the proper way to migrate to v8 if you had create_random_password?

[antonbabenko]

@nikolay Probably remove that random password resource from Terraform state (terraform state rm ...) and then set it as master_password value, and later migrate to use manage_master_user_password = true as the recommended approach for handling passwords.

@bryantbiggs Should we upgrade this process in the Upgrade Guide for version 8.0? I suspect this will be a frequent question.

[bryantbiggs]

Just simply changing to use manage_master_user_password = true worked as expected so I didn't see any need to document anything special

[nwalters512]

Will manage_master_user_password = true opt us in to automatic password rotation? I can't tell from the docs, and I'm pretty scared to test that out. If the database password changes on a regular basis, that's going to cause lots of problems for applications that aren't pulling credentials from Secrets Manager. Even for apps that do pull from Secrets Manager, I can't see how to avoid downtime when the password is rotated. I do think this needs to be documented.

[bryantbiggs]

It is documented in the AWS documentation - Aurora does automatically rotate the password when it manages the master password

The master password is not intended for applications - the master password should be treated like the root user of an AWS account. You use it for only certain scenarios, but you should create custom users and groups with the appropriate access permissions which are used by users and/or applications. You can also look at providing access using IAM auth - either directly through Aurora or through something like RDS Proxy

[nwalters512]

That's very helpful advice - thanks @bryantbiggs!

[nikolay]

@antonbabenko and/or @bryantbiggs When I removed create_random_password and added manage_master_user_password = true on a live cluster, the output with the secret is now an empty array ([]). Any idea why?

[bryantbiggs]

thats the point of the feature and why it was designed - when using manage_master_user_password the password is stored in a SecretsManager secret in your account; no secret is leaked into your Terraform state and no conflicts with Terraform if you rotate that secret

[nikolay]

@bryantbiggs I was expecting the ARN to the secret. I use the PostgreSQL provider to create app-specific users, i.e. we still need a way to authenticate as root in order to create the limited users.

[bryantbiggs]

you mean this https://github.com/terraform-aws-modules/terraform-aws-rds-aurora/pull/335/files#diff-de6c47c2496bd028a84d55ab12d8a4f90174ebfb6544b8b5c7b07a7ee4f27ec7R72

[bryantbiggs]

that link didn't seem to work as expected, but its on main

terraform-aws-rds-aurora/outputs.tf

Lines 72 to 75 in 4380f14

	output "cluster_master_user_secret" {
	description = "The generated database master user secret when `manage_master_user_password` is set to `true`"
	value = try(aws_rds_cluster.this[0].master_user_secret, null)
	}

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.