Upgrade aws-rds broker DB versions

[siennathesane]

In order to deploy supported PostgreSQL versions, the aws-rds broker needs to deploy supported versions.

This is the notification we received from a customer

The shared PostgreSQL version used is 9.4.7 (as per document) whose EOL is Feb 13, 2020. Anyone who will create a service with shared-psql will create an PSQL instance with version 9.4.

It aligns with the PostgreSQL version support: https://www.postgresql.org/support/versioning/

Prep:

• migrate to recreatable prarameter group in dev
• migrate to recreatable prarameter group in staging
• migrate to recreatable prarameter group in prod

Shared DBs:

• upgrade shared RDS in dev
• upgrade shared RDS in staging

Internal DBs:

• upgrade internal RDS in dev
• upgrade internal RDS in staging

During the maintenance window at 10:30 am PT on March 10th:

• plan upgrades for shared and internal DBs in prod
• snapshot shared and internal RDS in prod
• upgrade shared and internal RDS in prod
• update cg-site documentation on versions. PR in Update documentation on current PostgreSQL DB versions cg-site#1559

Security considerations

Will eventually pull in all updates and CVEs from major upgrades.

[karareinsel]

Have until 2/14/2020 until this increases in urgency. Address in this sprint or next sprint (sprint 20.2.7).

[tammersaleh]

OK, so I didn't read the original message closely enough, and assumed that this applied to all postgres RDS instances created by the aws-rds broker. Instead, this is only for the shared-psql plan, which provisions databases inside the single shared RDS instance.

So the good news is that new dedicated psql instances are created with whatever the latest AWS version is.

The bad news is that is looks like our shared RDS instance is likely going to be automatically upgraded out from under us at any moment:

As part of the retirement plan for Amazon RDS PostgreSQL 9.4, starting September 15, 2019, we will be disabling creation of any new PostgreSQL 9.4 instances using the AWS console. Starting November 15, 2019, we will be performing forced auto minor version upgrades on your remaining 9.4 instances to the recent minor version of 9.4. You can continue to do PostgreSQL 9.4 restores and read replica creations until January 15, 2020. Starting January 15, 2020, your remaining 9.4 instances will be force upgraded to the most recent minor version of PostgreSQL 11. If we are not able to upgrade to PostgreSQL 11 for any reason, we will upgrade to the 9.5 version. After that upgrade process is complete, any instance restored from a snapshot of PostgreSQL 9.4 version will automatically and immediately be upgraded to a currently supported version.

[eddietejeda]

Why would someone choose shared-psql, especially since it does not impact their cost?

How many folks do we have there? Would it be easier to get people to move away from this?

[bengerman13]

are the non-shared instances available in sandboxes?

[tammersaleh]

I can confirm that we are allowing the creation of non-shared databases in the sandbox spaces. If you think it's fine to tell users to move to dedicated instances (and shut down the creation of new shared ones), then that's probably easier.

[bengerman13]

@eddietejeda it might make sense to move folks off this, but it wouldn't get us out of this upgrade.
If we decide to drop this, we'd need to follow our deprecation procedure, which takes 180 days (at a minimum) so we'd need to do this upgrade anyway.

[eddietejeda]

If not needed, let's get deprecate.

[bengerman13]

I think that's something special with gsa sandboxes after more looking - the sandbox-gsa org has Provision Paid Services showing as yes in stratos, whereas other sandbox orgs say no, and only the -shared plans show as free in the marketplace - the rest show as paid.

[eddietejeda]

Consideration: If sandboxes are using their own DB, would the DB be deleted once sandbox trial period is over?

[tammersaleh]

@bengerman13 is mostly correct: we definitely need to put this fire out before we can do anything else, and removing the shared plans will take 180 days.

However, the plan of action is to upgrade everything to Postgres 9.5, which doesn't EOL for another year. That's the smallest change we're forced to make, and should disrupt our current users the least.

But we still need to get to an actually current version of PgSQL (11), which means we'll have to run two shared RDS instances (one at 11, and one at 9.5). We'll need to modify the plans and possibly broker code to work between them. At that point, I'd rather just kill the shared plan altogether.

So I suggest we continue upgrading the shared plans to 9.5, and start the shared-psql (and shared-mysql?) deprecation process at the same time.

[siennathesane]

if sandboxes are using their own DB, would the DB be deleted once sandbox trial period is over?

@eddietejeda yes they will, when you delete a space in CF, you can opt into force deleting their child resources (aka databases).

[tammersaleh]

This is merged as part of cloud-gov/aws-broker#55, and has gone through the pipeline to prod.

[soutenniza]

Customer during office hours pointed out Postgres was showing version 9.4.20. Confirmed with https://github.com/18F/aws-broker/blob/fb644c38f51104fe0b1b72d7e3e0c896fec39922/ci/pipeline.yml#L442-L491

[tammersaleh]

Confirmed that shared in dev is at 9.5.15.

[tammersaleh]

I've hit a known issue in upgrading RDS via terraform going back to 2017, and continuing to as recently as Jan 8, 2020:

• Modify aws_db_instance and delete aws_db_parameter_group breaks hashicorp/terraform-provider-aws#6448
• Report: aws_rds_cluster diffs didn't match during apply hashicorp/terraform-provider-aws#6049
• update aws_db_subnet_group/aws_db_parameter_group fail (with new resource required)  hashicorp/terraform-provider-aws#2402
• Destroying aws_db_instance doesn't wait until it has completely finished hashicorp/terraform-provider-aws#1571
• rds upgrade suggestion via terraform hashicorp/terraform-provider-aws#526

The upshot is that some manual work is required alongside the terraform run. I'm going to want to ensure we understand exactly what the flow is before proceeding. One comment implied we could reboot the RDS instance and re-apply.

[tammersaleh]

The maintenance window has been moved to Tuesday, March 10th, due to an incident that was in play on Tuesday, March 3rd.