rds-postgres-backup-s3-secure

This docker image is designed to be used on AWS ECS. It creates a dump of a postgres database on RDS, compresses it, encrypts it using assymetric crypto, and uploads it to s3. All permissions are managed via IAM.

Why should I use it?

This image was made to fullfil use cases where data is sensitive and security is of concern:

• No hardcoded access keys are used. Even if the container logs or the container configuration are leaked, the database can not be acessed.
• The backup is encrypted using a public key, while the private key does not need to be exposed, ever. Not even an administrative AWS user could decrypt the backup if the private key is kept secure.

This approach is a bit more complicated than just using access keys and relying on the encryption of S3. If that would be sufficient, consider using another image.

How it works in detail

This container performs the following steps when executed:

• Authenticate with a postgres database via IAM
• Create a dump of the database using pg_dump
• Compress the dump with bzip2
• Generate a secret key using openssl
• Encrypt the dump using the secret key
• Encrypt the secret key using a given public key
• Upload the encrypted key and dump to s3

Usage

You can find an ECS task definition here.

The container can be executed manually using

docker pull ejoebstl/rds-postgres-backup-s3-secure

Please mind the configuration below.

Generating a key pair

Optionally generate a new private key or use an existing one. The private key will be needed to decrypt the backups:

openssl genrsa -des3 -out private_key.pem 2048

To derive a public key from the private key:

openssl rsa -in private_key.pem -outform PEM -pubout -out public_key.pem

Execution role policy

• IAM authentication needs to be explicitely enabled on RDS
• A database user needs to be created with appropriate permissions
• The execution role needs an permission for RDS and S3

The following is a minimal example of a useable role policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "ssm:DescribeParameters"
      ],
      "Resource": "*"
    },
    {
      "Sid": "Stmt1482841904003",
      "Effect": "Allow",
      "Action": [
          "s3:Put*",
      ],
      "Resource": [
        "${s3_bucket_arn}"
      ]
    },
    {
        "Effect": "Allow",
        "Action": [
            "rds-db:connect"
        ],
        "Resource": [
            "arn:aws:rds-db:${db_region}:${aws_account}:dbuser:${db_rid}/${db_user}"
        ]
    }
  ]
}

s3_bucket_arn, the ARN of the bucket to upload the backup to
db_region, the region in which the RDS instance resides
aws_account, the AWS account identifier
db_rid, the resource identifier of the RDS instance
db_user, the database user

Environment

The container requires the following environment variables to be set:

REGION, Region of database. POSTGRES_DATABASE, name of the database
POSTGRES_HOST, the name of the RDS instance
POSTGRES_PORT, the port of the RDS instance (default: 5432)
POSTGRES_USER, the database user with permissions for pg_dump and IAM authentication enabled
S3_BUCKET, the name of the S3 bucket to upload the backup to S3_PREFIX, prefix which will be prepended to the upload path (default: backups) S3_REGION, region of S3 bucket. If not set, default to $REGION. OPENSSL_PUBLIC_KEY, the public key RATE_LMIT, rate limiting of data transfer out of pg_dump. This can be used to avoid runing out of IOPS in RDS. A t2.medium instance dumpy about 6MB/s of data at maximum speed. For details of the format, please refer the documentation of pv.

Decrypting

These commands were tested on OpenSSL 1.1.1

To decrypt a backup, decrypt the encrypted key fist, using your private key:

openssl rsautl -decrypt -inkey private_key.pem -in key.bin.enc -out key.bin.dec

Then, decrypt the backup using the encrypted key:

openssl enc -d -aes-256-cbc -salt -in backup.sql.gz.enc -out backup.sql.gz --pass file:./key.bin.dec

Finally, decompress the decrypted file:

bzip2 -d backup.sql.gz 

Two words of caution

Please test your backups regularly.
If the private key is lost, all backups encrypted with it are lost as well.

Planned todos

• A terraform module to schedule backups on an ECS cluster.
• Move to OpenSSL 1.1 with secure key derivation, as soon as alpine supports it.

Troubleshooting

If you get an error like the following during decrypting the key file, the private key is likely incorrect.

RSA operation error
139650942215488:error:0407109F:rsa routines:RSA_padding_check_PKCS1_type_2:pkcs decoding error:crypto/rsa/rsa_pk1.c:251:
139650942215488:error:04065072:rsa routines:rsa_ossl_private_decrypt:padding check failed:crypto/rsa/rsa_ossl.c:491: