Update postgresql backup chart (#7)

* Update postgresql in backup image * Add "latest" backup mirror, update documentation Also fixes postgresql backup * Bump chart versions * Bump cozy chart version
QuiltMC · Jun 25, 2024 · b837205 · b837205
1 parent 86e7222
commit b837205
Show file tree

Hide file tree

Showing 15 changed files with 85 additions and 32 deletions.
diff --git a/README.md b/README.md
@@ -28,9 +28,30 @@ helm delete my-<chart-name>
 
 ## Backups
 
-All the charts include automated daily backups of the databases to an S3.
-To restore those backups, you currently have to download the files manually,
-then copy them to a container and run either `mongorestore` or `pgrestore`.
+All the charts include automated daily backups of the databases to an S3-compatible bucket.
+The path for each backup is `/<helm-project-name>/yyyy-mm-dd` (the extension being `.gz` for mongodb backups, and `.pgdump` for postgresql backups).
+An additional copy of the latest backup is made to `latest/<helm-project-name>`.
+
+### Expiration
+
+Expiration is not built into the backup charts. Instead, you should configure
+your storage bucket with appropriate lifecycle rules.
+In Quilt's case, Backblaze is configured such that old backups are hidden after 29 days,
+then deleted the next day.
+
+### Restoration
+
+To restore those backups, you can create a temporary container, download a backup, and run either `mongorestore`
+or `pgrestore`. For example:
+
+```bash
+# Assuming you are connected to the Quilt cluster
+kubectl run backup-restore --image=ghcr.io/quiltmc/mongodb-s3-backup:4b26b78 -it --rm -n quilt -- sh
+# Then inside the temp shell session:
+aws configure # Enter your credentials
+aws s3 cp s3://quilt-backups/xxx/1970-01-01.gz .
+mongorestore -h "hostname" -u "user" -p "password"  --gzip --archive="1970-01-01.gz"
+```
 
 ## Required secrets
 
@@ -45,6 +66,11 @@ kubectl create secret docker-registry ghcr-token --docker-server=https://ghcr.io
 
 where `$GITHUB_PAT` is a simple [access token](https://github.com/settings/tokens) with no specific permission.
 
+### Modmail
+
+With default values, Modmail requires a generic secret named `modmail-quilt-discord-token`, with the variable `TOKEN` containing the discord token.
+It also requires another generic secret named `modmail-viewer-quilt`, with the variables `MODMAIL_VIEWER_DISCORD_OAUTH_CLIENT_SECRET` and `MODMAIL_VIEWER_SECRETKEY`.
+
 ### Forum
 
 The forum requires credentials for both SMTP (email, we use AWS SES) and S3 (storage, we use Backblaze) to be set from a secret:

diff --git a/charts/cozy/Chart.lock b/charts/cozy/Chart.lock
@@ -4,6 +4,6 @@ dependencies:
   version: 13.16.2
 - name: mongodb-s3-backup
   repository: file://../mongodb-s3-backup
-  version: 1.0.0
-digest: sha256:807610a2e401438e1bf29efda6c39dbced7dd9cc557681e2f23eb773cf091bd6
-generated: "2024-05-12T19:44:03.053907+02:00"
+  version: 1.1.0
+digest: sha256:536a9fcebdaf9a4fdb28df8a89661e588ba135d99f2beefd44d61818eb12bcc4
+generated: "2024-06-04T01:57:58.435558+02:00"
diff --git a/charts/cozy/Chart.yaml b/charts/cozy/Chart.yaml
@@ -4,7 +4,7 @@ name: quilt-cozy
 # This is the chart version. This version number should be incremented each time you make changes
 # to the chart and its templates, including the app version.
 # Versions are expected to follow Semantic Versioning (https://semver.org/)
-version: 1.1.6
+version: 1.1.7
 # This is the version number of the application being deployed (purely informational).
 appVersion: '1'
 description: helm chart to deploy the Quilt Cozy Discord bot
@@ -21,5 +21,5 @@ dependencies:
     version: 13.16.2
     repository: https://charts.bitnami.com/bitnami
   - name: mongodb-s3-backup
-    version: 1.0.0
+    version: 1.1.0
     repository: 'file://../mongodb-s3-backup'
diff --git a/charts/forum/Chart.lock b/charts/forum/Chart.lock
@@ -10,6 +10,6 @@ dependencies:
   version: 17.17.1
 - name: postgresql-s3-backup
   repository: file://../postgresql-s3-backup
-  version: 1.0.0
-digest: sha256:f4f3b3f6fb9ba88f2ee54fda6f7da94a2667bd88ec37210cda738a2a06c38b41
-generated: "2024-05-12T19:44:14.415514+02:00"
+  version: 1.1.0
+digest: sha256:7f5d43b804bb907131faba017d1455d5a0346de02450d8c604c2e14c44c16b11
+generated: "2024-06-04T01:58:08.795081+02:00"
diff --git a/charts/forum/Chart.yaml b/charts/forum/Chart.yaml
@@ -4,7 +4,7 @@ name: quilt-forum
 # This is the chart version. This version number should be incremented each time you make changes
 # to the chart and its templates, including the app version.
 # Versions are expected to follow Semantic Versioning (https://semver.org/)
-version: 0.2.2
+version: 0.2.3
 # This is the version number of the Discourse image being deployed (purely informational).
 appVersion: '3.2.1'
 description: helm chart to deploy the Quilt Discourse Forum
@@ -27,5 +27,5 @@ dependencies:
     repository: https://charts.bitnami.com/bitnami
     version: 17.X.X
   - name: postgresql-s3-backup
-    version: 1.0.0
+    version: 1.1.0
     repository: 'file://../postgresql-s3-backup'
diff --git a/charts/modmail/Chart.lock b/charts/modmail/Chart.lock
@@ -4,6 +4,6 @@ dependencies:
   version: 13.16.2
 - name: mongodb-s3-backup
   repository: file://../mongodb-s3-backup
-  version: 1.0.0
-digest: sha256:807610a2e401438e1bf29efda6c39dbced7dd9cc557681e2f23eb773cf091bd6
-generated: "2024-05-12T19:44:24.507366+02:00"
+  version: 1.1.0
+digest: sha256:536a9fcebdaf9a4fdb28df8a89661e588ba135d99f2beefd44d61818eb12bcc4
+generated: "2024-06-04T01:58:20.617544+02:00"
diff --git a/charts/modmail/Chart.yaml b/charts/modmail/Chart.yaml
@@ -4,7 +4,7 @@ name: quilt-modmail
 # This is the chart version. This version number should be incremented each time you make changes
 # to the chart and its templates, including the app version.
 # Versions are expected to follow Semantic Versioning (https://semver.org/)
-version: 1.1.5
+version: 1.1.6
 # This is the version number of the application being deployed (purely informational).
 appVersion: '1'
 description: helm chart to deploy Quilt's Modmail Discord bot
@@ -21,5 +21,5 @@ dependencies:
     version: 13.16.2
     repository: https://charts.bitnami.com/bitnami
   - name: mongodb-s3-backup
-    version: 1.0.0
+    version: 1.1.0
     repository: 'file://../mongodb-s3-backup'
diff --git a/charts/modmail/gdpr_erasure.sh b/charts/modmail/gdpr_erasure.sh
@@ -0,0 +1,24 @@
+# Anonymizes the ModMail data linked to a discord account identifier
+# Assumes an existing connection to a kubernetes cluster hosting the ModMail chart
+# Variables:
+#  - MODMAIL_MONGO_POD: the name of the pod hosting the ModMail database (e.g. "quilt-modmail-mongodb-1234-abcd")
+#  - DB_USERNAME: username for connecting to the database
+#  - DB_PASSWORD: password for connecting to the database
+#  - DISCORD_ID: identifier of the discord account of which to retrieve associated data (e.g. "123456789012345678")
+kubectl exec -n quilt $MODMAIL_MONGO_POD -- mongosh -u $DB_USERNAME -p $DB_PASSWORD \
+  --eval "use modmail_bot" \
+  --eval 'db.logs.updateMany('\
+    '{ "recipient.id": "'"$DISCORD_ID"'" }, '\
+    '{ $set: {'\
+      '"messages.$[elem].author.name": "deleted_user",'\
+      '"messages.$[elem].author.avatar_url": "",'\
+      '"messages.$[elem].content": "",'\
+      '"messages.$[elem].author.id": "0",'\
+      '"recipient.name": "deleted_user",'\
+      '"creator.name": "deleted_user",'\
+      '"recipient.avatar_url": "",'\
+      '"creator.avatar_url": "",'\
+      '"recipient.id": "0",'\
+      '"creator.id": "0"'\
+    '} },'\
+    '{ arrayFilters: [ { "elem.author.id": "'"$DISCORD_ID"'" } ] })'
diff --git a/charts/mongodb-s3-backup/Chart.yaml b/charts/mongodb-s3-backup/Chart.yaml
@@ -4,7 +4,7 @@ name: mongodb-s3-backup
 # This is the chart version. This version number should be incremented each time you make changes
 # to the chart and its templates, including the app version.
 # Versions are expected to follow Semantic Versioning (https://semver.org/)
-version: 1.0.0
+version: 1.1.0
 # This is the version number of the application being deployed (purely informational).
 appVersion: '1'
 description: helm chart handling Quilt's MongoDB backups to an S3

diff --git a/charts/mongodb-s3-backup/templates/cronjob.yaml b/charts/mongodb-s3-backup/templates/cronjob.yaml
@@ -17,9 +17,9 @@ spec:
               command: ["/bin/sh", "-c"]
               args:
                 - >
-                  EXT=gz &&
-                  mongodump -h="{{ tpl .Values.database.host . }}" -u="{{ .Values.database.username | default "root" }}" -p="{{ .Values.database.password }}" --gzip --archive="backup.$EXT" &&
-                  aws --endpoint {{ .Values.endpoint }} s3 cp backup.gz s3://quilt-backups/{{ .Release.Name }}/`date -I`.$EXT
+                  mongodump -h="{{ tpl .Values.database.host . }}" -u="{{ .Values.database.username | default "root" }}" -p="{{ .Values.database.password }}" --gzip --archive="backup.gz" &&
+                  aws --endpoint {{ .Values.endpoint }} s3 cp backup.gz s3://quilt-backups/{{ .Release.Name }}/`date -I`.gz
+                  aws --endpoint {{ .Values.endpoint }} s3 cp backup.gz s3://quilt-backups/latest/{{ .Release.Name }}.gz
               envFrom:
                 - secretRef:
                     # should contain AWS credentials environment variables

diff --git a/charts/postgresql-s3-backup/Chart.yaml b/charts/postgresql-s3-backup/Chart.yaml
@@ -4,7 +4,7 @@ name: postgresql-s3-backup
 # This is the chart version. This version number should be incremented each time you make changes
 # to the chart and its templates, including the app version.
 # Versions are expected to follow Semantic Versioning (https://semver.org/)
-version: 1.0.0
+version: 1.1.0
 # This is the version number of the application being deployed (purely informational).
 appVersion: '1'
 description: Helm chart handling Quilt's PostgreSQL backups to an S3

diff --git a/charts/postgresql-s3-backup/templates/cronjob.yaml b/charts/postgresql-s3-backup/templates/cronjob.yaml
@@ -17,9 +17,9 @@ spec:
               command: ["/bin/sh", "-c"]
               args:
                 - >
-                  EXT=pgdump &&
-                  pg_dump {{ .Values.database.dbName }} --clean --if-exists --quote-all-identifiers --no-password --file=backup.$EXT &&
-                  aws --endpoint {{ .Values.endpoint }} s3 cp backup.gz s3://quilt-backups/{{ .Release.Name }}/`date -I`.$EXT
+                  pg_dump {{ .Values.database.dbName }} --clean --if-exists --quote-all-identifiers --no-password --file=backup.pgdump &&
+                  aws --endpoint {{ .Values.endpoint }} s3 cp backup.pgdump s3://quilt-backups/{{ .Release.Name }}/`date -I`.pgdump
+                  aws --endpoint {{ .Values.endpoint }} s3 cp backup.pgdump s3://quilt-backups/latest/{{ .Release.Name }}.pgdump
               env:
                 - name: PGUSER
                   value: {{ .Values.database.username }}

diff --git a/charts/stats/Chart.lock b/charts/stats/Chart.lock
@@ -4,6 +4,6 @@ dependencies:
   version: 12.8.0
 - name: postgresql-s3-backup
   repository: file://../postgresql-s3-backup
-  version: 1.0.0
-digest: sha256:cb94b35f08891be94dabd90d1d4a35f2c741375db721642037334067b1cf74ba
-generated: "2024-05-12T19:44:32.821466+02:00"
+  version: 1.1.0
+digest: sha256:4c7d7df84f62a0fdde7ab205b8b51186f620bc11d170c904c7726ebd6f68e83f
+generated: "2024-06-04T01:58:31.566591+02:00"
diff --git a/charts/stats/Chart.yaml b/charts/stats/Chart.yaml
@@ -4,7 +4,7 @@ name: quilt-stats
 # This is the chart version. This version number should be incremented each time you make changes
 # to the chart and its templates, including the app version.
 # Versions are expected to follow Semantic Versioning (https://semver.org/)
-version: 1.1.5
+version: 1.1.6
 # This is the version number of the application being deployed (purely informational).
 appVersion: '1'
 description: helm chart to deploy Quilt's Stats Discord bot
@@ -21,5 +21,5 @@ dependencies:
     version: 12.8.0
     repository: https://charts.bitnami.com/bitnami
   - name: postgresql-s3-backup
-    version: 1.0.0
+    version: 1.1.0
     repository: 'file://../postgresql-s3-backup'
diff --git a/images/postgresql-s3-backup/Dockerfile b/images/postgresql-s3-backup/Dockerfile
@@ -4,8 +4,11 @@ WORKDIR /dump
 
 RUN apt update \
     && ln -fs /usr/share/zoneinfo/UTC /etc/localtime \
-    && apt-get install -y tzdata postgresql-client curl unzip \
+    && apt-get install -y tzdata gnupg wget curl unzip ca-certificates postgresql-common \
     && dpkg-reconfigure --frontend noninteractive tzdata \
+    && /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh -y \
+    && apt update \
+    && apt install -y postgresql-client-15 \
     && curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" \
     && unzip awscliv2.zip \
     && ./aws/install \