Add scripts to save and restore baseline data

[mkllnk]

What? Why?

• Closes Staging breaks when conflicting database migrations are deployed #12936

Sometimes a staging database gets changed by pull requests which are not finished yet. Those changes should be reverted and some of these changes may actually break the application. So here are a couple of scripts for testers to execute on the server.

• Save baseline data when you have a good pull request that gets merged and you want to keep your test data.
• Restore baseline data after you tested a pull request with migrations and that pull request is not merged as is. You will lose test data.
• Restore baseline data after you've done lots of destructive testing, deleting records that you would like back.

What should we test?

# Log into the staging server with ssh and run these:
~/apps/openfoodnetwork/current/script/ci/save-staging-baseline.sh
~/apps/openfoodnetwork/current/script/ci/restore-staging-baseline.sh

Release notes

Changelog Category (reviewers may add a label for the release notes):

• User facing changes
• API changes (V0, V1, DFC or Webhook)
• Technical changes only
• Feature toggled

The title of the pull request will be included in the release notes.

Dependencies

Documentation updates

[rioug]

Looks good 👍

[dacook]

Good one 👍
I like that it only ever stores one copy, and that it's compressed so we don't need to worry about disk usage ballooning out.

I have a couple of concerns though, can you please take a look?

[dacook]

I think postgres blocks you from dropping a db if there's still active connections, do we need to stop puma and sidekiq?

[dacook]

Note, I had an issue before when Metabase was also connected, remotely. We have a solution to force close the connection here:
https://github.com/openfoodfoundation/ofn-install/blob/master/roles/db_restore/tasks/restore_database.yml#L15-L20

[dacook]

👍

[filipefurtad0]

Hey @mkllnk ,

Thanks for this. Maybe a good way to verify all works as expected, could be to:

• Stage this PR, then Log into the staging server with ssh and save the baseline, by running:
  ~/apps/openfoodnetwork/current/script/ci/save-staging-baseline.sh

Then, I think we could test for two distinct scenarios:
i) create some test data - like a new order or user
ii) run a migration which could break things somehow, or just change the DB structure somehow

Then finally, run
~/apps/openfoodnetwork/current/script/ci/restore-staging-baseline.sh

And verify changes i) and ii) are correctly reverted.

I'll give this a try.

[filipefurtad0]

Staging failed in staging-UK:

https://github.com/openfoodfoundation/openfoodnetwork/actions/runs/11620487686

But worked for staging-FR:

https://github.com/openfoodfoundation/openfoodnetwork/actions/runs/11620645440

Running the command for saving the baseline, took a couple of minutes to complete, and yielded:

Last login: Wed Oct 30 21:50:00 2024 from 220.253.199.227
openfoodnetwork@vps-2a25f087:~$ ~/apps/openfoodnetwork/current/script/ci/save-staging-baseline.sh
Staging baseline data saved.

Making changes:

• created an order -> https://staging.coopcircuits.fr/orders/R555075285?order_token=22284cd168ce8d3a
• removed a migration -> down 20240828203544 Fully enable admin style v3 flag

After running the restore command:

Yielded quite a lot of output, among which these lines:

ERROR:  permission denied to create extension "pg_stat_statements"
HINT:  Must be superuser to create this extension.
ERROR:  extension "pg_stat_statements" does not exist

And ended with:

ERROR:  must be member of role "postgres"
ERROR:  must be member of role "postgres"
ERROR:  must be member of role "postgres"
ERROR:  must be member of role "postgres"
ERROR:  must be member of role "postgres"
ERROR:  must be member of role "postgres"
Staging baseline data restored.

Checked that:

• I should not be able to access the order confirmation page -> https://staging.coopcircuits.fr/orders/R555075285?order_token=22284cd168ce8d3a -> this now throws an error 500, I'd say this is expected
• the changed migration should be up again -> up 20240828203544 Fully enable admin style v3 flag

Looking great!! 🎉

Merging.