Create and manage Tigris bucket for PG/Barman backups #3693
Conversation
|
If the bucket name is already taken, we should generate a unique name instead of erroring out |
Looking at this naming logic, if a name isn't explicitly specified via OverrideName, it's automatically set to the target app name. I think the extension creation process in web will autogenerate random names but I don't see this easily exposed by flyctl's extension code. So in practice this means that PGs created with my-name will have buckets also named my-name.
I agree that the ideal case is not to bomb out on unique bucket names, but how likely is this in practice? I imagine:
The crash isn't ideal but it isn't entirely unexpected and easy enough to work around by removing the PG storage bucket. Either way, I don't want to spend much time overthinking this, so possible approaches if this needs to be addressed:
If this definitely needs addressing then I think I'd prefer 2, or 1 as long as we aren't pulling in another dependency. 2 is simple enough to just do and is probably a good idea anyway, so I'll just do that and we can decide what additional needs doing. |
Apparently somewhat, we hold onto recently-deleted bucket names so create/destroy/create cycles will fail even if the bucket itself is removed. Going to try scraping the error text for naming conflicts and appending a suffix, unless there's a better approach. |
|
Awesome start on this! Couple notes:
I think we should actually make this opt-in to start. There's a cost implication that some users may not be cool with.
It looks like this currently breaks if the bucket name doesn't match the App name.
|
…pped to arbitrary buckets. Try to fix, but currently non-functional.
Sorry, which log output do you mean specifically, and which additional secret am I injecting? Looks like I'm only injecting the one as per our docs. Thanks for the feedback. |
Agreed. Any thoughts about the easiest way to implement this? Version checks seem like the most immediately obvious, but that feels wrong and brittle. Seems like the better way is checking for the existence of this specific feature in the image, either with a command we can exec in and run, or something in the machine metadata. Thoughts? Unless of course there's something I'm missing with the version check, but semver version comparison feels like something we'd either need another dependency for or would have to be very careful about. |
There's already logic in place for this: |
…lain how they are enabled.
…ig to restored instance.
… Postgres clusters.
Usage:
fly postgres [command]
Aliases:
postgres, pg
Available Commands:
attach Attach a postgres cluster to an app
barman Manage databases in a cluster
config Show and manage Postgres configuration.
connect Connect to the Postgres console
create Create a new Postgres cluster
db Manage databases in a cluster
detach Detach a postgres cluster from an app
events Track major cluster events
failover Failover to a new primary
import Imports database from a specified Postgres URI
list List postgres clusters
renew-certs Renews the SSH certificates for the Postgres cluster.
restart Restarts each member of the Postgres cluster one by one.
users Manage users in a postgres cluster
Flags:
-h, --help help for postgres
Global Flags:
-t, --access-token string Fly API Access Token
--debug Print additional logs and traces
--verbose Verbose output
Use "fly postgres [command] --help" for more information about a command.
…'s using a custom imageRef.
| } | ||
|
|
||
| // Resolve the leader | ||
| leader, err := pickLeader(ctx, machines) |
There was a problem hiding this comment.
This will fail if the DB is having problems, or a host is down. What I did in ui-ex is spin up a new temporary machine in the same app - https://github.com/superfly/ui-ex/blob/add83100682486e9d9e8df2d9faf16a5e4a2342a/lib/fly/postgres.ex#L241
Change Summary
Integrates the new Tigris WAL storage into flyctl.
--enable-backupflag tofly pg createcommand to enable bucket/secret provisioning on cluster creation.--restore-target-app,--restore-target-name,--restore-target-timeand--restore-target-inclusiveflags tofly pg createwhich behave as documented here. When a new instance with a cluster size of 1 is created with an app and optional target set, the specified backup is restored into the newly-created instance.fly pg backup enablecommand for provisioning buckets and secrets into clusters where WAL backup is disabled.fly pg backup createfor creating backups, optionally named.fly pg backup listto list existing backups.Related to:
Documentation