Brand new user, unclear on generations and retention

[rwboyer]

I just started using Litestream and have it running on a small production server as a "test" in addition to my homegrown hacky backups. Works great!!! thanks!

I am unclear on a few configuration items and how they interact or are used.

1. Snapshots
2. Retention
3. Generations

Specifically how these relate to what happens on the target backup copy and how they interact if at all. When and why does a new generation get created as they seem to be a backup thing only. I was assuming that somehow Snapshots related to generations as seen on the backup bucket but that doesn't seem to be the case from what I am seeing.

How and why are generations useful or used? Do these get pruned on the target backup bucket? Do the snapshot and retention configuration items have any influence on what's being seen on the S3 bucket?

Any help clarifying these things would be great.

In the end I am looking for how any of this fits into the goal of keeping a rolling 30 days of backups and restoring various points in time via Litestream. Maybe it doesn't but I can't quite follow what Litestream is doing on the backup end of things or how that gets used. I am not even clear on when and why a "generation" gets created or if it ever gets removed by Litestream.

Thanks

[benbjohnson]

Hi @rwboyer, good question. There's a section on the How it works page of the web site called Snapshots & Generations that gives some more detail but I'll try to be more clear here.

Essentially, a "generation" contains at least one snapshot and then a contiguous series of WAL pages. Since Litestream can be stopped and restarted separate from your application, it could miss updates to the database while it is stopped. If Litestream detects that it can't resume from where it left off, it'll start a new generation.

Here's a visual. Generation XXXXXXXX starts with a snapshot #1 and begins replicating its WAL. Eventually it may create another snapshot #2 (depending on your snapshot frequency settings). If Litestream gets shutdown or perhaps you restore to a new server after WAL #3 then Litestream detects that it can't resume a contiguous set of WAL pages, and it will start generation YYYYYYYY.

Within a generation, you can restore from a snapshot and replay WAL files forward to get to a specific point-in-time. However, you can't use WAL files across generations as there was a break in the replication so Litestream wouldn't be able to guarantee that your restored database has all the correct pages.

 Generation XXXXXXXX ═══════════════════════════════════════════════════════════════════════════╗
║                                                                                               ║
║                                                                                               ║
║   ┌────────────┐                          ┌────────────┐                                      ║
║   │  Snapshot  │                          │  Snapshot  │                                      ║
║   │  #0000000  │                          │  #0000002  │                                      ║
║   └────────────┘                          └────────────┘                                      ║
║   ┌────────────┐      ┌────────────┐      ┌────────────┐      ┌────────────┐                  ║
║   │    WAL     │      │    WAL     │      │    WAL     │      │    WAL     │                  ║
║   │  #0000000  │      │  #0000001  │      │  #0000002  │      │  #0000003  │                  ║
║   └────────────┘      └────────────┘      └────────────┘      └────────────┘                  ║
║                                                                                               ║
║   ───────────────────────────────────────────────────────────────────────────────────────▶    ║
║                                                                                               ║
╚═══════════════════════════════════════════════════════════════════════════════════════════════╝
                                                                                                 
                                                                                                 
                                                                                                 
  Generation YYYYYYYY ═════════════════════════════════════════════╗                             
 ║                                                                 ║                             
 ║                                                                 ║                             
 ║   ┌────────────┐                                                ║                             
 ║   │  Snapshot  │                                                ║                             
 ║   │  #0000000  │                                                ║                             
 ║   └────────────┘                                                ║                             
 ║   ┌────────────┐      ┌────────────┐      ┌────────────┐        ║                             
 ║   │    WAL     │      │    WAL     │      │    WAL     │        ║                             
 ║   │  #0000000  │      │  #0000001  │      │  #0000002  │        ║                             
 ║   └────────────┘      └────────────┘      └────────────┘        ║                             
 ║                                                                 ║                             
 ║   ──────────────────────────────────────────────────────────▶   ║                             
 ║                                                                 ║                             
 ╚═════════════════════════════════════════════════════════════════╝                             

As for your other questions:

Do [generations] get pruned on the target backup bucket?

Yes. The retention enforcement removes old snapshots once the retention period elapses. If no more snapshots exist in a generation then the generation is removed as well. If no more snapshots exist in the current generation then a new snapshot will be created before removing the old one. WAL files are only retained from the index of the oldest snapshot.

Do the snapshot and retention configuration items have any influence on what's being seen on the S3 bucket?

Yes. The snapshot-interval configuration determines how often a new snapshot is created. If it isn't set then a new snapshot is created when an old snapshot is being removed because of retention. The retention configuration option specifies how old snapshots can be before they are removed.

For example, you could set retention to 720h for 30 days of retention. You can also set snapshot-interval to 24h to create a new snapshot every day. The benefit to creating more frequent snapshots is that it will take less time to restore since it only needs to replay WAL files since the most recent snapshot.

There's more details in the Replica settings section on the web site.

[rwboyer]

Thanks so much for clearing this up as it wasn’t quite explicit in connecting the how it works page to the parameters used for config. One last clarification would be great… Exactly how would you restore to a point in time given a retention of 720 and a snapshot frequency of 24? I guess that’s the last piece of the puzzle I am unclear on given the CLI documentation. Thanks so much RB

…

On Dec 24, 2021, at 6:09 PM, Ben Johnson ***@***.***> wrote: Hi @rwboyer <https://github.com/rwboyer>, good question. There's a section on the How it works page of the web site called Snapshots & Generations <https://litestream.io/how-it-works/#snapshots--generations> that gives some more detail but I'll try to be more clear here. Essentially, a "generation" contains at least one snapshot and then a contiguous series of WAL pages. Since Litestream can be stopped and restarted separate from your application, it could miss updates to the database while it is stopped. If Litestream detects that it can't resume from where it left off, it'll start a new generation. Here's a visual. Generation XXXXXXXX starts with a snapshot #1 <#1> and begins replicating its WAL. Eventually it may create another snapshot #2 <#2> (depending on your snapshot frequency settings). If Litestream gets shutdown or perhaps you restore to a new server after WAL #3 <#3> then Litestream detects that it can't resume a contiguous set of WAL pages, and it will start generation YYYYYYYY. Within a generation, you can restore from a snapshot and replay WAL files forward to get to a specific point-in-time. However, you can't use WAL files across generations as there was a break in the replication so Litestream wouldn't be able to guarantee that your restored database has all the correct pages. Generation XXXXXXXX ═══════════════════════════════════════════════════════════════════════════╗ ║ ║ ║ ║ ║ ┌────────────┐ ┌────────────┐ ║ ║ │ Snapshot │ │ Snapshot │ ║ ║ │ #0000000 │ │ #2 │ ║ ║ └────────────┘ └────────────┘ ║ ║ ┌────────────┐ ┌────────────┐ ┌────────────┐ ┌────────────┐ ║ ║ │ WAL │ │ WAL │ │ WAL │ │ WAL │ ║ ║ │ #0000000 │ │ #1 │ │ #2 │ │ #3 │ ║ ║ └────────────┘ └────────────┘ └────────────┘ └────────────┘ ║ ║ ║ ║ ───────────────────────────────────────────────────────────────────────────────────────▶ ║ ║ ║ ╚═══════════════════════════════════════════════════════════════════════════════════════════════╝ Generation YYYYYYYY ═════════════════════════════════════════════╗ ║ ║ ║ ║ ║ ┌────────────┐ ║ ║ │ Snapshot │ ║ ║ │ #0000000 │ ║ ║ └────────────┘ ║ ║ ┌────────────┐ ┌────────────┐ ┌────────────┐ ║ ║ │ WAL │ │ WAL │ │ WAL │ ║ ║ │ #0000000 │ │ #1 │ │ #2 │ ║ ║ └────────────┘ └────────────┘ └────────────┘ ║ ║ ║ ║ ──────────────────────────────────────────────────────────▶ ║ ║ ║ ╚═════════════════════════════════════════════════════════════════╝ As for your other questions: Do [generations] get pruned on the target backup bucket? Yes. The retention enforcement removes old snapshots once the retention period elapses. If no more snapshots exist in a generation then the generation is removed as well. If no more snapshots exist in the current generation then a new snapshot will be created before removing the old one. WAL files are only retained from the index of the oldest snapshot. Do the snapshot and retention configuration items have any influence on what's being seen on the S3 bucket? Yes. The snapshot-interval configuration determines how often a new snapshot is created. If it isn't set then a new snapshot is created when an old snapshot is being removed because of retention. The retention configuration option specifies how old snapshots can be before they are removed. For example, you could set retention to 720h for 30 days of retention. You can also set snapshot-interval to 24h to create a new snapshot every day. The benefit to creating more frequent snapshots is that it will take less time to restore since it only needs to replay WAL files since the most recent snapshot. There's more details in the Replica settings <https://litestream.io/reference/config/#replica-settings> section on the web site. — Reply to this email directly, view it on GitHub <#258 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAU2GS5PBIIF5N7UBWTZTDUST4UHANCNFSM5KVTL5NQ>. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.

[benbjohnson]

@rwboyer You can restore to a specific point-in-time by using the -timestamp flag:

$ litestream restore -timestamp 2020-01-01T00:00:00Z /path/to/db

https://litestream.io/reference/restore/#point-in-time-restore

[ChrisDrit]

How does one understand what timestamps are available to use with this config option?

The only way I could make it work was by listing snapshots:

litestream snapshots  "${REPLICA_URL}"

...and using the created date time stamp.

replica  generation        index  size     created
s3       15d6a2b7e3826017  125    4255500  2024-02-08T04:43:17Z
s3       15d6a2b7e3826017  176    4288053  2024-02-09T03:17:08Z

Is there another way?