Roadmap for fixing cover archival

[mekarpeles]

Part of #6822

Background

Open Library runs a coverserver with ~1TB of space. Re: #3386, at least monthly, the coverstore is backed up at the disk level.

Originally posted by @mekarpeles in #3386 (comment)

This doesn't prevent it from filling up. Typically, when new covers are added to Open Library, they end up (a) on the coverstore mounted storage drive and (b) tracked in the coverstore database.

Since landing #7230, we now have documentation and a tested/understood process for moving covers off disk into archives that can be uploaded to archive.org -- See README.

Problem(s)

There are 4 problems with the current state of manual cover archival:

1. When we move covers off disk into .tar archives and upload them to archive.org, we don't update the cover's location in the db to point to resulting archive.org path. This makes it so we have to guess if the file is in a zip or tar.
   • Presently, all covers with IDs between 8M and 8.81M are in .tar format on archive.org items. These db rows can easily be updated so their location is correct in the db
   • Moving forward, we should update the archival script so each row of the db is updated with the right archive.org path
2. Investigate migrating cover tars -> zips #7478 The current archival process results in .tars which are much slower to access from archive.org than .zips
   • We want to update our existing archive process to make .zips instead of .tars
   • A cleanup job is desired to go back and convert our hundreds of existing .tar dumps (e.g. every tar and index partial in https://archive.org/download/covers_0008) to zip. And then to update the database references, as is specified in [1].
3. Monthly Cron to archive book cover files: Creating a daily cron job that...
   • Moves covers off of disk into .zip files
   • Uploads these zips into archive.org items
   • Confirms (using the audit mechanism) that the zip partials have been uploaded
   • Updates the cover db paths to the corresponding archived .zip url
4. Internal basic cover archival documentations & troubleshooting
5. Cover archival tests

[anandology]

The challenge with time based backups was that cover archive expects fixed
number of images per archive.

It may be good idea to seperate the backup and the archive.

A nightly job backups all the covers uploaded in the previous day as a tar
ball with date in the filename to ol-covers-backup item.

A separate monthly job would archive the covers on the machine, only when
enough covers are avilable for an item.

What do you think @mekarpeles ?