Store blobs outside database

[matthew-white]

User stories

As a Central systems administrator with CLI access to the server
I want all blobs on my server to be stored in S3-compatible storage
So that I can back it up separately from my database

As a Central systems administrator with CLI access to the server
I want all existing blobs in my server database to be migrated to S3
So that I can reduce the size of my existing database

As a Central systems administrator with CLI access to the server
I want blobs to be purged from S3-compatible storage when the last resource that references them is purged
So that I can keep my storage usage down

As a Central systems administrator with CLI access to the server
I want to a way to see counts of blobs uploaded and blobs pending
So that I can follow up on any failures that happened

As a Central systems administrator with CLI access to the server
I want a way to force uploading db blobs to S3
So that I can control when that migration happens (e.g. to match a scheduled maintenance window)

As an ODK data collector or data analyst
I don’t want to know anything about where blobs are stored
So that I can focus on the tasks I’m an expert on

As an ODK developer
I want to get analytics reports on how many blobs are stored in the db vs storage
So that I can iterate on this feature area according to user needs

Expectations

• If a blob fails to transfer, it will seamlessly continue to be served from the database and will be retried within a week
• Trying to upload the same blob multiple times has no effect (e.g. if I restore a db backup that still had blobs that already had been migrated)

Out of scope

These stories could add value but are currently considered out of scope.

As a Central administrator with CLI access to the server
I want to migrate from storing media in S3-compatible storage back to storing it in the database
So that I can reduce the complexity of my deployment

As a Central administrator with CLI access to the server
I want Central to manage backups of my media in S3-compatible storage
So that I can have a unified backup and restore experience

As an ODK project manager
I want to directly use ODK-collected blobs in S3-compatible storage
So that I can build public dashboards

As an ODK project manager
I want to browse ODK-collected blobs in S3-compatible storage
So that I can download just the ones I need

[alxndrsn]

As a Central administrator

Does this user have access to the server CLI scripts, or only the web frontend?

[lognaturel]

Does this user have access to the server CLI scripts

Yes! I guess we should say Central system administrator.

[matthew-white]

This feature was released after QA tried it out, so I'm going to go ahead and mark it as "behavior verified".