Receiving (potentially large) uploads

[chaoran-chen]

This issue is to discuss how we should implement the upload/receival of user submissions, i.e. the way of the data from the submitter's device (browser or CLI) to our (relational core) database. The main challenge is that the uploads can be huge and I think that our software should be able to accept arbitrarily large uploads. The admin can/should (of course) limit the maximum size.

Some thoughts:

• Do we want to be able to resume after a short connection disruption? Maybe we can use tus.io or resumable.js Not relevant for MVP.
• What are the compression formats that we should support? zip, zstd, gz, and xz?
• Should the user also be able to upload large uncompressed files? If they do, maybe we can compress them in the browser/CLI software first? Or do we (for now) simply reject large uncompressed files (e.g., above 1 GB)?
• We need to have protection again zip bombs.
• The implementation needs to be memory-efficient (we shouldn't hold multiple GBs of data in RAM for one upload).

[theosanderson]

I guess there's a general question about what the submission server (backend) will be built using. Have we decided that?
(I presume it could still be Astro, just checking)

[chaoran-chen]

We were thinking about using Kotlin and Spring (which we also use for LAPIS).