Improving Uploads of Big Files via Web Browsers

[lidel]

This a a meta-issue tracking known issues and ongoing improvements related to uploads of big (1GB+) files in web browser contexts.

Known Problems with HTTP API

Invalid progress reporting:

• Streaming versions of ipfs.files.add buffer entire data in memory before doing HTTP POST
  • upstream: https://github.com/ipfs/js-ipfs-api/issues/842
• Chromium crash when sending <200MB over js-ipfs-api
  • IN PROGRESS waiting for release with upstream fix to stream-http: fix: fix >150mb bodies from crashing jhiesey/stream-http#93
  • related issues
    • crash when dropping a big file (crash when dropping a big file ipfs-companion#464)
    • Uploading big files crush page in Chrome (Uploading big files crush page in Chrome ipfs-inactive/js-ipfs-http-client#654)
    • upload-file-via-browser can only upload file size less than 165MB (upload-file-via-browser can only upload file size less than 165MB ipfs-inactive/js-ipfs-http-client#788)
• Silent crash in Firefox when Buffer max size is breached:
  • upstream issue: crash when dropping a big file ipfs-companion#464
    • IN PROGRESS waiting for release with upstream fix to stream-http: fix: fix >150mb bodies from crashing jhiesey/stream-http#93
• Silent error while files.add over HTTP API (http: invalid Read on closed Body)
  • go-ipfs: Error while files.add over HTTP API (http: invalid Read on closed Body) kubo#5168 (should land in go-ipfs v0.4.19 - Error while files.add over HTTP API (http: invalid Read on closed Body) kubo#5168 (comment))
    • IN PROGRESS set the connection close header if we have a body to read go-ipfs-cmds#116
  • js-ipfs-api: https://github.com/ipfs/js-ipfs-api/issues/797

Ongoing Work

• Fix to stream-http is ready, waiting to be released: fix: fix >150mb bodies from crashing jhiesey/stream-http#93
• Add support for chunked uploads to ipfs.add ([WIP] feat: add support for chunked uploads ipfs-inactive/js-ipfs-http-client#851 | [WIP] feat: support chunked add requests js-ipfs#1540 | go-ipfs/?)

[olizilla]

This looks worth tracking, as it's being in an origin trial in chromium and has support from firefox, edge, and safari

Fetch upload streaming lets web developers make a fetch with a ReadableStream body.

– https://www.chromestatus.com/feature/5274139738767360

The linked design doc is great https://docs.google.com/document/d/e/2PACX-1vQ26giyHPPhxrgVorNtkVSsJPOWecc6c6wHFLFbbl27r0BIBEgRwahh2b37Uk7HfXvZoW1Cu_ed-bRm/pub