Skip to content
This repository has been archived by the owner on Feb 12, 2024. It is now read-only.

Streaming versions of ipfs.files.add buffer entire data in memory before doing HTTP POST #2863

Closed
lidel opened this issue Aug 29, 2018 · 4 comments
Labels
exp/wizard Extensive knowledge (implications, ramifications) required kind/bug A bug in existing code (including security flaws) pkg:http-client Issues related only to ipfs-http-client

Comments

@lidel
Copy link
Member

lidel commented Aug 29, 2018

ipfs.files.addPullStream([options])
ipfs.files.addReadableStream([options])

Problem Summary

As noted here streaming over HTTP does not work the way people expect: it ends before HTTP transport and entire thing is buffered in memory before HTTP POST is called with contents of the buffer.

This means when HTTP POST is sent to /api/v0/add browser has entire payload in memory.
Streaming of huge files is impossible over HTTP.

This is bad because:

Potential Solutions

Option A: Streams API

Use a fetch call with Streams API.
Looks like something that will be possible in the future, step 10 from fetch.spec mentions ReadableStream

Blocked missing vendor support (https://caniuse.com/#feat=streams)

  • Chrome: No support for BYOB ("bring your own buffer") stream readers
  • Firefox: No support for WritableStream

Option B: Poor Man's Streaming

We could buffer up to a point and then send a chunk as a separate POST call to HTTP API.
Implementation may get ugly very fast, requires increasing complexity of go-ipfs.

Option C: (?)

(?)

Related Issues

cc ipfs-inactive/js-ipfs-http-client#654, ipfs-inactive/js-ipfs-http-client#788, https://github.com/ipfs/js-ipfs-api/issues/748, ipfs/ipfs-companion#464

@hacdias
Copy link
Member

hacdias commented Aug 29, 2018

cc ipfs/ipfs-webui#762

@JiaJiaJiang
Copy link

JiaJiaJiang commented Nov 25, 2018

I haven't read all code of the uploading logic because Iam not familiar with react, but if the data will not be processed before uploading ,here is an advice:
XHR can send a FormData object, and the formdata object can carry files that gave into the input element. This way will not cause the memory problem and all thing are handled by the browser. The uploading progress can be got from the xhr object too, nothing will be blocked.
Here is the example of FormData object with input files: https://developer.mozilla.org/en-US/docs/Web/API/FormData/append

@achingbrain achingbrain transferred this issue from ipfs-inactive/js-ipfs-http-client Mar 9, 2020
@achingbrain achingbrain added kind/bug A bug in existing code (including security flaws) exp/wizard Extensive knowledge (implications, ramifications) required pkg:http-client Issues related only to ipfs-http-client labels Mar 9, 2020
@thienpow
Copy link

thienpow commented Jul 5, 2020

Screenshot 2020-07-06 at 3 36 11 AM

looks like a particular problem of Chromium based browser having this problem. Safari is ok to upload bigger than 150mb.

@lidel
Copy link
Member Author

lidel commented Aug 4, 2020

I believe this was fixed by @Gozala in #3184

@lidel lidel closed this as completed Aug 4, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
exp/wizard Extensive knowledge (implications, ramifications) required kind/bug A bug in existing code (including security flaws) pkg:http-client Issues related only to ipfs-http-client
Projects
None yet
Development

No branches or pull requests

5 participants