ipfs.add async Iterable not truly asynchronous. Does not function as intended.

[vaultec81]

Hello
I have noticed after doing some experimentation that ipfs-http-client requires the input async iterable to be completed in order for the request to proceed. There is no output until the input async iterable is completed. Meaning the code functions no different then if an array was specified instead.
However, if using js-ipfs in process daemon ipfs.add will function as intended. Returning CIDs through the output async iterable as the input async iterable is processed.
In my use case I am reading the output, and continuing to pass input as it is processed.
Example code

//Source with no definite end.
for await (var out of ipfs.add(source)) {
    console.log(out.cid.string);
    //Never gets executed
}

[achingbrain]

When you see this behaviour are you running against a remote go or js daemon?

[vaultec81]

@achingbrain go-ipfs. I have confirmed it works once async iterable is finished.

[achingbrain]

Is the behaviour the same with a js-ipfs daemon? (I'm assuming when you said "if using js-ipfs ipfs.add will function as intended" you meant an in-process node rather than a remote one).

[vaultec81]

The behavior is different (working as intended) with an in process js-ipfs daemon. Sorry I did not make that clear.

[vaultec81]

Doing some further analysis reveals no request is actually sent to HTTP API Port 5001 for go-ipfs daemon. I believe the issue lies here https://github.com/ipfs/js-ipfs-http-client/blob/e9ce88d2523663b9730a0ffa28c6791f45976549/src/add/form-data.js#L15-L61

[alanshaw]

@vaultec81 are you running your code in the browser or in Node.js? If in the browser, ensure you've setup CORS correctly on your daemon. If in Node.js then what version are you running? There's an issue with ky in Node.js v13 that could be the same problem as this. If you're using v13 please try again with Node.js v12.

[vaultec81]

@alanshaw I am running node v12.

[alanshaw]

Ah, I just re-read your issue and yes this is a problem with how FormData works with the fetch API. File contents may be streamed but FormData needs to know the full list of files before it can be passed to the fetch API.

[hugomrdias]

yet there is no support for streaming Requests in browsers yet, only streaming Responses.
Either we implement chunking support or we wait for browsers to implement streaming.

[vaultec81]

@hugomrdias I don't see it being too hard to implement streamed uploads. it does require some changes to the ipfs http api specifications. I don't if its possible to do streamed multipart uploads. Potentially a websocket connection for /api/v0/add could be used along side the normal mechanism.

Either we implement chunking support or we wait for browsers to implement streaming.

This isn't specifically a browser issue.

[hugomrdias]

browsers can not stream requests using HTTP, no way around it.

[vaultec81]

@hugomrdias Can you please go into more detail what you mean by "stream requests"

[hugomrdias]

one can only send a blob/buffer from browsers using HTTP, there is no way to stream bytes from the browser.

[vaultec81]

@hugomrdias I don't see it being a big deal implementing chunked request multipart uploads, Or better yet websocket which can upload data in a streamed fashion. Although websocket is not exactly HTTP, it is widely supported by the majority of web browsers, and common web infrastructure.

[hugomrdias]

its not that's why i changed this from bug to feature.
we are totally open to PRs that implement chunked uploads. want to give it a go ?

[vaultec81]

@hugomrdias I believe it should still be treated as if its a bug as it causes functionality differences between js-ipfs in process daemon, and js-http-client. I am open to the possibly of submitting a PR. However, I believe it should be up to the IPFS team to develop a universal solution.

[achingbrain]

The problem here is that in order for ipfs.add to be streaming in the browser, browsers need to support streaming request bodies.

The documented way to do that is to pass a ReadbleStream as the request body, the problem is that no browser has implemented this - see the Send ReadableStream in request body line of the MDN docs.

It doesn't help that the MDN docs talk about it like it's something you can use today.

I think we have a few options:

1. Ditch RPC over HTTP and do it over websockets
2. Chunked uploads as per [WIP] feat: support chunked add requests #1540
3. Copy some of the IPFS core logic into the client, e.g. get it to use the unixfs-importer and orchestrate the various other API calls the core internals make when importing files

My thoughts are not fully-formed on the matter but I think my preference would be for 3 though it'll have an affect on the bundle size.

[bluelovers]

maybe some thing like this
https://javascript.info/resume-upload