This repository has been archived by the owner on Feb 12, 2024. It is now read-only.
ipfs.files.add is missing error handling on invalid response from go-ipfs #2864
Labels
exp/wizard
Extensive knowledge (implications, ramifications) required
kind/bug
A bug in existing code (including security flaws)
pkg:http-client
Issues related only to ipfs-http-client
Comments
|
Ok, so Client-side workaround for While it sounds like something Are there any other options, or is this blocked until upstream fix lands? |
|
This is a blocker to releasing the share files feature in the new web ui, as it's purpose is to let the user upload multiple files for sharing with others. |
|
I think this might be where we might want to throw an error rather than ignore/smother: https://github.com/ipfs/js-ipfs-api/blob/63682dd7078197af5d820234e66aff29552288d4/src/utils/file-result-stream-converter.js#L32-L34 Looks like there's a PR open for go-ipfs now so hopefully this will go away soon, however if someone wants to do something in the mean time I'd start looking there ^^^ |
achingbrain
added
kind/bug
SgtPooki
referenced
this issue
in ipfs/js-kubo-rpc-client
Aug 18, 2022
Adds a server running a gRPC endpoint over websockets running on port 5003, a `ipfs-grpc-client` module to access the server and a `ipfs-client` module that uses the gRPC client with HTTP fallback. This is to solve shortcomings and limitations of the existing HTTP API and addresses the concerns raised in the 'Streaming HTTP APIs and errors, y u no work?' session we had at IPFS team week in NYC. ## Key points 1. Enables full duplex communication with a remote node When making an HTTP request in the browser, a [FormData][] object must be created. In order to add all the values to the FormData object, an incoming stream must be consumed in its entirety before the first byte is sent to the server. This means you cannot start processing a response before the request has been sent, so you cannot have full-duplex communication between client and server over HTTP. This seems unlikely to change in the near future. With a websocket transport for gRPC-web, individual messages can be sent backwards and forwards by the client or the server enabling full-duplex communication. This is essential for things like progress events from `ipfs.add` in the short term, and exposing the full stream capabilities of libp2p via remote client in the long term. 2. Enables streaming errors The existing HTTP API sends errors as HTTP trailers. No browser supports HTTP trailers so when a stream encounters an error, from the client's point of view the stream just stops with no possibility of finding out what happened. This can also mask intended behaviour cause users to incorrectly interpret the API. For example if you specify a timeout to a DHT query and that timeout is reached, in the browser the stream ends without an error and you take away the results you've received thinking all is well but on the CLI the same operation results in a non-zero exit code. A websocket transport has no restrictions here, since full-duplex communication is possible, errors can be received at any time. 3. Listens on websockets with no HTTP fallback gRPC-web exists and is a way of exposing a gRPC service over HTTP. Whereas gRPC supports four modes (unary, e.g. one request object and one response object, client streaming, server streaming and bidirectional streaming), gRPC-web only supports [unary and server streaming](https://github.com/grpc/grpc-web#wire-format-mode). This is due to limitations of the web platform mentioned above and doesn't give us anything over our existing HTTP API. The gRPC-web team are evaluating several options for client and bidirectional streaming, all of which require new capabilities to be added to browsers and none of which will be available in a reasonable time frame. Notably they have [no plans to use websockets](https://github.com/grpc/grpc-web/blob/master/doc/streaming-roadmap.md#issues-with-websockets) as a transport, even though it solves the problems we have today. The team from [improbable](https://improbable.io/) maintain a [gRPC-web-websockets bridge](https://github.com/improbable-eng/grpc-web) which the client added by this PR is compatible with. Their bridge also has a go implementation of a [reverse proxy](https://github.com/improbable-eng/grpc-web/tree/master/go/grpcwebproxy) for use with gRPC servers to turn them into gRPC-web servers with an optional websocket transport. My proposal is to embrace the use of websockets to solve our problems right now, then move to whatever streaming primitive the gRPC-web team settle on in the years to come. As implemented there's only websockets here and no HTTP fallback as the existing HTTP API works fine for unary operations so there's little to be gained by blocking this work on reimplementing the whole of the HTTP API in gRPC-web, and the client can pick and choose which API it'll use per-call. By running the websocket server on a different port to the existing HTTP API it gives us room to add gRPC-web fallback for the API if we find that useful. 4. Has protobuf definitions for all requests/responses See the [ipfs-grpc-protocol](https://github.com/ipfs/js-ipfs/tree/feat/add-grpc-server-and-client/packages/ipfs-grpc-protocol) module, which contains definitions for API requests/reponses. They've been ported from the existing API and will need some checking. The [ipfs-grpc-server/README.md](https://github.com/ipfs/js-ipfs/blob/feat/add-grpc-server-and-client/packages/ipfs-grpc-server/README.md) has a rundown of the websocket communication protocol that was ported from [improbable-eng/grpc-web](https://github.com/improbable-eng/grpc-web). 5. Options as metadata When making a request, metadata is sent during the preamble - these take the form of a string identical to HTTP headers as the initial websocket message - I've used this mechanism to send the options for a given invocation. Notably these are not defined as a protocol buffer, just an unspecified list of simple key/value pairs - maybe they should be to ensure compatibility between implementations? This will be trivial in the implementation in the PR as it contains a server implementation too but to do it in go will require patching or forking the improbable gRPC proxy. 6. Errors as metadata Similar to the existing HTTP API, message trailers are used to send errors. Four fields are used to re-construct the error on the client side: | Field | Notes | | ----- | ----- | | grpc-status | 0 for success, 1+ for error | | grpc-message | An error message | | grpc-stack | A stack trace with `\n` delimited lines | | grpc-code | A string code such as `'ERROR_BAD_INPUT'` that may be used for i18n translations to show a message to the user in their own language | Similar to options these fields are unspecified, if a convention is not enough, perhaps they should be specified as a protobuf and the trailer sent as binary? 7. Streams When sending data as part of an `ipfs.add`, we send repeated messages that contain a path, a content buffer and an index. The index is used to differentiate between streams - path cannot be used as it could be empty. Only the first supplied `path` is respected for a given index. On the server separate input streams are created for each file being added. A file stream is considered closed when an unset or empty content buffer is received. Ultimately this will allow us to apply backpressure on a per-file basis and read from different file streams in parallel and asymmetrically based on the available server capacity. 8. Performance Observed performance pegs gRPC-web over websockets as similar to the HTTP Client with pretty much zero optimisation work performed 9. Security Browsers require TLS for all use of websocket connections to localhost. They do not require it for the loopback address, however, which this PR uses, though loopback means the traffic will not leave the local machine. The incoming requests start as HTTP requests so have a referer header and user agent so would follow the same restrictions as the existing HTTP API. Fixes #2519 Fixes #2838 Fixes #2943 Fixes #2854 Fixes #2864 [FormData]: https://developer.mozilla.org/en-US/docs/Web/API/FormData
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
The Bug
Sometimes (for specific multipart payloads in browser context) the
ipfs.files.addbacked by go-ipfs returns an error in the middle of valid payload:Then, upon receiving this payload,
js-ipfs-apidoes not throw an error, but silently ignores the last line and returns a partial result list with only two first items. (screenshot)A similar problem occurs for some single file uploads (with
wrapWithDirectory: true), js-ipfs-api returns an empty list of responses[](screenshot)Note: I created upstream issue for go-ipfs ipfs/kubo#5168, this issue is about handling "partial responses with error" in js-ipfs-api in a way that does not fail silently.
How to Reproduce
Turns out the
.zipwith sample app triggers the issue 🙃 , so its easy to reproduce::5001upload-multiple-files-via-browser-ipfs-api-bug-demo.zip(uses
ipfs-api ^22.1.1and uploads files viaipfs.files.addwithwrapWithDirectory: true)npm install && npm start, then open form at http://localhost:3000upload-multiple-files-via-browser-ipfs-api-bug-demo.zipThe text was updated successfully, but these errors were encountered: