diff --git a/src/ipips/ipip-0410.md b/src/ipips/ipip-0410.md new file mode 100644 index 000000000..0d4535d95 --- /dev/null +++ b/src/ipips/ipip-0410.md @@ -0,0 +1,95 @@ +--- +title: "IPIP-0410: Streaming NDJSON in Routing HTTP API" +date: 2023-05-12 +ipip: proposal +editors: + - name: Henrique Dias + github: hacdias + url: https://hacdias.com/ +relatedIssues: + - https://github.com/ipfs/specs/issues/344 + - https://github.com/ipfs/boxo/pull/18 + - https://github.com/ipfs/kubo/pull/9868 + - https://github.com/ipfs/kubo/pull/9874 +order: 410 +tags: ['ipips'] +--- + +## Summary + +Introduce backwards-compatible streaming support to the Routing V1 HTTP API. +For this, we use the `Accept` HTTP header (:cite[rfc9110]) for content type negotiation, as well +as the Newline Delimited JSON ([NDJSON]) format. + +## Motivation + +The main motivation for this change is to allow servers to respond faster to the +client with provider records, as soon as they are available. In the current state, +the client requests a list of providers for a CID from the server. Then, the client +has to wait for the server to collect their final list of providers. After that, +the server can respond with the full list of providers. + +This is a big source of latency when `/routing/v1` is used for delegating DHT lookups, +where the client is forced to wait for the server to finish DHT walk. + +With streaming support, the server is able to respond with provider records as soon +as they are available. This reduces latency and allows for faster content discovery. + +In addition, streaming responses may produce an unlimited amount of results, which +is not the case for non-streamed responses. + +## Detailed Design + +In summary, streaming is supported by using the `Accept` HTTP header, which is used +for content type negotiation as described in :cite[rfc9110]. The client sends an +`Accept` HTTP header starting with `application/x-ndjson`, which is the content +type for [NDJSON]. The following happens: + +- The client adds the `Accept` HTTP header in the request starting with `application/x-ndjson`. +- The server checks the `Accept` HTTP header from the request and, if it contains +`application/x-ndjson`, they reply with NDJSON. If they don't support NDJSON, they +can reply with JSON. +- The server response MUST contain a `Content-Type` HTTP header indicating the +response type, which may be either `application/json` for non-streaming responses, +and `application/x-ndjson` for streamed responses. + +For more details regarding the design, check :cite[http-routing-v1]. + +## Design Rationale + +This feature is designed such that it does not break compatibility with existing +clients and servers. The `Accept` HTTP header is OPTIONAL. By default, the server +MUST respond with `application/json` unless the client explicitly asked for +`application/x-ndjson`. If the server does not support NDJSON, it is allowed +to still respond with non-streamed JSON. + +### User Benefit + +Users (clients) will benefit from this change as the servers will now be able +to respond more promptly to provider record requests. Instead of waiting for the whole +list to be constructed, servers can now return each provider record one by one, +in a streaming fashion. + +The client will be able to close connection at any time, reducing load on both ends. + +The main use cases for this IPIP are light clients and services which want to +delegate DHT lookups to external service. With streaming, clients will be able +to receive results as soon the delegated service learns about new record, which +directly impacts the content load speeds perceived by the end user. + +### Compatibility + +The introduced changes are backwards-compatible. The introduced header is completely +optional, and a server that does not support streaming is able to respond with a non-streaming +response to the client. Equally, non-streaming responses are the default. Therefore, a +client that does not support streaming will not receive a streamed response. + +### Security + +Security considerations are equivalent as the ones in :cite[ipip-0337]. + +### Copyright + +Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). + +[NDJSON]: http://ndjson.org/ diff --git a/src/routing/http-routing-v1.md b/src/routing/http-routing-v1.md index 5e6e9e412..e521aa604 100644 --- a/src/routing/http-routing-v1.md +++ b/src/routing/http-routing-v1.md @@ -11,13 +11,16 @@ editors: github: guseggert - name: Masih H. Derkani github: masih + - name: Henrique Dias + url: https://hacdias.com/ + github: hacdias xref: - ipns-record order: 0 tags: ['routing'] --- -Delegated routing is a mechanism for IPFS implementations to use for offloading content routing and naming to another process/server. This specification describes an HTTP API for delegated content routing. +Delegated routing is a mechanism for IPFS implementations to use for offloading content routing and naming to another process/server. This specification describes a vendor-agnostic HTTP API for delegated content routing. ## API Specification @@ -144,7 +147,34 @@ This API does not support pagination, but optional pagination can be added in a ## Streaming -This API does not currently support streaming, however it can be added in the future through a backwards-compatible update by using a content type other than `application/json`. +JSON-based endpoints support streaming requests made +with `Accept: application/x-ndjson` HTTP Header. + +Steaming responses are formatted as +[Newline Delimited JSON (ndjson)](https://github.com/ndjson/ndjson-spec), +with one result per line: + +```json +{"Schema": "", ...} +{"Schema": "", ...} +{"Schema": "", ...} +... +``` + +:::note + +Streaming is opt-in and backwards-compatibile with clients and servers that do +not support streaming: + +- Requests without the `Accept: application/x-ndjson` header MUST default to + regular, non-streaming, JSON responses. +- Legacy server MAY respond with non-streaming `application/json` response even + if the client requested streaming. It is up to the client to inspect + the `Content-Type` header before parsing the response. +- The server MUST NOT respond with streaming response if the client did not + explicitly request so. + +::: ## Error Codes