Set HTTP cache-control headers

[ns4plabs]

Currently, someguy does not have a local cache. As a temporary measure we could introduce cache-control HTTP headers and store the results on the reverse proxy.

Is this a reasonable strategy and if so, what is the reasonable time to cache the results? 5 minutes? 1 hour?

[2color]

For reference, since this was already brought up:

https://github.com/protocol/bifrost-infra/issues/2758#issuecomment-1716761794
ipshipyard/waterworks-community#1 (comment)

@aschmahmann Do you have thoughts on whether the cache header configuration belongs in the someguy, boxo (routing server), or the proxy?

I saw that for IPNS we set the cache header based on the TTL in the IPNS record. Though for peer and provider records, it seems like this might be something we want in someguy based on the source, e.g. for cid.contact longer than DHT records which are inherently more ephemeral.

[2color]

Notes from Helia WG meeting (11.1.2024):

Reasons to have different cache headers for empty responses (no providers) and responses with records

For example, when a CID is advertsied, but someguy has yet to get results for, if it's cached for too long, it won't be retrievable in that time frame for which the empty response is cached.

Therefore, we should have a lower cache max-age value if someguy returns an empty response.

Some initial numbers to start with

response with provider/peer records: 5m
empty response/no providers: 15 seconds

[lidel]

Implementing this in someguy requires delaying sending HTTP headers until we have at least one result.

Right now, HTTP headers are set before results can be inspected, these are the places where we would add cache-control based on having at least one result:

• without streaming Accept: application/json https://github.com/ipfs/boxo/blob/79cb4e2886d72af06f3b102982520a743835c02f/routing/http/server/server.go#L461
• with streaming Accept: application/x-ndjson https://github.com/ipfs/boxo/blob/79cb4e2886d72af06f3b102982520a743835c02f/routing/http/server/server.go#L498

[hacdias]

See ipfs/boxo#584

[lidel]

I believe this was closed in ipfs/boxo#584 and ipfs/specs#466