Redirect Opt-out via URL Hints

[lidel]

TL;DR: this PR adds support for x-ipfs-no-redirect x-ipfs-companion-no-redirect (#510 ) symbol as a way to disable gateway redirect for a single request, without disabling anything globally.

That is all. The rest is just some cleanup/refactoring without any functional changes.

Background

This PR adds an URL-based opt-out from global gateway redirect that website developers can rely on to ensure request goes to the specific HTTP gateway.

Potential consumers of this feature:

• Public gateway preloaders (IPFS Companion already does this for uploads)
• Health checks (eg. https://ipfs.github.io/public-gateway-checker)
• Self-hosted clusters of IPFS gateways optimized for specific use case (eg. video streaming)

Details

• IPFS Path normalization and potential redirect happens in webRequest.onBeforeRequest.
  • This is where we skip redirect if a hint (x-ipfs-no-redirect x-ipfs-companion-no-redirect) is present in URL.
• The opt-out hint can be anywhere, but we expect people to use it:
  • ad-hoc as a part of hash (cache-friendly way): https://ipfs.io/ipfs/QmbWqxBEKC3P8tqsKc98xmWNzrzDtRLMiMPL8wBuTGsMnR#x-ipfs-companion-no-redirect
  • as one of query args: https://ipfs.io/ipfs/QmbWqxBEKC3P8tqsKc98xmWNzrzDtRLMiMPL8wBuTGsMnR?foo=bar&x-ipfs-companion-no-redirect
• Preloading Change: until now all HTTP HEAD requests were excluded from gateway redirect. This PR changes this behavior: restores consistent handling for all request types and requires explicit redirect opt-out for preload requests to work.

Thoughts?

cc @lgierth @victorbjelkholm @kyledrake

[victorb]

Shouldn't this be done as a header on the request instead of having to change the URL?

[lidel]

@victorbjelkholm that was my first thought, but the idea got hit by API limitations quite fast:

• Gateway redirect happens in onBeforeRequest. This hook does not provide access to headers, but enables us to do redirects..

• We can modify headers in onBeforeSendHeaders, but can't trigger redirect from there. 🙃

But now that I think about it, what we can do is to detect header in onBeforeSendHeaders, store requestId as a skip flag somewhere and then check if skip flag exists in onBeforeRequest.
Will look into this.

Unfortunately, onBeforeSendHeaders happens after onBeforeRequest, so we are unable to know if header is present when redirect decision is made in onBeforeRequest:

[olizilla]

unless I'm missing something, I feel like if a dev wants to force a specific gateway then the urls should include that gateway. Is there a issue with /ipfs/<hash> style addresses being redirected to the global gateway when they shouldn't? Couldn't they just use https://my-rad-gateway/ipfs/<hash>

[lidel]

@olizilla right now we detect IPFS paths on any website and redirect them. This means browser extension opportunistically upgrade transport to local IPFS node everywhere IPFS paths are used.

The problem this PR solves is that there is no opt-out mechanism for rare cases such as public gateway preloads and bug workarounds.

@victorbjelkholm Even if opt-out via a header was doable, it would come at a price: fetch with additional header breaks CORS preflight request, because by default header is not whitelisted for cross-origin requests to gateways.

This means opt-out via HTTP Header would require additional setup step from gateway administrators, namely to include x-ipfs-no-redirect x-ipfs-companion-no-redirect in Access-Control-Allow-Headers, and we can't assume everyone will have a public gateway under own control or be able to work around CORS validation.

[olizilla]

Should we avoid re-writing IPFS urls that explicitly refer to a custom gateway?
It feels like we're dropping some useful info if we do... they are trying to tell us which gateway to use.

/ipfs/<hash> -> redirect to local ipfs gateway
https://ipfs.io/ipfs/<hash> -> redirect to local ipfs gateway
https://dweb.video/ipfs/<hash> -> leave it alone.

[lidel]

Ok, time for some context:
What our browser extension does since 2017 is a global, opportunistic protocol upgrade that is enabled by default. It detects /ipfs/<CID> on any website and redirects it to local HTTP2IPFS gateway. This is enabled by default for every webpage.

The gateway at ipfs.io is not special: the same rules apply to all public gateways.
As long your assets are under a path that starts with valid /ipfs/<CID>/* it will be loaded over IPFS.

What this PR changes is to add optional "opt-out" mechanism: extension looks at URL of requests made to any public gateway and if it sees x-ipfs-no-redirect x-ipfs-companion-no-redirect in will leave request as-is, won't redirect anything. That is the only change proposed here: an explicit opt-out mechanism for website and dapp developers to handle edge cases when they want to be sure request goes to the specific gateway.

ps. Historical discussion about opportunistic HTTP2IPFS upgrade on every website can be found in #16.
I strongly think it should remain "opt-out" (enabled by default) and not "opt-in" due to the tyranny of the default.

[lidel]

I think I was just not clear enough in my initial description what this PR does, sorry about that.

The change is really, really small:

TL;DR: this PR adds support for x-ipfs-no-redirect x-ipfs-companion-no-redirect (#510) symbol as a way to disable gateway redirect for a single request ad-hoc, without disabling anything globally.

https://ipfs.io/ipfs/QmbWqxBEKC3P8tqsKc98xmWNzrzDtRLMiMPL8wBuTGsMnR#x-ipfs-companion-no-redirect

function isSafeToRedirect (request, runtime) {
  // Do not redirect if URL includes opt-out hint
  if (request.url.includes('x-ipfs-companion-no-redirect')) {
   return false
 }

That is all. The rest is just some cleanup/refactoring without any functional changes.

[lidel]

Apart from the cleanup, actual change introduced by this PR is 4 lines long.
Let's release it to Beta.

[olizilla]

I don't feel totally comfortable about supporting this. If it's just for companion, perhaps the token to check for should be x-ipfs-companion-no-redirect.