Redirect of CID-in-subdomain and DNSLink websites

[lidel]

Parent issues: https://github.com/ipfs/ipfs/issues/337, ipfs/in-web-browsers#89, #527

Problem

There is an open problem of handling CID in Subdomain. CID-in-subdomain gives a website Origin-based isolation if loaded from public gateway, but we continue to have a single Origin for all websites if it is redirected and loaded from local gateway (/ipfs/<cid>).

Current status:

• IPFS Companion extension DOES NOT redirect from cidv1b32.ipfs.foo.tld to http://127.0.0.1:8080/ipfs/cidv1b32 because that would decrease security guarantees (example)

• However, we DO redirect DNSLinked websites eg. https://docs.ipfs.io to http://127.0.0.1:8080/ipns/docs.ipfs.io/ because at the time nobody relied on Origin-based guarantees on IPFS websites.

I feel we should come up with a solution that is addressing both use cases, as having different behavior is confusing.

Short Term Fix

In the short term we have two options: either redirect or not.

( A ) 💢 Redirect to http://127.0.0.1:8080/ip(f|n)s/.. even IF it would decrease Origin-based guarantees

• This assumes that website owner set it up knowing about the lack of Origin-based isolation at the local gateway (detecting Origin change and disabling sensitive features). I am not convinced this is a realistic assumption, and would like to go away from it for DNSLink websites.
• Potential mitigation would be to add redirect opt-out via additional DNS TXT record, but it overcomplicates DNSLink even further

( B ) 👼 (@lidel's plan) DO NOT redirect to http://127.0.0.1:8080/ip(f|n)s/.. by default, but give opt-in option to open at local gateway

• display IPFS page actions and add an entry there to either:
  • Open from Local Gateway – displays a warning dialog informing about Origin limitation of local gateway and if user accepts the risk opens the website from http://127.0.0.1:8080/ip(f|n)s/.. (easy to reason about, so far my favourite, UX can be improved by supporting (F) in Firefox 🍏 )
  • ( B.1 ) For now, we can add option to Enable redirect to Local Gateway - a permanent opt-in to redirect this specific website (may be hard to get the UX right, we may add this as a part of Redirect opt-out per website #633)
  • ( B.2 ) In the future, when local gatewy supports *.ipfs.localhost and *.ipns.localhost we could restore the redirect by default (because it no longer breaks the origin) – see details in Redirect of CID-in-subdomain and DNSLink websites #667 (comment)

I lean on the safe side (B). Below are other options.

Other Ideas

( C ) 💢 "A mess": redirect to a custom hostname for localhost HTTP gateway

• dependency on DNS: *.ipfs.foo.tld → 127.0.0.1 (Hostnames for localhost HTTP gateway in-web-browsers#109)
• as this is not a raw IP, HTTP gateways does not fall under mixed-content whitelist nor is a Secure Context
  • localhost certs are problematic, but without HTTPS going with HTTP means..
    • ..mixed-content errors (Mixed Content error when i view https website #645)
    • ..websites can't run js-ipfs or use window.ipfs due to missing Web Crypto API

( D ) 👼 "Ideal": redirect to native protocol handler

• in the future we may have something like libdweb+experiment: protocol handler API, but even then, it won't be available in all browsers

( H ) 🤔 Service Worker pulls data from 127.0.0.1 and serves it from the <CID>.ipfs.local.dweb.link Origin. It would be origin-white-washing everything on the fly. Should work the same as js.ipfs.io does today except the worker talks to loopback only.

• Actively researched in The Future of "accessing API of remote IPFS node" in-web-browsers#137
• Would have to be HTTP-pull, as WebSocket connections through a loopback HTTP address in Firefox are blocked due to bug-1376309
• Unknown performance characteristics
• Breaks websites that themselves require Service Worker
• Companion should be able to remove the need for adding CORS to local node

( F ) (Firefox only) Open from Local Gateway in (B) could open in a new tab with isolation provided by browser.contextualIdentities API (see #194)

( G ) Like (B) but we could allow redirects for some content types like images or videos. Rationale: subresources are easy to detect via ResourceType – we can detect image and video that way. (Other ways to tell the content-type: look at extension [naive] or fetch a few bytes and do mime-sniff on them [expensive])

• ??? (ideas are welcome)

[lidel]

I think it makes sense to reason about CID-in-subdomain/DNSLinked websites and path-based (/ipfs/<cid>) assets separately.

If asset is referred via IPFS path (/ipfs/<cid>) it should be safe to redirect in most cases. If asset is the main page loaded from an explicitly defined Origin, then we should preserve it by default (B), until we have proper protocol handler APIs (D).

[da2x]

( E ) We’ve got all of 127.0.0.0/8 to work with (16 million + addresses) that are all considered to be a secure context. It’s probably possible to think of a clever way to match CIDs to loopback addresses. We couldn’t guarantee a unique origin as there would potentially be collisions. But it’s a huge improvement over ( A ) that requires a more dedicated attacker with exact knowledge of the hash/origin he want to attack.

( F ) Relay on containerized first-party isolation. Firefox only (but I do think Brave is working on it for Chromium).

( G ) Allow redirects without prompting for confirmation for some content types like image/* and text/plain but require user-confirmation for text/html, application/javascript, etc.

[da2x]

Regarding ( C ): The issues you mention are all addressable: Use a placeholder domain for the origin with a wildcard certificate. The extension redirects to <CID>.ipfs-gateway.local.page. That site is a wildcard domain with a certificate that hosts a simple page that tells people to use a public gateway or install the companion extension. With the extension installed, requests to that origin are hijacked and the extension returns content appearing to be from that origin. Also fixes the "how do I share a link to localhost" issue. Firefox only.

[da2x]

Regarding ( B ): How would that work for sub-resources loaded on a HTTPS page? The pageAction doesn’t display for these today (#662) and it would be really confusing to prompt people to redirect a sub-resource to the local gateway. This would completely break DTube among other CDN-like use cases.

[da2x]

OK, I think I just thought of something that should work for every browser:

( H ) Use what I said above for ( C ) except don’t use webRequest.filterResponseData. Instead, use a WebSocket and a Service Worker that pulls data from 127.0.0.1 and serves it from the <CID>.ipfs-gateway.local.page origin. It would be origin-white-washing everything on the fly. it Should work the same as js.ipfs.io does today except the worker talks to loopback only.

[lidel]

@da2x good feedback, thank you!

( E ) We’ve got all of 127.0.0.0/8 to work with (16 million + addresses) that are all considered to be a secure context.

I am afraid it won't work. Only 127.0.0.1 and ::1 are viable options, as only those raw IPs are whitelisted by both Firefox and Chrome. Fun fact: localhost is not whitelisted: #328.

( F ) Relay on containerized first-party isolation. Firefox only (but I do think Brave is working on it for Chromium).

Use of this requires opening a new tab, which significantly limits the usefulness of this API for background requests. Still, a good idea for isolating /ipfs/ and /ipns/ paths on Firefox when Open from Local Gateway is clicked. Added to the list. Uses of Contextual Identities will be tracked in #194

Regarding ( B ): How would that work for sub-resources loaded on a HTTPS page? The pageAction doesn’t display for these today (#662) and it would be really confusing to prompt people to redirect a sub-resource to the local gateway. This would completely break DTube among other CDN-like use cases.

Should not be a problem. As noted in #662 (comment) we can apply rules to all sub-resources loaded in a tab. In this case, Sub-resources would not be redirected on CID-in-subdomain and DNSLink websites (unless we allow for (G) below).

( G ) Allow redirects without prompting for confirmation for some content types like image/* and text/plain but require user-confirmation for text/html, application/javascript, etc.

Indeed, we can detect image, video and others by looking at ResourceType of each request. Added to the list, but as this type of mixed mode may break websites, should start as an experimental flag.

( H ) WebSocket and Service Worker that pulls data from 127.0.0.1 and serves it from the <CID>.ipfs-gateway.local.page origin.

Somehow related research happens in comments of ipfs/in-web-browsers#137, but service worker route is not robust enough yet. There are pending issues related to keeping worker alive and cross-vendor incompatibilities like bug-1376309. When issues are solved it may be a solution (with some ugly limitations), but before it is ready, it looks like the best we can do is (B) and (F) on Firefox.

[da2x]

( I ) Redirect by default but enforce stricter security policies. Most of the websites that are out there today are static blogs and the like with no need for a strict origin policy. The local gateway can enforce better security in modern browsers by purging data on every load by adding the Clear-Site-Data: "*" response header to everything coming from the local gateway. (Removes service workers, cookies, or anything else persistent between requests.) Could also disable JavaScript and frames unless a strict origin can be guaranteed using a Content-Security Policy when loading from the gateway. Easier to implement than ( G ) and more predictable behaviour for developers. This would allow for today’s DNSLink usecases to still [mostly] function until a better long-term solution can be devised.

[da2x]

( J ) RFC 7838 may offer an alternative solution for DNS-to-IPFS that is currently handled by DNSLink.

A HTTP servers could announce IPFS support by including either of the following headers:

Alt-Srv: ipfs="CID"
Alt-Srv: ipns="dweb.example.com"

The browser could then switch to loading content from the local gateway as the original origin. This depends on browser support to allow extensions to handle Alternate Service announcement keywords like ipfs and ipns, however.

[da2x]

anything.localhost is not treated as a secure origin.

[lidel]

anything.localhost is not treated as a secure origin.

Initial version at https://w3c.github.io/webappsec-secure-contexts/#is-origin-trustworthy read:

If origin’s host component is "localhost" or falls within ".localhost", and the user agent conforms to the name resolution rules in let-localhost-be-localhost, return "Potentially Trustworthy".

Which meant it would be treated the same as 127.0.0.1.

Just as you noted, browser vendors follow more conservative approach from https://www.w3.org/TR/secure-contexts/#is-origin-trustworthy:

If origin’s host component matches one of the CIDR notations 127.0.0.0/8 or ::1/128 [RFC4632], return "Potentially Trustworthy".

https://www.w3.org/TR/secure-contexts/#localhost:

Section 6.3 of [RFC6761] lays out the resolution of localhost. and names falling within .localhost. as special, and suggests that local resolvers SHOULD/MAY treat them specially. For better or worse, resolvers often ignore these suggestions, and will send localhost to the network for resolution in a number of circumstances.
Given that uncertainty, this document errs on the conservative side by special-casing 127.0.0.1, but not localhost.

Safari does not even allow 127.0.0.1: https://bugs.webkit.org/show_bug.cgi?id=171934

---

Some takeaways:

• We could work with browser vendors on changing that. I believe in let-localhost-be-localhost and making it Secure Context, at least for *.ip(f|n)s.localhost which come with content-addressing guarantees

• In parallel, we could look into registering Special-Use *.dweb, similar to *.onion which created a good precedent for being marked as a secure context in Firefox: https://bugzilla.mozilla.org/show_bug.cgi?id=1382359

• Something I'd like to avoid is playing with self-signed cartificates and injecting CA Root to users truststore.

• Even without secure context, HTTP Proxy would be valuable for a lot of use cases, and there will be a way to disable redirect per website (Per-site Redirect Opt-out #687)

[ghost]

I'm not sure what the coverage is in other OSes, but *.localhost subdomains don't work out of the box on all Linuxes. The dnsmasq local resolver and systemd-resolved do resolve them though, so that might be good enough.

[lidel]

@lgierth correct, we identified cross-platform issues in ipfs/in-web-browsers#109 and decided to try something else.

The <cid>.ipfs.localhost mentioned here should work cross-platform because it relies on the idea of on-the-fly domain support via go-ipfs as HTTP proxy (ipfs/kubo#5982) + ipfs-companion setting go-ipfs a proxy for specific requests.

[da2x]

*.localhost behaves according to the RFC for it’s special use domain. (Meaning it doesn’t do squat by default.) However, this isn’t a problem as the extension or a browser proxy configuration can be setup to handle *.,localhost which I assumed was the idea to begin with.

[da2x]

@lidel any progress on using ipns://example.com or ipfs://b32cidv1 directly in browsers? The Beaker Browser’s use of dat://hash have saved them a lot of trouble in this area.

[lidel]

@da2x we created a prototype using experimental WebExtension API libdweb/#protocol-api to deliver persistent ipfs://b32cidv1 in #533, and signaled similar need to Chrome team, but it is hard to tell when such API lands in mainstream browsers (right now it requires special setup and Firefox Nightly).

The HTTP Proxy workaround discussed here aims to improve things much sooner, as we can deliver it without making any changes to browsers.

[lidel]

Ok, I believe with the latest Beta (v2.7.5.762) we have almost all the pieces of the UX for ( B ):

• It enables people to opt-out from redirects per hostname. (Per-site Redirect Opt-out #687)
• It shows IPFS actions on DNSLink websites even when redirect is disabled (Show IPFS Actions on DNSLink Sites #694)

What is missing: in case of DNSLink websites, we want to tweak this behaviour and disable redirect of DNSLink websites by default. People will still be able to opt-in via toggle introduced in (#687) and a global one in Preferences if they choose to do so, but the default should not change the Origin.

Tracked in: #701

[da2x]

I'm not arguing against the decision and preserving the origin boundaries is super important. However, this change also kills IPFS' browser intgration for any purpose other than direct file swapping. Without DNSLink there is no way to publish a website and simultaniously let IPFS Companion users automatically access it on IPFS without manual intervention and a willingness of users to sacrifice security.

Is there a detailed roadmap or plan for future browser integration? or a plan to produce a prototype browser like the Beaker Browser?

[lidel]

@da2x the plan for mainstream browsers is to disable redirect of DNSLink websites until support for localhost HTTP Proxy and <fqdn>.ipns.localhost lands in go-ipfs (ipfs/kubo#5982). That is the best we can do for now.

However, this change also kills IPFS' browser intgration for any purpose other than direct file swapping.

Now that you mentioned it, I see how users may see this as a regression and start asking myself: is the cure is worse than the disease? Namely:

• Does it makes sense to disable DNSLink redirect now, after it being the status quo for so long?
• Perhaps we should just leave it as-is until we can switch local redirect to <fqdn>.ipns.localhost?

---

Is there a detailed roadmap or plan for future browser integration? or a plan to produce a prototype browser like the Beaker Browser?

• Ongoing: We are working with Brave (see notes on bringing IPFS to Brave), which is based on Chromium. Got some initial success there around embedded js-ipfs with self-hosted HTTP Gateway via chrome.sockets APIs (WIP). URLs ipfs:// and ipns:// are planned to be working "natively" there as well, but I can't share any dates yet.

• Long term: ipfs/roadmap is WIP. IPFS Web Browsers WG is growing, and we are having short summit next week to revisit and discuss the WG roadmap for 2019 and beyond.

[da2x]

Now that you mentioned it, I see how users may see this as a regression and start asking myself: is the cure is worse than the disease? Namely:

• Does it makes sense to disable DNSLink redirect now, after it being the status quo for so long?
• Perhaps we should just leave it as-is until we can switch local redirect to .ipns.localhost?

I’ve used static.example.com and video.example.com to serve media assets on www.example.com over IPFS or fallback to HTTP for users without IPFS Companion. DNSLink with automated redirection to localhost in a secure context is the glue that holds that together.

How would you load this over IPFS?

<video><source src="https://video.example.com/video6.webm" type="video/webm"></video>

How about an opt-in signal by extending the existing DNS prefetch mechanism?

<link rel="dns-prefetch" href="//video.example.com" data-ipns-enabled>

IPFS Companion could resolve that domain as an DNSLink IPNS name and load it via the local gateway. Browsers will just resolve the normal DNS name for this origin and ignore the extra attribute. It significantly reduces unnecessary DNSLink lookup requests too.

URLs ipfs:// and ipns:// are planned to be working "natively" there as well, but I can't share any dates yet.

I though IPNS:// couldn’t work? The origin has to be lower-case, right? —and I’ve not even seen a proposal to transition to lowercased base32 encode IPNS names.

[da2x]

@lidel what about proposal J above? It’s what the Tor browser uses to upgrade a .com to a .onion (cutting the need to route traffic through an exit node after the first visit to the web server. It’s a mechanism where the origin is preserved (https://example.com/) while the browser has switched to an alternate service (another HTTPS server, such as an IPFS gateway).

[lidel]

I’ve used static.example.com and video.example.com to serve media assets on www.example.com over IPFS or fallback to HTTP for users without IPFS Companion. DNSLink with automated redirection to localhost in a secure context is the glue that holds that together.

This should continue to work. I clarified in #701 that the idea is to disable redirect only for the root document (main_frame – a URL in the location bar), to keep the Origin intact.
Assets loaded as subresources would continue to be redirected to IPFS.

I though IPNS:// couldn’t work? The origin has to be lower-case, right? —and I’ve not even seen a proposal to transition to lowercased base32 encode IPNS names.

Yes, Origin need to be case-insensitive, and existing PeerIDs in Base58 are not. We had some discussions this week to take ipfs/kubo#5287 over the finish line by wrapping PeerID in CID with a special codec. That way it could be represented in Base32 like everything else.

what about proposal J above? It’s what the Tor browser uses to upgrade a .com to a .onion [..]

Alt-Srv provides seamless upgrade, but right now requires native support from browser vendor. I've created ipfs/in-web-browsers#144 to track it.

WebExtension is unable to register handler for Alt-Srv, but we may look at adding support for the header in Brave.

[lidel]

As of 2019-10 localhost (hostname) is a secure context in both Firefox 69 and Chrome 78.
Firefox does not support *.localhost yet, but everything suggests it will at some point – more details in ipfs/in-web-browsers#109 (comment)

[lidel]

We decided (#701 (comment)) to experiment with temporary measures until we have native support for subdomain gateway in go-ipfs (ipfs/kubo#6498) that works with *.ipfs.localhost and *.ipns.localhost

• v2.9.0.872 (Beta) no longer breaks Origin of DNSLink websites and provides the same UX as content loaded from public subdomain gateways.
  • We may decide to do background prefetch to IPFS node for future offline use (as proposed in Load site via http first, fetch via IPFS in background #710).
  • Feedback welcome!

Update: we restored redirect by default in #831, but there is now a toggle on Preferences screen to control DNSLink redirect behavior.
(We are working towards enabling subdomain gateways in go-ipfs running on localhost, which will provide Origin isolation between different content roots on IPFS. We decided to not change how default configuration of IPFS Companion handles DNSLink websites until that lands.)

[lidel]

PSA: 📢 *.localhost subdomain gateway preview

I just shipped a preview of *.localhost subdomain gateway support (over http proxy to ensure it works on all platforms): v2.10.0.893

It redirects both DNSLink and public subdomain gateways to the local one by default

If you find anything that looks or behaves odd, let us know.
It is pretty significant change security and UX-wise, and I am sure there will be kinks to iron out.

[lidel]

I believe this is closed by v2.11.0.904, but let's keep this issue open until Stable v2.11.

[lidel]

v2.11.0 hit the Stable channel and is ready for go-ipfs 0.5, which will be released in a few weeks.

For now, one can give it a try with go-ipfs 0.5.0-rc1.

Download it from here or use Docker.

Testing ephemeral container:

$ docker run --rm -it --net=host ipfs/go-ipfs:v0.5.0-rc1

Keeping this open until go-ipfs 0.5.0 ships with ipfs-desktop (ipfs/ipfs-desktop#1392)

[lidel]

• IPFS Companion redirects DNSLink websites and public gateways to a local subdomain gateway that provides proper Origin isolation
• go-ipfs 0.5.0 shipped, updated docs in Update notes about hosting own subdomain gateway ipfs-docs#190
• Secure Context issue in Firefox will be solved when https://bugzilla.mozilla.org/show_bug.cgi?id=1220810 is closed