secure urls - poor man's ipfs

[dominictarr]

I've recently been working on this thing https://github.com/dominictarr/web-bootloader

Which uses an idea called a secure url. A secure url is just a url with a base64 encoded hash in it.
it just detects {base64}.sha256. This is similar to https://en.wikipedia.org/wiki/Subresource_Integrity except subresource integrity only applies to scripts and stylesheets. 😢 so you can't use to to have secure XHR for example.

You could also do {alg}-{base64} like in SRI, or enforce that the hash is at the end of the url (right before the ?)

content-addressed data means you have caching that is easy to reason about (finally). If you already have an object with that hash, just don't do the request.

The best part is that it's dead easy to implement.
you can implement a content addressable store in 100 lines in any language.

This makes it very good for bootstrapping other protocols, etc.

what do you think about having first class support for secure urls in beaker?

[pfrazee]

I agree. This is one of the principle ideas for Beaker's new site protocols. It has a lot of beneficial security and caching benefits.

The dat protocol supports this in current master. Its URL scheme is dat://{name_or_key}/{path...}, where:

• name may be a DNS TXT entry that points to a dat-URL that contains a key†
• key may be a hex-encoded public key or sha256 hash

Dat's can be either "live" or "static." If they are static, the author "finalizes" the archive at some point, and the key is a hash of that final state. If they are live, the dats remain mutable.

Live-dats may be easier to work with, but it'd also be possible to create a flow around deploying static dats, where you update the DNS entry with each new deployment. (Zeit Now does this, but they aren't able to implement the cryptographic guarantees in the client like we can.)

We can look at additional protocols with secure URLs. Do you have any in mind? I'm just wrapping up Dat support, now.

† This will need DNSSEC to defeat MITM injections, which isnt implemented yet

[dominictarr]

Oh, I just meant http. It doesn't really matter which protocol, if you know the hash you want. you could even provide a list of possible urls to try, as long as the hash is specified.

If it's mutable, but say, signed with a known key (key is in url) that is still a good security feature, but that is more complex and somewhat less secure. Of course, I'm all for that, but just proposing that we also have access to the simplest possible thing.

[pfrazee]

Ah, for HTTP we'd need to modify the URL scheme somehow, so that the browser knows what to check against. We could standardize some query parameters, for instance:

http://foo.com/resource.js?hash={hash}

Or, perhaps more appropriately, put it in the hash-segment:

http://foo.com/resource.js#sha256-{hash}

[pfrazee]

Another solution might be to put the hash into the filename. It's easy to detect the hash, and you could be very loose about it.

http://foo.com/sha256-{hash}
http://foo.com/sha256-{hash}.zip
http://foo.com/mystuff.sha256-{hash}.zip

Maybe have sha256 be default, so the tag is optional

http://foo.com/{hash}

We've talked about have dat archive-files that can be hosted over HTTP. This scheme could be useful for that. cc @mafintosh

[dominictarr]

sorry, that was what I was trying to suggest initiall.

For sbot i'm serving blobs over https:{host}/blobs/get/&{base64(hash)}.{alg}
I think it's definitely wise to specify a hash. since it's so simple, I don't think it's too much to support multiple syntaxes. {alg}-{base64(hash)} is the SRI syntax, {base64(hash)}.{alg}. camlistore uses {alg}-{hex(hash)}.

stronglink uses /{alg}/{hex(hash)} (and the hash may be truncated)

There are some advantages to base58, but it's much harder to find a base58 implementation.
also, those advantages (easy to copy/paste) arn't so important these days.
(we don't type in a url after seeing an ad on tv, instead links are shared on social media, etc)

[dominictarr]

@btrask

[btrask]

@dominictarr thanks for pinging me!

I think the end-goal should be content addressing, where hashes can be loaded from arbitrary resolvers. This secure URL proposal and SRI are both good intermediate approaches.

Once your browser has a way of verifying hashes embedded in URLs, there's very little reason to actually use the specified source instead of some other resolver. (One consideration being w3c/webappsec-subresource-integrity#22.)

I think it's worth being very careful about this design, since it's carving out a universal namespace across every web server on the net (similar to /.well-known and ?escaped_fragment). Unfortunately that tends to make these proposals very ugly.

My suggestions are:

1. Keep resolvers in mind as a distinct concept and eventual goal
2. Choose a format that makes the embedded content address distinct (ideally as its own URI)
3. Be careful about possible collisions (per above)
4. Aside from that, the primary design/aesthetic constraint should be making it as appealing as possible to engineers of Chrome and Firefox

If you can come up with a decent proposal, I'd be happy to support it in StrongLink. Current raw file URL example, which doesn't follow my suggestions above :). Normal page URL which is much better but obviously highly vulnerable to namespace problems with the name q).

I suspect a query parameter is the way to go. Something like http://example.com/file?_hash_=[hash-uri-of-some-sort].

As for the hash URI itself, I'd suggest supporting all of them until a winner emerges. This JS code might be a helpful starting point (no npm module yet, sorry!).

[BigBlueHat]

@dominictarr fwiw, there was an earlier Sub-Resource Integrity draft that provided the integrity attribute on pretty much any element with an href or src: https://www.w3.org/TR/2014/WD-SRI-20140318/

It also used RFC 6920 URI's for the hashes.

Mostly, that seems to have all been taken out due to how it might effect user experience during navigation--i.e. what happens when I click a link with an invalid integrity match.

There's some fresh consideration of putting integrity on <a> if it has download, but again, navigation UX is the "blocker": w3c/webappsec-subresource-integrity#68

[dominictarr]

I think the problem here is that it's still the link that identifies the resource, not the hash. I think with subresource integrity the server doesn't actually know what hash you were looking for (is this correct?) hence I think it should just be a url convention.