Decentralized Mutable Torrents

[lmatteis]

Discussion in #34

Also included a reference implementation at the bottom.

[arvidn]

I think it may be appropriate to propose an algorithm for determining how frequently to re-announce here. For very popular pieces of content, it could be that having every consumer re-announcing would put too much load on the nodes storing it. I believe @the8472 described an algorithm like this when we originally discussed the put/get protocol for distributed RSS feeds, 5 years ago or so.

basically, if it's possible to estimate how many other peers are re-publishing it, or how frequently it is being republished, one could adjust the probabilities to lower the load.

[the8472]

I think a republishing tuning algorithm is general enough that it should go into BEP44.

For a single-item refresh the main idea would be to check how many nodes returned the current value and only refresh when it falls under the redundancy limit.

[lmatteis]

@arvidn like described here? http://libtorrent.org/dht_rss.html#re-announcing

[lmatteis]

I agree with @the8472 that a "keep-alive" algorithm is general enough to go directly in bep44. After all, other DHT-store apps would need to keep things alive, for different use case (not just for feeds and mutable torrents).

I'm wondering though, doesn't regular DHT employ a similar algorithm for announces?

[ssiloti]

I'm nervous about defining the format of the salt parameter before any use cases for it are even proposed. If salts end up being mostly human readable then URL encoding is obviously appropriate, but if they are usually going to be hashes then it looks oddly inconsistent next to the hex encoded key. Does anyone have any use cases they see for salt values?

[the8472]

Thinking about it, human-readable salts probably are not a good idea since they have a maximum length of 64 bytes, which is fairly short for a name, especially if we consider multi-byte characters.

If we want human-readable names it might be better to have a separate name field and hash that to produce a salt.

I guess the use-case would be a single content provider (same pubkey) providing a bunch of named things?

But if we really wanted to allow human-readable strings we need IRI examples, encoding, test vectors to produce the right salts. Otherwise things are likely to go wrong when someone tries unicode.

Salt in base would be a more conservative approach.

[lmatteis]

My problem with other N base encodings in my opinion is that we're asking some torrent clients to probably import a library just to do the encoding. For instance to use the base64url in WebTorrent, I have to use an external lib. Other bases are widely supported, but add ugly = chars for padding. So base16 (hex) is certainly the safest/easiest to implement.

I agree that also doing Unicode is hard, but certainly well supported. Also 64 bytes seem like a lot for a name:

> Buffer('My salt is pretty huge 漢字 foo bar. Can reach 64 bytes? Yup..', 'utf8').length
64

But it's true that we need tests if we want to go down this human-readable path. And since most magnet links are shared inside browsers, are we sure IRIs are fully supported by most common browsers?

[the8472]

My problem with other N base encodings in my opinion is that we're asking some torrent clients to probably import a library

Not a strong argument in my opinion. Many things need a library or a custom implementation, I don't see how this is any different.

Anyway, we haven't ruled ruled out base16. I'm just agreeing with @ssiloti that urlencoding probably is not a good choice.

As someone mentioned in #34 the advantage of base64 for the pubkey is that one can brute-force some nice-looking keys if desired. And if we're already using base64 for the pubkey we may as well use it for the salt.

Also 64 bytes seem like a lot for a name:

English is not a good test because latin fits into 1 byte per character in utf8. Kanji may not be a good test either because the writing system is more information-dense per character, which may compensate the need for more bytes per character. Arabic, brahmi or hebrew scripts are less byte-dense.

Anyway, we don't have to do the human-readable at all. We can just go with an encoded salt and save ourselves from all the trouble.

But if we want to do it then we should go down the hashing route in my opinion.

are we sure IRIs are fully supported by most common browsers?

Basically, yes. And it's always possible to provide backwards-compatible, percent-encoded URIs if necessary.

[ssiloti]

I like the idea of having the salt use the same encoding as the pubkey with a separate, mutually exclusive, field which is explicitly human readable and hashed to produce the salt value.

[lmatteis]

I guess we don't need the actual salt in the link then?

magnet:?xs=urn:btpk:[ Public Key (Hex) ]&n=[ Name of channel (URL encoded) ]

From a publisher's UI perspective I imagine a section in the settings:

• Enable publishing mutable torrents

<import/export keypair button>

Using public key: 8543d3e6115f0f98c944077a4493dcd543e49c739fd998550a1f614ab36ed63e

Name	Torrent	Generated magnet link
Archive.org-2016	Archive.org-2016-dump.torrent	magnet: ?xs=urn:btpk:8543d3e6115f0f98c944077a4493 dcd543e49c739fd998550a1f614ab36ed63e &n=Archive.org-2016
Archive.org-2012	Archive.org-2012-dump.torrent	magnet: ?xs=urn:btpk:8543d3e6115f0f98c944077a4493 dcd543e49c739fd998550a1f614ab36ed63e &n=Archive.org-2012

last published: 15m ago

---

I'm a little weary about making the name editable, as a publisher could easily lose the link between all of her consumers if she changes such field.

[lmatteis]

So guys I'm a little doubtful about assigning readable names to salts. The use case above seems fragile. Anyway the name of the mutable torrent is given by the actual torrent.

Just a hex salt ?s= parameter seems to make more sense. We can auto-increment it to allow multiple torrents under the same pub key.

magnet:?xs=urn:btpk:8543d3e6115f0f98c944077a4493dcd543e49c739fd998550a1f614ab36ed63e
magnet:?xs=urn:btpk:8543d3e6115f0f98c944077a4493dcd543e49c739fd998550a1f614ab36ed63e&s=1
magnet:?xs=urn:btpk:8543d3e6115f0f98c944077a4493dcd543e49c739fd998550a1f614ab36ed63e&s=2
magnet:?xs=urn:btpk:8543d3e6115f0f98c944077a4493dcd543e49c739fd998550a1f614ab36ed63e&s=3
etc...

Also I'm thinking of changing the name of the BEP into "Updating torrents via DHT mutable items", or "DHT mutable torrents". "Decentralization" is too strong of a word.

[feross]

This is quite an exciting BEP -- thanks for putting in the work to hash this out, @lmatteis!

[ssiloti]

please label this BEP 46

[lmatteis]

@ssiloti done

[ssiloti]

To allow for future extensions there should be a sentence here specifying that data beyond the first 20 bytes should be ignored.

[the8472]

BEP44 accepts any bencoded structure, not just plain strings. so if we want extensibility it probably should be a dictionary with, for now, one key.

[lmatteis]

Also, apart from the magnet link, the client requesting the mutable item has no idea whether it is an updatable torrent.

So perhaps it can be:

{ v: { ih: <20 byte info hash> } }

@ssiloti let me know what you think.

[ssiloti]

A bencoded dict is fine. I just want to avoid painting ourselves into a corner where we can't change the item's value without breaking existing clients.

[lmatteis]

@ssiloti ok check latest changes.

[the8472]

the publisher's?