WIP: Feature/torrent

[ChucklesTheBeard]

Needs review + multi platform testing.

Ideally, it also needs unit testing, but I'm not sure how to implement that for torrents... maybe some checks against a known-good .torrent file, I guess...?

---

This PR is an implementation of a torrent-based download feature that has been requested at least twice in issues (#783, #1591), and multiple times in IRC. With this feature, mods can be distributed in the absence of any file hosts, as long as any one person will seed them.

---

When NetKAN generates the .ckan for a mod,

it can optionally (-T... / --generate-torrents=...) generate everything needed to download the file over the BitTorrent protocol - including the download field as an http seed, if available.

1. A hex string to be used in a magnet link, stored in the .ckan file as "btih" (BitTorrent Information Hash).
2. A .torrent file to a configurable directory, to allow for immediate seeding (this isn't strictly necessary for the official NetKAN instance, but it allows users to set up automated seedboxes if they want).

When a user goes to install a mod,

if the feature is enabled, a magnet link will be generated (and opened) for any mods with a btih field, and the rest will be downloaded the same way they always have. Torrent clients generally let you specify where to put completed files, which works out swimmingly - users have the option of configuring where CKAN looks for completed downloads in the GUI, on the settings page. Once the download completes, it's added to the cache and everything proceeds as usual.

Opt-In

Everything about this is opt-in - mods without licenses which allow redistribution are ignored. If a user doesn't want to share bandwidth, they can configure their external torrent client however they'd like. It doesn't even run unless a user sets the "download complete" directory.

FAQ:

Q: How do we prevent users from getting malware? / Is this secure?

A: The btih string is a SHA-1 cryptographic hash of the .torrent file, which can be reproducibly generated from the .netkan (well, as long as the download url stays static). If you can break SHA-1, you have better things to do with your time than upload malware. It's significantly more secure (as in, "the resulting file is the same as what I asked for") than downloading over HTTP, which does no verification whatsoever.

Q: Don't you need a tracker or something?

A: Trackers speed up the process somewhat, but any client with a DHT feature doesn't need them. Technical spec for the DHT used in the BitTorrent protocl is here, for the curious.

Q: Will this be fast?

A: Depends on how many people use it. More peers = more available upload bandwidth. The .torrent files have a "web-seed" with the original download link, so as a fallback, if the client can't find any peers, the torrent client can still get the file from the original host.

Q: Who will host the original .torrent files?

A: Anyone with access to the netkan metadata and netkan.exe can reproduce the .torrent, which is ideal for seedboxes, because generating the .torrent does require downloading the original file anyway. Generally, clients which have already downloaded the file (and leaves the client open) also supply a copy of the .torrent over the DHT.

I think I can supply .torrent files without seeding actual content, but I'm not sure.

---

Todo list

• https -> http for web seed compatibility
• Discuss: should netkan / ckan spec versions be bumped?
• Write unit tests
• add monotorrent.dll to netkan build scripts
  • Assistance required - ilrepack apparently hates monotorrent.dll...
    • apparently monotorrent tries to look at assembly information, which isn't available?
• ckan.exe crash on updating new ksp install - https://gist.github.com/ChucklesTheBeard/8fcdff49fcd96e848ba9

[ChucklesTheBeard]

rebased onto master because this PR contained some commits which were addressed in #1602.

[Dazpoet]

I am amazed someone finally wrote a PR for this 👍

I think this could be one of the ways we could alleviate the effekts of a potential host shutdown in the future.

[techman83]

@ChucklesTheBeard the internet archive generates Torrent links for all files uploaded to it. The added benefit is that we would then be only re-distributing mods that allow it.

https://archive.org/details/SpaceshuttleSRB-0.5

[wilcoxm]

@ChucklesTheBeard can't you use SHA-256 for torrent checking?