Torrent based cache sharing plugin

[raziel420]

Using torrents has shown time and time to reduce load on centralized servers (such as kerbal stuff, sorry for the hug of death). Many people are against using torrents mainly due to piracy accusations associated with torrents.

I think torrents would be a great way to distribute load and reduce bandwidth usage on servers, the hash function of torrents are typically good enough to maintain versioning. Using a centralized server for listing the torrent files (or more commonly used magnet links these days) can keep files accurate. Using caches of people who already have it downloaded of course will mean less data the central servers give out.

Due to concerns of people over torrent usage, rather then have it even in the default client I advise against, but ckan does support plugins, so a plugin designed to utilize torrents available would be a good alternate resource for those willing to use them. I would even suggest that if the plugin is intalled it uses torrents as a priority over the central repository.

[ChucklesTheBeard]

I have started in on something like this, feature branch is here:
https://github.com/ChucklesTheBeard/CKAN/tree/feature/torrent

From what I can tell, it's not possible to do this entirely in a plugin, because it requires extra metadata, and the classes it would need to rewrite aren't overloadable.

In order to download a torrent, you need, at minimum, a hash of the .torrent file and the size of the target (this is how magnet: links work, with xt=urn:btih:&xl=). That means either adding an optional hash field to the .ckan spec, or keeping a separate repository of hashes matched to .ckan files.

There are no up-to-date / actively developed implementations of the bittorrent protocol in C#, but there are an abundance of bittorrent clients on every platform. Why write a bunch of complicated netcode into CKAN, when others have already done it better? I'm planning on generating a magnet link, then calling Process.Start("magnet link here") and watch the (user-configured) directory for the finished downloads. Http sources from the download field can be added to magnet links as a backup when there are no seeders.

[ghost]

It seem like a good idea from one side, but on the other side, to reduce the bandwidth of the central server, there should me "mirrors" all time up and running anyway. What i mean is, i doubt that every user that download a mod, will seed it at least for a short time. And there are users with limited traffic/month. They dont wanna spare bandwidth for this. This come to having a few dedicated people keeping their torrent clients always running and up to date. At this state of things, it is the same, as it with the current "delivery mechanism" i think. The mechanism is ok, but the infrastructure needs to be thought and implemented. And i may be wrong, but is it easier then completely changing the delivery mechanism and the infrastructure both. Keeping two mechs and structures at the same time looks weird for me and doubtful useful.

[ChucklesTheBeard]

"Should be" is great, but nobody's actually implemented it yet, and the available populated "mirror" (Curse) doesn't allow automated downloads anyway.

i doubt that every user that download a mod, will seed it at least for a short time

That's ok! Torrents work fine without everybody having a >1.0 ratio. They always have. Users can configure their clients to upload as much or as little as they want. This is simply an option to give a little back to the community at low cost.

And there are users with limited traffic/month. They dont wanna spare bandwidth for this.

I intend to implement this as explicitly opt-in. Those with low bandwidth don't have to spare anything.

I'm not sure what you're trying to say regarding changing the mechanism, but this is an extension to the existing download mechanism - almost no existing code will be touched (well, alright, except for possibly breaking up the beast at

CKAN/Core/ModuleInstaller.cs

Line 170 in 85516dd

// TODO: Break this up into smaller pieces! It's huge!

in the process, but that has to happen anyway).

[ghost]

I mean that for now, ckan does not use torrent to get the files. And implementing this would take some effort, and probably involve more issues.