Add support for using a proxy for crate downloads

[cswindle]

This pull request adds support for using a proxy to cache downloads of crates from crates.io, it is expected that a lot of the time this will be used in conjunction with source replacement to allow full mirroring of crates.io. This has the advantages that:

• If there is an issue with crates.io you are still able to build using any crate anyone using the cache has previously downloaded
• Reduces bandwidth to crates.io as normally the binaries will be served from the cache

This works by adding the following new config:

[registry]
proxy = "http://a.b.c.d/path/"

or

[registries.cool-private-server]
proxy = http://d.c.b.a/path/"

When this is used any request to download files from crates.io is sent to the server provided with the original path, so for example a request to download the following file:

http://crates.io/api/v1/new -> http://a.b.c.d/path/api/v1/new

This then allows setting up a registry cache in artifactory by performing the following:

• Create a new remote repository
• Set to a generic repository type
• Set a key for this repository (this will form part of the URL that is used in the proxy config)
• Set the URL to "https://crates.io/"

Once that is done any cargo download will be cached in artifactory and future requests will not go to crates.io.

Note that currently I have not written any docs on how to do this and I have also not written any unit tests for the function, as I wanted to ensure that people were happy with the general approach.

[rust-highfive]

r? @alexcrichton

(rust_highfive has picked a reviewer for you, use r? to override)

[alexcrichton]

Hm so I believe that Cargo already supports proxies, so how would this be different than a generic normal proxy?

[cswindle]

I was not aware of the proxy config, but I do not believe that can be made to work with things like artifactory. The main difference is that the generic HTTP proxy is setup to use a HTTP connect request and sets up a tunnel to forward the messages to the destination. In the case of using artifactory, that is acting as the terminating HTTP server and looks exactly the same as a real HTTP server with a particular path, it then sends a new request to the remote server if it does not have the file, it will then store it and use it locally for subsequent requests. This is particularly useful for things like having multiple registries, where you are likely to only want to cache downloads from crates.io.

[withoutboats]

I think what @alexcrichton is talking about is where you route the default registry index to a mirror of the index, but this PR just overrides the dl route in the index, while still using the default index. Is that a good summary of the distinction, @cswindle?

[withoutboats]

cc @wycats also

[cswindle]

@withoutboats, that is exactly what this is for.

[cswindle]

I am also happy to rename the field to something like "cache" as there is already a proxy key in config and it will be less confusing not using the same key name.

[alexcrichton]

Ok thanks for the info @cswindle! In that case I think this sounds a lot like registry mirrors, right? That's something that we've long wanted to support but haven't gotten around to designing/implementing yet. I'm somewhat wary that it'll look like just swapping out a URL, though, so we may want to think of a more targeted solution for just mirrors perhaps?

[cswindle]

It can be used both as part of a registry mirror and also as part of a cache to reduce bandwidth, where in the first you would likely want to have an updated index, but the second you would likely want to only have the cache.

[cswindle]

Within my company I have now setup a cache by mirroring the index and updating the dl key in config.json to point at artifactory, so I have now got a solution for this problem internally (albeit in a way that is a bit of a pain to do).

I now think that the correct way for this to be implemented is to allow updating the dl key (possibly via source replacement), on that basis I am going to close this PR, if there is general interest in updating the dl key then I may sort out an RFC for it, otherwise I will live with our workaround.