add Downloads stdlib

[StefanKarpinski]

This PR adds a Downloads stdlib and replaces the Base.download function which relies on a miscellany of external programs (curl, wget, PowerShell) which may or may not happen to be present in order to download files, with Downloads.download which uses libcurl to download files in the same way everywhere. In the process, we also gain support for passing headers and more (to be exposed and used in Pkg in future PRs).

[StefanKarpinski]

Aside from passing CI, I think this PR is mergeable. @KristofferC, you had marked it as "needs docs" but I'm not entirely sure why—was that because the Downloads stdlib doesn't have docs? Currently, I'm just viewing this as a replacement for existing functionality and I've updated the Base.download docs in this PR.

[StefanKarpinski]

The Win64 tester keeps hanging, I have no idea why. It was passing previously, so I'm not sure if this is related to this PR or just because there's something up with the infrastructure. @staticfloat, any idea what's up?

[musm]

This is awesome, and closes many long standing issues.

If this is now a stdlib, shouldn't there now be a depwarn for Base.download?

[StefanKarpinski]

Yes, I could add one since deprecation warnings default to off these days, right?

[StefanKarpinski]

This is awesome, and closes many long standing issues.

Yep, and the next step is to use it for downloads in Pkg as well, so even more problems solved there. Figuring out how to use libcurl correctly from Julia with libuv was pretty tricky, but now that it works, libcurl is great: it's very portable and has every feature we might ever want for downloading/uploading stuff—all the protocols, progress support, authentication, etc. Since I have a good understanding of the API now, if we need any additional features in the future, I'm pretty confident that libcurl has them and that we can add them easily.

[KristofferC]

Regarding the docs question, I just meant an entry in the manual so it is listed as an stdlib with the API it has, like the other stdlibs.

[StefanKarpinski]

Right, I can add that once I've got it working.

[StefanKarpinski]

Ok, well, this finally works. Turns out the Win64 hang was a libcurl bug. Upgrading libcurl fixes it, but requires adding nghttp2 as a dependency. That's fine, since we want to support HTTP/2 anyway. For the record fde145d was the commit that worked.

[StefanKarpinski]

Anything else needed here besides news and a manual section?

[StefanKarpinski]

Deprecating Base.download is non-trivial since we run tests with depwarn=error. As these things so often seem to be, it's more involved than you'd imagine. Having Base.download be an error during testing can be worked around and I have a pull-request that does that, but LibCURL uses Base.download in its tests, which presents a particular challenge. This is fine when it's running its own tests, since those are run with depwarn=no, but when we run the LibCURL tests, we're stricter, so it fails.

The best fix for that is to just change it to use Downloads.download instead of Base.download, but in order for that to work on older Julia's, it requires Downloads to be registered as an official package, a process that I've started but which won't go through for another three days. Once that happens, I can fix LibCURL's tests to use Downloads as a normal package on 1.3-1.5 and as a stdlib on 1.6+, after which it will pass tests as a stdlib with depwarn=error and we can actually deprecate Base.download without breaking CI.

[ViralBShah]

This is pretty amazing!

Off-topic, we really need to move away from having a file for each download hash.

[haampie]

@StefanKarpinski can you try ] add JuMP on this PR, seems like it's broken:

(Pkg) pkg> add JuMP
   Updating registry at `/dev/shm/.julia/registries/General`
  Resolving package versions...
ERROR: Dependency `Test` in target `test` not listed in `deps` or `extras` section.

[fredrikekre]

s

[haampie]

Same here

[fredrikekre]

ERROR: Dependency Test in target test not listed in deps or extras section.

Where is this thrown from?

[haampie]

I've already compiled an older commit, so I cannot easily go back to see :(