-
-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Future of Base.download #27043
Comments
Personally, I think having a first-class implementation of HTTP in the stdlib makes sense. |
I do not like the fact that so many packages implement their own version of download. I think using a single implementation in stdlib everywhere makes the most sense. And in an edge case, where it is necessary to hack this functionality to get through some firewall/policies on corporate network one has to modify only one function and the rest of the ecosystem will just work. |
Should we as a bare minimum for 1.0 unexport |
Yes I think it could make sense to put HTTP in the stdlib. It also seems reasonable to me to separate the client functionality, since that's needed much more often than server. |
Though I have to say this doesn't seem urgent enough to me to block 0.7. |
Triage: It seems too late right now to do anything about this at the moment, and doesn't seem super urgent since packages can always just use HTTP themselves. We can also likely just start using HTTP as the backend in 1.x if we move HTTP in. |
I think we don't even need to do that; let's just leave EDIT Aaaand, that's pretty much exactly what Keno said. |
Now that we're in 1.1 land, let's revisit this. I think it makes sense to have a julia-native HTTP stack in |
Could we just bundle https://github.com/curl/curl-for-win (https://bintray.com/vszakats/generic/curl) with windows installs? It's only ~3MB. Versions of windows newer than Build 17063 |
I don't necessarily see including a base download function that relies on curl as orthogonal to a julia-native HTTP stack with it's own download function. We can either include and update the Thoughts? |
Why don't we just ship |
Seems like a non-starter since we need to download things before Julia is built. |
What do we need to download before Julia is built? |
Tarballs in |
Is there any reason why that needs to be the same as |
No, but if we're going to ship |
Because the The main reason for using libcurl over curl is that it was designed exactly for this purpose, and so we would be able to provide better experience, i.e. we could hook into a standard Julia progress meter, directly handle logging rather than capturing output, etc. |
Using EDIT: In case this wasn't clear, I'm specifically talking about |
Maybe @oxinabox can provide his feedback since he implemented LibCurl supports retry from an existingly stopped download and numerous features that could provide a robust download function. Personally I'm in favor of using LibCurl (I'm not against using HTTP.download but have some reservations)
Indeed, I think this discussion should focus solely on the future of the user facing |
So i added Its features are
If Julia's |
@oxinabox How does HTTP.jl fare with proxies? That's the main concern AFAICT. |
@nalimilan I have no idea. Considering the word proxy has been mentioned in this thread exactly once before your comment now, I wouldn't have thought it was the main concern (I would say main concern is getting off the hack that is shelling out to commandline tools, and all related issues that come from that). If it doesn't work right in HTTP that probably also should be fixed, even if |
Yes, proxies are a huge concern. I want to be able to use Julia at work, and, like many people, must access the internet through a proxy at work. In my experience, wget is most likely to work from behind a proxy. |
For some reasons curl is very slow and fail almost all the time in China. Some guys successfully used aria2c instead especially when downloading binary dependences by redefining Base.download. Is it possible to add aria2c into the list of external programs? And it would be nice if we can choose the tool from the list via say config entry. |
|
I'd like to point out that now that more and more packages are being provided through
In the two first cases the users claimed that A pure Julia client would likely improve the situation -- no more strange interaction with random libraries floating on the system. What do other languages (R, Python, etc...) use to download files from the Internet? |
Here is R https://stat.ethz.ch/R-manual/R-devel/library/utils/html/download.file.html. Seems to be libcurl for Linux and wininet for Windows. |
same here for me: https://discourse.julialang.org/t/curl-certificate-issue-with-pkg3-on-mac/33299/7. |
The
How about we just enhance the following line of code to accept a flag from the environment e.g. Line 40 in 1728ada
|
#37340 may help since we will fully control the download process and it will be libcurl properly wrapped from Julia. |
Since #37340 adds it as a stdlib will |
That was my plan rather than putting it in Base. |
Coupling Base -> stdlibs is a bit similar to type piracy in the sense that "normal" Bas functionality that doesn't require the stdlib start working differently when the stdlib is removed. So if we make a new sysimage that doesn't include the Download stdlib (perhaps because no package in our project depends on it), then what will happen to the |
It will fail: if you want to use |
I think I can safely close this now. The answer is that the future of downloading things is here and it is Downloads.jl, which internally uses |
We currently export a
download
function from Base for downloading files. It has some issues:wget
,curl
orfetch
on unix, or a PowerShell command on Windows. Since these utilities are not bundled with Julia, they can cause problems if they don't exist on the system or are otherwise misconfigured (New download implementation doesn't work on Win7 #26459, file testset fails unless curl is installed (0.6.0) #22783, RPATH issue with bundled CURL binary #17889)As a result, several packages implement their own functionality, notably BinDeps.jl, BinaryProvider.jl, and Pkg3.jl (pending JuliaLang/Pkg.jl#280). In fact, it doesn't appear to be currently used anywhere in Base or stdlib (it is mentioned in a couple of comments in the REPL code).
Some options:
Sys.which()
, use that to findcurl
indownload()
#26559 (comment))cc: @staticfloat @tkelman @samoconnor
The text was updated successfully, but these errors were encountered: