Feature request: override artifact downloading for specific URLs

[aplavin]

Currently Pkg assumes that artifact URLs can be reached by plain anonymous HTTP GET requests without any specific headers. This is clearly not the case for many services - notably including github releases of private repos. For now I don't see a reasonable way to specify how downloading from specific services/URL happens (see below for a "non-reasonable" approach that works). This ability would be very useful and allow reusing Pkg.Artifacts machinery to handle lazy downloads and invalidation.

I have a working example for private github repos that looks like this:

# in Artifacts.toml:
# [<art_name>]
# url = https://api.github.com/repos/<user>/<repo>/releases/assets/<asset_id>
# lazy = true because it cannot be downloaded before importing the code below

# has to be imported before using artifacts in the code
# concrete types of positional args - more specific than the original method
function Pkg.PlatformEngines.download(url::String, dest::String; kwargs...)
        if occursin(r"^https://api.github.com/repos/.*/releases/assets/.*$", url)
            github_api_download(url, dest)  # set proper headers and download
        else
            # call original method for other URLs
            target_m = only([m for m in methods(Pkg.PlatformEngines.download) |> collect if m.module == Pkg.PlatformEngines])
            invoke(Pkg.PlatformEngines.download, Tuple{fieldtypes(target_m.sig)[2:end]...}, url, dest; kwargs...)
        end
end

# artifacts can be accessed normally afterwards:
artifact"<art_name>"

As you see, this is not extensible and just wrong - that's why I'm asking for a better override mechanism.

[QuantumBits]

Now that the beta for Julia 1.6.0 is available, it looks like it might be easier to address this issue?

https://github.com/JuliaLang/Downloads.jl/blob/master/src/Downloads.jl#L161

Not sure what the best way to manage private headers would be though - proooobably not a great idea to store that info in the Artifacts.toml file, but might be possible to look in an environment variable?

It would be great to be able to specify downloads that need custom headers for private repos!

[QuantumBits]

One additional use case for this that I would like to see is integration of AWS services, specifically private S3 buckets.

[StefanKarpinski]

Downloads now supports ~/.netrc files (and session cookies): JuliaLang/Downloads.jl#98. I'm not sure if that addresses these requirements. We could potentially have a mechanism for automatically injecting headers based on the URL during downloads, the main question for that would be how to design that mechanism.