Pkg Protocol: client authentication support (part 2)

[StefanKarpinski]

WIP. The first commit reverts the API part of header support, which was broken anyway since only download actually used the headers argument—I forgot to pass it through in all the other functions. Testing fail.

[codecov]

Codecov Report

Merging #1538 into master will decrease coverage by 1.03%.
The diff coverage is 8.69%.

@@            Coverage Diff             @@
##           master    #1538      +/-   ##
==========================================
- Coverage   86.87%   85.84%   -1.04%     
==========================================
  Files          24       24              
  Lines        5188     5255      +67     
==========================================
+ Hits         4507     4511       +4     
- Misses        681      744      +63

Impacted Files	Coverage Δ
src/PlatformEngines.jl	63.66% <8.69%> (-12.93%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6825b48...9c357bb. Read the comment docs.

[StefanKarpinski]

So, to write this down somewhere public and non-ephemeral, what's being implemented here is support for HTTP authorization with bearer tokens as standardized in RFC6750. This means that authorized access is accomplished by the client by making an HTTPS request including a Authorization: Bearer $access_token header. The Pkg client also supports automatic token refresh, since bearer tokens are recommended to be short-lived (no more than a day). The authorization information is saved locally in $(DEPOT_PATH[1])/servers/$server/auth.toml which is a TOML file with the following fields:

• access_token (REQUIRED): the bearer token used to authorize normal requests
• expires_at (OPTIONAL): an absolute expiration time
• expires_in (OPTIONAL): a relative expiration time
• refresh_token (OPTIONAL): bearer token used to authorize refresh requests
• refresh_url (OPTIONAL): URL to fetch new a new token from

The auth.toml file may contain other fields (e.g. user name, user email), but they are ignored by Pkg. The two other fields mentioned in RFC6750 are token_type and scope: these are omitted since only tokens of type Bearer are supported currently and the scope is always implicitly to provide access to Pkg protocol URLs. Pkg servers should, however, not send auth.toml files with token_type or scope fields, as these names may be used in the future, e.g. to support other kinds of tokens or to limit the scope of an authorization to a subset of Pkg protocol URLs.

Initially, the user or user agent (IDE) must acquire a auth.toml file and save it to the correct location. After that, Pkg will determine whether the access token needs to be refreshed by examining the expires_at and/or exipres_in fields of the auth file. The expiration time is the minimum of expires_at and mtime(auth_file) + expires_in. When the Pkg client downloads a new auth.toml file, if there is a relative exipres_in field, an absolute exipres_at value is computed based on the client's current clock time. This combination of policies allows expiration to work gracefully even in the presence of clock skew between the server and the client.

If the access token is expired and there are refresh_token and refresh_url fields in auth.toml, a new auth file is requested by making a request to refresh_url with an Authorization: Bearer $refresh_token header. Pkg will refuse to make unless refresh_url is an HTTPS URL. Note that refresh_url need not be a URL on the Pkg server: token refresh can be handled by separate server. If the request is successful and the returned auth.toml file is a well-formed TOML file with at least an access_token field, it is saved to $(DEPOT_PATH[1])/servers/$server/auth.toml.

Checking for access token expiry and refreshing auth.toml is done before each Pkg client request to a Pkg server, and if the auth file is updated the new access token is used, so the token should in theory always be up to date. Practice is different from theory, of course, and if the Pkg server considers the access token expired, it may return an HTTP 401 Unauthorized response, and the Pkg client should attempt to refresh the auth token. If, after attempting to refresh the access token, the server still returns HTTP 401 Unauthorized, the Pkg client server will present the body of the error response to the user or user agent (IDE); we'll add a hook to allow the user agent may to handle an auth failure, e.g. by presenting a login page to get a new auth token.

[StefanKarpinski]

This has been tested against a mocked up authenticated PkgServer instance. The testing setup is a bit involved and not something we can check into the Pkg repo at this point, unfortunately. In the future, once we vendor a copy of HTTP, we can use it to test this, but for now, this is going to have to remain untested. Fortunately, it's pretty isolated from the rest of Pkg. Adding proper testing will be part of the great excision of PlatformEngines, replacing it with vendored HTTP and Tar.