FileFetcher/HttpCache should respect cache control headers

[kitsonk]

Current with the Deno dependency cache, we currently don't support cache control headers. Basically once a module is cached, it is never reloaded until it is deleted, unless the --reload flag is specified. At that point, the cached ETag is sent back for efficiency.

While this may not be a huge issue in some cases, some time-sensitive modules and other information will never be invalidated, even though the server is giving clear guidance to how long to persist the data without checking back. For example a "latest" release of a module or information about registry completions which the Deno language server persists won't ever get updated through "normal use".

We have sufficient controls in place to ensure that people can "control" what is in their cache, including package locks and cache only flags, so it makes sense that we should be more "browser like" and respect the cache headers and allow the hoster to give guidance about the remote modules.

[lucacasonato]

I think we have to make a distinction between cached user code, and registry manifests / completions here. Registry manifests and registry completions should respect the cache control headers (and only those). I think however that user code should not. User code should be indefinitely cached, like it is now, and only reloaded when specifying the --reload flag. Changing this behaviour would be a huge breaking change.

I think the registry completions and manifests should be cached separately from the user code. We store user code in the deps and gen folders. We should store registry completions separately in a registry folder IMO.

[kitsonk]

@lucacasonato that is already being done (the seperate path)...

[wperron]

Now that we have SQLite in the dependencies, I wonder if we could be able to leverage it make an efficient index of the cached files and their cache headers?

[kitsonk]

We now cache an empty document when the configuration is not available for an origin. We should also write out a expiration header that is +24 hours, so that we try a maximum of once every 24 hours to check the host.