Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

invocation specific rustdoc static files are cached too long on /latest/ #1593

Open
syphar opened this issue Jan 14, 2022 · 7 comments
Open

invocation specific rustdoc static files are cached too long on /latest/ #1593

syphar opened this issue Jan 14, 2022 · 7 comments

Comments

@syphar
Copy link
Member

syphar commented Jan 14, 2022

I want to clarify / wrap up a caching issue I'm seeing. We discovered it in the discussion in #1590 (partially fixed by rust-lang/rust#92742).

For rustdoc we serve three kinds of static file types:

  • unversioned-shared-resources like fonts. Will be served from static-root-path with its filename, not hashed. Cached forever, can never change.
  • toolchain-shared-resources are static files that might only change when the toolchain (nightly) changes. Will be served from static-root-path but includes the toolchain version in the filename (via --resource-suffix). Will be cached forever, when these change, the file name and its references in the HTML changes.

Now we come to invocation-specific, for example sitebar-items.js and search-index.js:

  • in the past we only served these under a URL that includes create/version.
  • so under the assumption that a certain release with a certain nightly will always get the same output for these, just adding the same night resource suffix is good enough.

Now with /latest/ the URL for these invocation-specific items don't contain the version any more.

so, if there are two releases of a crate with the same nightly version, the cache will still serve the invocation-specific files for both releases

possible solutions?

  • I'm not sure about the best one. Our first thought was adding a timestamp or the release to the resource-suffix. The problem is that this would also change the suffix for toolchain shared resources. Which would be a performance issue, and also we would have to change static file serving for these.

Another approach could actually be doing the first step towards #1552 by creating an invalidation-request for /crate_name/* after each release.

I'm open for other ideas.
@syphar
Copy link
Member Author

syphar commented Jan 14, 2022

cc @Nemo157 @jsha follow-up on #1590

@jsha
Copy link
Contributor

jsha commented Jan 14, 2022

It seems like there should be a separate suffix for invocation-specific items - in fact, I thought that was the point of invocation-specific, though I hadn't looked at the code too closely. That would be a change in rustdoc - would that solve the problem for docs.rs?

Another possibility would be to make a prefix for invocation-specific items. That prefix could point to the explicitly versioned path (../../0.1.0/<crate_name>/search-index-XXXX.js).

@Nemo157
Copy link
Member

Nemo157 commented Jan 14, 2022

Another possibility would be to make a prefix for invocation-specific items. That prefix could point to the explicitly versioned path (../../0.1.0/<crate_name>/search-index-XXXX.js).

That would have to be a dynamic prefix, the sidebar.js files get put into every subdirectory, so it'd have to prepend the correct number of ../ to get back to the doc-root before adding the ../../<version>/<crate>.

@syphar
Copy link
Member Author

syphar commented Jan 15, 2022

Also one additional thing I remembered: when we go the invalidation-way, we have to forbid caching in the browser for these.

Does @GuillaumeGomez has an opinion on this? This feels more like a design-question for rustdoc.

Should there be an additional suffix for invocation-specific files? Do sidebar-items.js and search-index.js even count as invocation specific?

@syphar
Copy link
Member Author

syphar commented Jan 15, 2022

we could also go another way:
don't cache these files on /lastest/, not in the CDN and not in the browser.

@GuillaumeGomez
Copy link
Member

I don't really have an opinion on this. You have needs and if we can match your needs on rustdoc side without making breaking changes, I think it's fine to do it (depending on the size of the change too).

@jsha jsha mentioned this issue Sep 9, 2022
Merged
@jsha jsha mentioned this issue Sep 19, 2022
Open
@jsha
Copy link
Contributor

jsha commented Sep 19, 2022 •
edited
Loading

If we land rust-lang/rust#101702, the toolchain specific files won't get the resource suffix anymore (which is great, better cacheability!). That means docs.rs is free to add crate-specific info to the --resource-suffix: in particular the crate version number. That would fix this issue. Instead of requesting e.g. https://docs.rs/crate_foo/latest/crate_foo/some_module/sidebar-items-20220517-1.63.0-nightly-4c5f6e627.js, the HTML would request https://docs.rs/crate_foo/latest/crate_foo/some_module/sidebar-items-20220517-1.63.0-nightly-4c5f6e627-1.0.0.js (where 1.0.0 is the version of crate_foo). That could be infinitely cached, since version 1.0.1 of the crate would load a different JS URL.

Edit: Also I should mention I just filed #1851 which proposes a different, more heavyweight intervention: adding an additional flag. What I'm proposing in this comment would be a very lightweight fix (I think); In #1851 I contemplate a more comprehensive but more complicated fix.

@syphar syphar mentioned this issue Sep 26, 2022
Merged
@jsha jsha mentioned this issue Sep 29, 2022
Merged
@syphar syphar mentioned this issue Oct 13, 2022
Closed
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this issue Nov 5, 2022
@matthiaskrgr
Rollup merge of rust-lang#101702 - jsha:static-files2, r=notriddle,Gu…
542cffe 
…illaumeGomez

rustdoc: add hash to filename of toolchain files

All static files used by rustdoc are now stored in static.files/ and their filenames include a hash of their contents. Their filenames no longer include the contents of the --resource-suffix flag. This clarifies caching semantics. Anything in static.files can use Cache-Control: immutable because any updates will show up as a new URL.

Invocation-specific files like crates-NN.js, search-index-NN.js, and sidebar-items-NN.js still get the resource suffix.

This has a useful side effect: once toolchain files aren't affected by resource suffix, it will become possible for docs.rs to include crate version in the resource suffix. That should fix a caching issue with `/latest/` URLs: rust-lang/docs.rs#1593. My goal is that it should be safe to serve all rustdoc JS, CSS, and fonts with infinite caching headers, even when new versions of a crate are uploaded in the same place as old versions.

The --disable-minification flag is removed because it would vary the output of static files based on invocation flags. Instead, for rustdoc development purposes it's preferable to symlink static files to a non-minified copy for quick iteration.

Example listing:

```
$ cd build/x86_64-unknown-linux-gnu/doc/ && find . | egrep 'js$|css$' | egrep -v 'sidebar-items|implementors' | sort
./crates1.65.0.js
./rust.css
./search-index1.65.0.js
./source-files1.65.0.js
./static.files/ayu-2bfd0af01c176fd5.css
./static.files/dark-95d11b5416841799.css
./static.files/light-c83a97e93a11f15a.css
./static.files/main-efc63f77fb116394.js
./static.files/normalize-76eba96aa4d2e634.css
./static.files/noscript-5bf457055038775c.css
./static.files/rustdoc-7a422337900fa894.css
./static.files/scrape-examples-3dd10048bcead3a4.js
./static.files/search-47f3c289722672cf.js
./static.files/settings-17b08337296ac774.js
./static.files/settings-3f95eacb845293c0.css
./static.files/source-script-215e9db86679192e.js
./static.files/storage-26d846fcae82ff09.js
```

Fixes rust-lang#98413
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this issue Nov 5, 2022
@matthiaskrgr
Rollup merge of rust-lang#101702 - jsha:static-files2, r=notriddle,Gu…
3eac639 
…illaumeGomez

rustdoc: add hash to filename of toolchain files

All static files used by rustdoc are now stored in static.files/ and their filenames include a hash of their contents. Their filenames no longer include the contents of the --resource-suffix flag. This clarifies caching semantics. Anything in static.files can use Cache-Control: immutable because any updates will show up as a new URL.

Invocation-specific files like crates-NN.js, search-index-NN.js, and sidebar-items-NN.js still get the resource suffix.

This has a useful side effect: once toolchain files aren't affected by resource suffix, it will become possible for docs.rs to include crate version in the resource suffix. That should fix a caching issue with `/latest/` URLs: rust-lang/docs.rs#1593. My goal is that it should be safe to serve all rustdoc JS, CSS, and fonts with infinite caching headers, even when new versions of a crate are uploaded in the same place as old versions.

The --disable-minification flag is removed because it would vary the output of static files based on invocation flags. Instead, for rustdoc development purposes it's preferable to symlink static files to a non-minified copy for quick iteration.

Example listing:

```
$ cd build/x86_64-unknown-linux-gnu/doc/ && find . | egrep 'js$|css$' | egrep -v 'sidebar-items|implementors' | sort
./crates1.65.0.js
./rust.css
./search-index1.65.0.js
./source-files1.65.0.js
./static.files/ayu-2bfd0af01c176fd5.css
./static.files/dark-95d11b5416841799.css
./static.files/light-c83a97e93a11f15a.css
./static.files/main-efc63f77fb116394.js
./static.files/normalize-76eba96aa4d2e634.css
./static.files/noscript-5bf457055038775c.css
./static.files/rustdoc-7a422337900fa894.css
./static.files/scrape-examples-3dd10048bcead3a4.js
./static.files/search-47f3c289722672cf.js
./static.files/settings-17b08337296ac774.js
./static.files/settings-3f95eacb845293c0.css
./static.files/source-script-215e9db86679192e.js
./static.files/storage-26d846fcae82ff09.js
```

Fixes rust-lang#98413
@syphar syphar mentioned this issue Oct 5, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@Nemo157 @jsha @syphar @GuillaumeGomez