docs.rs link is missing from search results

[jturner314]

For crates without a documentation key in their Cargo.toml, crates.io is supposed to generate a Documentation link to docs.rs. However, this Documentation link is sometimes missing on search results.

Steps to reproduce:

1. Open a fresh browser instance. (For example, I'm using chromium --user-data-dir="$(mktemp -d)" on Arch Linux.)
2. Navigate to https://crates.io/search?q=ndarray-npy
3. Observe that there is no Documentation link for the ndarray-npy crate. (It should be there, but it is missing. That's what this issue is about.)
4. Click on the ndarray-npy crate link (URL: https://crates.io/crates/ndarray-npy).
5. Click the "back" button on your browser to navigate back to the previous page.
6. Observe that the Documentation link has now appeared.

I'm observing this behavior for crates that do not include a documentation key in their Cargo.toml. Currently, examples of this include the ndarray-npy, py_literal, aho-corasick, unreachable, and void crates, among many others.

[jtgeibel]

I think this is a general bug (not just for crates that are missing a documentation key). I think similar behavior will be seen with the following fields as well:

• keywords
• categories
• homepage
• documentation

(I'm not sure if any of these other fields show up on the results page. This may be the only missing field that is observable.)

The search results code constructs the crate data by using the minimal_encodable function, and this defaults those fields to null in the JSON response. Once you click through to the full page for the crate, then a full response is received and cached for the crate. This is why the link shows up after navigating back to the results.

[carols10cents]

When we added auto-linking to docs.rs in this PR, we discussed adding this to the search results page as well. However, we make a request to docs.rs to see if the docs were built successfully before adding the link to the page. We concluded that making a large number of requests to docs.rs wouldn't be a great user experience.

Some other thoughts:

• This is even more of a concern since we just increased the default number of results per page from 10 to 50.
• Now that we have a background job framework, after publishing a crate, we could enqueue a job that polls docs.rs until getting a build status or hitting some timeout, then storing the URL in that version's documentation field if the build_status is true and stopping polling if the build_status is false. Oops, just checked-- documentation is still a per-crate field, not per-version, so for this to work correctly we'd have to move that field to the versions table instead. Or we could watch the feed of releases from docs.rs, which would limit the requests we make of docs.rs to a number independent of the number of crates published. I think this is likely the direction I'd prefer, but whew, this is a bigger change than I expected.
• Onur mentioned in that PR thinking about adding a way to ask docs.rs about the status of many crates at once; as far as i can tell that hasn't been added but there are building blocks that wouldn't make it too bad. I don't know if it's worth adding such a thing to docs.rs if we're going to be the only consumer of it. Waiting for that one request to return and then updating the page to have a bunch of doc links or not might also be a non-optimal user experience.

[withoutboats]

Hey, not to be a bother but I just wanted to mention that now that the auto linking to docs.rs has existed for a while, it has become very common for libraries to no longer include their own documentation key in Cargo.toml. This means it increasingly seems like most libraries don't have links to documentation on the search index.

Sucks that this is such a big change, but if someone were looking for something to hack on in crates.io it would be very appreciated!

[jyn514]

Onur mentioned in that PR thinking about adding a way to ask docs.rs about the status of many crates at once; as far as i can tell that hasn't been added but there are building blocks that wouldn't make it too bad. I don't know if it's worth adding such a thing to docs.rs if we're going to be the only consumer of it. Waiting for that one request to return and then updating the page to have a bunch of doc links or not might also be a non-optimal user experience.

I would be ok with adding an API for fetching many crates at once. I do think we should tailor the API for this use case a bit and only show the status, not the crate details, to make it cheaper. (Right now we show a bunch of metadata about the versions: https://docs.rs/crate/tokio/1.0.0/builds.json)

[jtgeibel]

In our team meeting we discussed that on the search results page it probably makes sense to just link to the crate on docs.rs without providing a version. Then docs.rs would redirect to the latest rendered version available.

This might open small windows of inconsistency, such that if a new version was just published then the search results might show version 0.4.1 but the link redirects to 0.4.0 because the new version hasn't been rendered yet. Or a brand new crate could show up in search results but have no rendered version at all on docs.rs. However, this tradeoff seems acceptable in order to consistently show a fallback link to docs.rs if no link is explicitly provided in Cargo.toml.

[jyn514]

@jtgeibel what if the build for the crate failed? Then crates.io will link to the docs, but they won't be there.

This will also mean you get newer versions that the one currently published, which could be very confusing if they're different major versions.

[jyn514]

This will also mean you get newer versions

Oh wait, I guess crates.io search only shows the latest version. Never mind then.

[eggyal]

Perhaps docs.rs can provide a hook by which crates.io gets notified whenever a new doc build is completed? Then crates.io can update its metadata for that crate without having to make requests of docs.rs.

[jyn514]

Heh, docs.rs was actually hoping to get a hook from crates.io at some point: rust-lang/docs.rs#14. I guess it wouldn't hurt to have both.

[RalfJung]

Related cargo issue: rust-lang/cargo#11777

I honestly thought that not setting documentation is 100% equivalent to setting it to the docs.rs page. This automatic is-the-build-done check sounds like it's actually better to not add a link? Or does the check still happen if I set the link manually?

I think adding the documentation link to the search results as-if it was set in the Cargo.toml would already be a good step forward, even without doing any kind of fancy status check.

[epage]

I agree with the idea of removing the check. I could see the publish process falling back to this when populating the database with a backfill (upto a specific date if docs.rs never had a backfill). Then the javascript could be removed.

For cargo-info, we are taking the approach of always showing the link.

[epage]

For the gap between publish and built, some options

• Keep the javascript but assume its published if its weeks old.
• docs.rs posts a "build in progress" page that links to the queue. This would also be a usability improvement as i always struggle to find the queue.

For build errors, docs.rs should post a page, saying it failed, linking to the error with a link to the previous successful build.

[syphar]

[...] upto a specific date if docs.rs never had a backfill [...]

I don't know if docs.rs ever had an actual backfill, but I'm currently working on some changes to docs.rs ( specifically rust-lang/docs.rs#1011, with some more things attached), which will enable us running the consistency check, which will take care of adding all missing releases.

• docs.rs posts a "build in progress" page that links to the queue. This would also be a usability improvement as i always struggle to find the queue.

This will be made possible after rust-lang/docs.rs#1011 is done, and I was planning something like this.

For build errors, docs.rs should post a page, saying it failed, linking to the error with a link to the previous successful build.

We're already doing this.

So generally the question will be, just for search results, if you rather link to /latest/ on docs.rs, where it might not be the same version as stated in the search results, or you link to the specific latest version on crates.io, which might not be built on docs.rs yet.

And, even when we show in-progress builds on docs.rs, there would be a delay between the publish on crates.io, and docs.rs picking up the new release.

[RalfJung]

So generally the question will be, just for search results, if you rather link to /latest/ on docs.rs, where it might not be the same version as stated in the search results, or you link to the specific latest version on crates.io, which might not be built on docs.rs yet.

I think it should link to the version number that is shown in the search results. Then if that version is still building docs.rs can show a message saying "this is still in the queue" and offer a link to an earlier version that has already been built.

[RalfJung]

rust-lang/cargo#13660 has now been closed because the cargo teams considers this to be a bug in crates.io.

So the status quo is: we have docs saying

If no URL is specified in the manifest file, [crates.io] will
automatically link your crate to the corresponding [docs.rs] page when the
documentation has been built and is available (see [docs.rs queue]).

but these docs do not match reality: no such link appears in the search results. These docs make it sound like it's fine for crate authors to not set any docs URL, it won't make a difference if they have their docs on docs.rs -- but actually that's wrong.

So clearly there is a bug somewhere, when we have docs that directly contradict reality. And yet, this issue is marked as "enhancement", not as "bug". Could we get some clarification from @rust-lang/crates-io on this?

[RalfJung]

The docs here are also bad advice right now due to this behavior.

[LawnGnome]

It feels to me like the right behaviour would be for the search results to implement the same fallback logic as the crate version page — basically, if there's no documentation key, try to find a docs.rs build that matches the crate (presumably at the most recent version).

[epage]

@LawnGnome see #1484 (comment)

rust-lang/docs.rs#2533 being implemented and crates.io always showing docs.rs links would be the ideal solution.

The question is whether we need to block on rust-lang/docs.rs#2533 (and have broken links) or not.

[Turbo87]

FWIW the reason why we've not implemented the same fallback for the search results is to avoid the resulting N+1 queries. we'd basically be sending 10 requests to docs.rs for every search result page, which I don't think makes a lot of sense. we could potentially redirect the requests through our own server and cache the results there, but that certainly needs a bit of design work.

[syphar]

rust-lang/docs.rs#2533 being implemented and crates.io always showing docs.rs links would be the ideal solution.

while this would improve the difference between versions on crates.io and on docs.rs, there are still some minutes of delay between between the release itself, and the build being started.

Also the mentioned PR won't change behavior (yet) for queued builds, which you also would need to provide a stable URL for a certain version.

A simpler approach could be to:

• always link the specific version from crates.io
• have a better 404 page in case the version is not found (Improve behavior after publishing a new version causes /latest/ to lose children docs.rs#2568), perhaps showing other releases, queued releases, in progress builds, and an explanation of remaining edge cases, ...

[syphar]

FWIW the reason why we've not implemented the same fallback for the search results is to avoid the resulting N+1 queries. we'd basically be sending 10 requests to docs.rs for every search result page, which I don't think makes a lot of sense. we could potentially redirect the requests through our own server and cache the results there, but that certainly needs a bit of design work.

I would need to think it through, but we also could activate caching in the CDN for these requests, and then I even assume that the load isn't that big of an issue.

edit: thinking about this, you're probably right that this it's a risk to do these N+1 queries. we could just provide an API that can fetch statuses for a list of releases (then we would have an N+1 in docs.rs on the db).
Not sure if just caching on the single requests would be good enough.