MCP: Resolve documentation links in rustc and store the results in metadata

[petrochenkov]

Proposal

What are doc links

Documentation links or intra-doc links are things like this:

/// Simple [path::to::somewhere].
/// Or fancier [`path`] [type@to] [somewhere#anchor] [else!()].
pub struct S;

Rustdoc extracts such links when processing doc comments (or #[doc = "string"] attributes), resolves Rust paths in them and generates HTML links to corresponding path destinations in the produced documentation.

How rustdoc resolves them

Rustdoc resolves paths in the links

• either during a separate pass that runs right next to regular name resolution in rustc_resolve.
• or at some time later during compilation in a lazy way

Both ways are bad, mostly due to doc inlining.

• In the first case we don't know the full set of links we need to resolve.
  Rustdoc has a thing called doc inlining which may insert an arbitrary set of items from dependency crates into documentation of the current crate.
  • This set is hard to estimate early during name resolution, and if we try to estimate it conservatively the performance penalties are large - up to 25% (see the endless battle for performance in Resolve documentation links in rustc and store the results in metadata rust#94857, which ended in defeat).
  • Besides that, links in inlined items are resolved in the original scope of those items before inlining.
    So we have something like "rustdoc hygiene" very similar to macro 2.0 hygiene that makes it work.
    In fact it works exactly through the partially implemented macro 2.0 support in the compiler, but it's pretty buggy (e.g. preludes are not supported correctly).
• In the second case we need to keep name resolver for much longer than it's correct and desirable to keep it.
  • In fact we have to clone the resolver and the two versions may eventually diverge causing issues.
    See rustdoc: Stop cloning the resolver rust#83761 for the history here.
  • Besides that, we still have to resolve link paths from the whole dependency tree, instead of just the current crate, due to doc inlining. That's also not good for performance.

Proposal

Resolve doc links during regular name resolution in rustc_resolve and store the results to metadata, so rustdoc can just use the readily available results for the current or any dependency crate, instead of doing the resolution itself.

Pros:

• Rustdoc performance is greatly improved - up to 6% in the initial benchmark Resolve documentation links in rustc and store the results in metadata rust#94857 (comment).
• A major architectural issue (support for cloning resolver) that affects both rustdoc and rustc is eliminated.
• "Rustdoc hygiene" is eliminated, correctness of path resolution in links is improved.
• The change is not externally observable (besides the improved correctness), rustc won't produce errors or warnings on broken links or anything like that.

Cons:

• Rustc needs to depend on a markdown parsing library.
  The currently used library is pulldown-cmark, it's MIT licensed and should be compatible with rustc.
• Some logic needs to be moved from rustdoc to rustc_resolve - doc comment gluing and whitespace tweaking, backtick/prefix/suffix stripping, path generic argument stripping (sort of emulation of parse_path from rustc_parse).
  The logic is currently about 300 lines total, and can be found in compiler/rustc_resolve/src/rustdoc.rs in Resolve documentation links in rustc and store the results in metadata rust#94857.
• Rustc performance is slightly regressed - up to 1.8% in the initial benchmark Resolve documentation links in rustc and store the results in metadata rust#94857 (comment).
  The benchmarked prototype wasn't optimized in any way, and I'm pretty sure we can get this number to sub 1% (or even lower if some more filtering logic is moved from rustdoc to rustc_resolve).

Mentors or Reviewers

@GuillaumeGomez @oli-obk

Process

The main points of the Major Change Process are as follows:

• File an issue describing the proposal.
• A compiler team member or contributor who is knowledgeable in the area can second by writing @rustbot second.
  • Finding a "second" suffices for internal changes. If however, you are proposing a new public-facing feature, such as a -C flag, then full team check-off is required.
  • Compiler team members can initiate a check-off via @rfcbot fcp merge on either the MCP or the PR.
• Once an MCP is seconded, the Final Comment Period begins. If no objections are raised after 10 days, the MCP is considered approved.

You can read more about Major Change Proposals on forge.

Comments

This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

[rustbot]

This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

cc @rust-lang/compiler @rust-lang/compiler-contributors

[oli-obk]

@rustbot second