Rework Profile-Guided Optimization (PGO) documentation

[zamazan4ik]

Location

https://doc.rust-lang.org/rustc/profile-guided-optimization.html

Summary

Rust PGO documentation refers to the Clang PGO documentation for further details. But even if the Rust compiler completely relies on the LLVM PGO implementation, there are noticeable differences between PGO implementations in Clang and Rustc + each documentation has its own lifecycle. Some differences can even confuse Rust users (e.g. that Clang mentions -fprofile-instr-generate PGO way (also known as FrontEnd PGO (or simply FE PGO)) but Rust has no support for that).

In this issue, I propose adding missing parts directly to the Rustc PGO documentation. I want to propose the following changes/additions:

Describe in the Rustc docs Sampling PGO. This was already mentioned in #117023

In the documentation, there is a piece of advice about llvm-profdata: "Alternatively, an llvm-profdata coming with a recent LLVM or Clang version usually works too.". This is a dangerous statement since there are no guarantees about preserving backward compatibility for .profraw formats. It's not a theoretical problem - I already met multiple times this problem when during the PGO optimization I mistakenly used system-provided llvm-profdata instead of the right version from the Rust toolchain. I suggest at least writing a warning about such behavior so users will be aware about it.

We need to write backward/forward compatibility guarantees for the profile file format directly in the documentation. Right now it's not easy to find them even in the Clang documentation.

We need to document how to manually implement PGO profiles dumping to the disk or to a memory buffer (for the filesystem-less use case). Clang documentation does not describe Rust-specific details. I already met at least one use case for that for a real project. Most of the useful functions are located here.

We need to document specific LLVM behavior about PGO counters atomicity. By default the counters are not atomic - it could be important for the high-contention apps and could influence the profile reproducibility. Also, we need to document how to switch Rustc to using the atomic counters - right now the only way to do it is to pass -Cllvm-args=-instrprof-atomic-counter-update-all=true to the compiler since the Rustc hasn't a dedicated frontend option for that.

We need to describe well LLVM_PROFILE_FILE environment variable semantic and supported filename modifiers (like %m, %p, and others) in Rustc. Right now in Clang this variable has different supported variables for -fprofile-instr-generate and -fprofile-generate modes - it could be confusing for PGO users with Rustc.

We need to document how to disable PGO instrumentation for specific pieces of a Rust program like it's described for Clang. Is it even possible right now? I do not know and it's not clear at all from the current documentation.

Probably there are other missing pieces of PGO documentation - we can extend this list later.