Evaluate using Profile-Guided Optimization (PGO) and LLVM BOLT for Perspective C++ data engine

[zamazan4ik]

Hi!

Recently I checked Profile-Guided Optimization (PGO) improvements on many projects. All current results are available here. According to the multiple tests, PGO can help with improving performance in many cases (including databases). Since Perspective cares about the performance, I think trying to optimize Perspective's C++ part with PGO would be a good idea.

I can suggest the following action points:

• Perform PGO benchmarks on Perspective. And if it shows improvements - add a note to the documentation about possible improvements in Perspective performance with PGO.
• Providing an easier way (e.g. a build option) to build scripts with PGO can be helpful for the end-users and maintainers since they will be able to optimize Perspective according to their own workloads.
• Optimize pre-built Perspective binaries with PGO.

Since the Perspective native part is the library, I think the Pydantic-core experience can be reused here. Also, Clang supports PGO for shared libraries.

Maybe testing Post-Link Optimization techniques (like LLVM BOLT) would be interesting too (Clang and Rustc already use BOLT as an addition to PGO) but I recommend starting from the usual PGO.

Here are some examples of how PGO optimization is integrated in other projects:

• Rustc: a CI script for the multi-stage build
• GCC:
  • Official docs, section "Building with profile feedback" (even AutoFDO build is supported)
  • A part in a "wonderful" configure script
• Clang: Docs
• Python:
  • CPython: README
  • Pyston: README
• Go: Bash script
• V8: Bazel flag
• ChakraCore: Scripts
• Chromium: Script
• Firefox: Docs
  • Thunderbird has PGO support too
• PHP - Makefile command and old Centminmod scripts
• MySQL: CMake script
• YugabyteDB: GitHub commit
• FoundationDB: Script
• Zstd: Makefile
• Foot: Scripts
• Windows Terminal: GitHub PR
• Pydantic-core: GitHub PR
• file.d: GitHub PR
• OceanBase: CMake flag

[texodus]

perspective-python benchmarks with PGO enabled (master) vs v2.6.0:

It would be a bit a CI hassle to get this working, from my (limited) experience the profile data format is volatile and we already have multiple llvm toolchains in the mix.

[zamazan4ik]

Thanks a lot for the benchmark! Is it possible to perform "PGO vs non-PGO" benchmark on the same code version? Like "master PGO vs master non-PGO" or "2.6.0 PGO vs 2.6.0 non-PGO"? In this case, it would be much simpler to say that we got the performance boost just from PGO and not from the code difference between versions.

And what do we measure on the Y-axis? Time (in seconds/minutes/hours) or anything else?

[texodus]

There is no relevant code difference between these versions. You can learn more about our benchmark suite here.

[zamazan4ik]

Thanks for the benchmark link!

It would be a bit a CI hassle to get this working, from my (limited) experience the profile data format is volatile and we already have multiple llvm toolchains in the mix.

It could be the problem only if you decide to commit the profile to the repository and use it later (profile caching). You can try to implement the approach like it's done in other projects like pydantic-code - generate profiles on-demand. So if you have multiple LLVM toolchains - you will need multiple on-demand PGO builds (for each LLVM toolchain). Which approach to implement - is up to you.