[Clang][CMake] Add CSSPGO support to LLVM_BUILD_INSTRUMENTED #79942
+125
−21
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
llvmbot
added
cmake
|
@llvm/pr-subscribers-clang Author: Amir Ayupov (aaupov) ChangesBuild on Clang-BOLT infrastructure to collect sample profiles for CSSPGO. Differential Revision: https://reviews.llvm.org/D155419 Test Plan: Full diff: https://github.com/llvm/llvm-project/pull/79942.diff 8 Files Affected:
diff --git a/clang/CMakeLists.txt b/clang/CMakeLists.txt
index 47fc2e4886cfc..5d16442ac7bc3 100644
--- a/clang/CMakeLists.txt
+++ b/clang/CMakeLists.txt
@@ -741,11 +741,21 @@ if (CLANG_ENABLE_BOOTSTRAP)
if(BOOTSTRAP_LLVM_BUILD_INSTRUMENTED)
add_dependencies(clang-bootstrap-deps llvm-profdata)
set(PGO_OPT -DLLVM_PROFDATA=${LLVM_RUNTIME_OUTPUT_INTDIR}/llvm-profdata)
+ string(TOUPPER "${BOOTSTRAP_LLVM_BUILD_INSTRUMENTED}" BOOTSTRAP_LLVM_BUILD_INSTRUMENTED)
+ if (BOOTSTRAP_LLVM_BUILD_INSTRUMENTED STREQUAL "CSSPGO")
+ add_dependencies(clang-bootstrap-deps llvm-profgen)
+ list(APPEND PGO_OPT -DLLVM_PROFGEN=${LLVM_RUNTIME_OUTPUT_INTDIR}/llvm-profgen)
+ endif()
endif()
if(LLVM_BUILD_INSTRUMENTED)
+ string(TOUPPER "${LLVM_BUILD_INSTRUMENTED}" LLVM_BUILD_INSTRUMENTED)
add_dependencies(clang-bootstrap-deps generate-profdata)
- set(PGO_OPT -DLLVM_PROFDATA_FILE=${CMAKE_CURRENT_BINARY_DIR}/utils/perf-training/clang.profdata)
+ if (LLVM_BUILD_INSTRUMENTED STREQUAL "CSSPGO")
+ set(PGO_OPT -DLLVM_SPROFDATA_FILE=${CMAKE_CURRENT_BINARY_DIR}/utils/perf-training/clang.profdata)
+ else()
+ set(PGO_OPT -DLLVM_PROFDATA_FILE=${CMAKE_CURRENT_BINARY_DIR}/utils/perf-training/clang.profdata)
+ endif()
# Use the current tools for LTO instead of the instrumented ones
list(APPEND _BOOTSTRAP_DEFAULT_PASSTHROUGH
CMAKE_CXX_COMPILER
diff --git a/clang/cmake/caches/CSSPGO.cmake b/clang/cmake/caches/CSSPGO.cmake
new file mode 100644
index 0000000000000..34159068d5ea3
--- /dev/null
+++ b/clang/cmake/caches/CSSPGO.cmake
@@ -0,0 +1,3 @@
+set(CMAKE_BUILD_TYPE RelWithDebInfo CACHE STRING "")
+set(BOOTSTRAP_LLVM_BUILD_INSTRUMENTED "CSSPGO" CACHE STRING "")
+include(${CMAKE_CURRENT_LIST_DIR}/PGO.cmake)
diff --git a/clang/utils/perf-training/CMakeLists.txt b/clang/utils/perf-training/CMakeLists.txt
index 601f40902fa34..489a097cd49c8 100644
--- a/clang/utils/perf-training/CMakeLists.txt
+++ b/clang/utils/perf-training/CMakeLists.txt
@@ -14,8 +14,33 @@ if(LLVM_BUILD_INSTRUMENTED)
DEPENDS clang clear-profraw ${CLANG_PERF_TRAINING_DEPS}
)
+ add_custom_target(generate-profdata-deps)
+ string(TOUPPER "${LLVM_BUILD_INSTRUMENTED}" uppercase_LLVM_BUILD_INSTRUMENTED)
+ if (uppercase_LLVM_BUILD_INSTRUMENTED STREQUAL "CSSPGO")
+ set(PROFDATA_SAMPLE "--sample")
+ if(NOT LLVM_PROFGEN)
+ find_program(LLVM_PROFGEN llvm-profgen)
+ endif()
+
+ if(NOT LLVM_PROFGEN)
+ message(STATUS "To enable converting CSSPGO samples LLVM_PROFGEN has to point to llvm-profgen")
+ endif()
+
+ # Convert perf profiles into profraw
+ add_custom_target(convert-perf-profraw
+ COMMAND "${Python3_EXECUTABLE}"
+ ${CMAKE_CURRENT_SOURCE_DIR}/perf-helper.py perf2prof ${LLVM_PROFGEN}
+ $<TARGET_FILE:clang> ${CMAKE_CURRENT_BINARY_DIR}
+ COMMENT "Converting perf profiles into profraw"
+ DEPENDS generate-profraw)
+ add_dependencies(generate-profdata-deps convert-perf-profraw)
+ else()
+ add_dependencies(generate-profdata-deps generate-profraw)
+ endif()
+
add_custom_target(clear-profraw
COMMAND "${Python3_EXECUTABLE}" ${CMAKE_CURRENT_SOURCE_DIR}/perf-helper.py clean ${CMAKE_CURRENT_BINARY_DIR} profraw
+ COMMAND "${Python3_EXECUTABLE}" ${CMAKE_CURRENT_SOURCE_DIR}/perf-helper.py clean ${CMAKE_CURRENT_BINARY_DIR} perf.data
COMMENT "Clearing old profraw data")
if(NOT LLVM_PROFDATA)
@@ -26,9 +51,12 @@ if(LLVM_BUILD_INSTRUMENTED)
message(STATUS "To enable merging PGO data LLVM_PROFDATA has to point to llvm-profdata")
else()
add_custom_target(generate-profdata
- COMMAND "${Python3_EXECUTABLE}" ${CMAKE_CURRENT_SOURCE_DIR}/perf-helper.py merge ${LLVM_PROFDATA} ${CMAKE_CURRENT_BINARY_DIR}/clang.profdata ${CMAKE_CURRENT_BINARY_DIR}
+ COMMAND "${Python3_EXECUTABLE}"
+ ${CMAKE_CURRENT_SOURCE_DIR}/perf-helper.py merge ${LLVM_PROFDATA}
+ ${CMAKE_CURRENT_BINARY_DIR}/clang.profdata ${CMAKE_CURRENT_BINARY_DIR}
+ ${PROFDATA_SAMPLE}
COMMENT "Merging profdata"
- DEPENDS generate-profraw)
+ DEPENDS generate-profdata-deps)
endif()
endif()
diff --git a/clang/utils/perf-training/lit.cfg b/clang/utils/perf-training/lit.cfg
index 0bd06c0d44f65..c6b54f461f1e2 100644
--- a/clang/utils/perf-training/lit.cfg
+++ b/clang/utils/perf-training/lit.cfg
@@ -28,6 +28,12 @@ config.name = 'Clang Perf Training'
config.suffixes = ['.c', '.cc', '.cpp', '.m', '.mm', '.cu', '.ll', '.cl', '.s', '.S', '.modulemap', '.test']
cc1_wrapper = '%s %s/perf-helper.py cc1' % (config.python_exe, config.perf_helper_dir)
+if config.llvm_build_instrumented.upper() == "CSSPGO":
+ perf_wrapper = "%s %s/perf-helper.py perf --lbr --call-graph --event=br_inst_retired.near_taken:uppp -- " % (
+ config.python_exe,
+ config.perf_helper_dir,
+ )
+ cc1_wrapper = perf_wrapper
use_lit_shell = os.environ.get("LIT_USE_INTERNAL_SHELL")
config.test_format = lit.formats.ShTest(use_lit_shell == "0")
diff --git a/clang/utils/perf-training/lit.site.cfg.in b/clang/utils/perf-training/lit.site.cfg.in
index fae93065a4edf..0ae05c10b6d88 100644
--- a/clang/utils/perf-training/lit.site.cfg.in
+++ b/clang/utils/perf-training/lit.site.cfg.in
@@ -8,6 +8,7 @@ config.test_exec_root = "@CMAKE_CURRENT_BINARY_DIR@"
config.test_source_root = "@CLANG_PGO_TRAINING_DATA@"
config.target_triple = "@LLVM_TARGET_TRIPLE@"
config.python_exe = "@Python3_EXECUTABLE@"
+config.llvm_build_instrumented = "@LLVM_BUILD_INSTRUMENTED@"
# Let the main config do the real work.
lit_config.load_config(config, "@CLANG_SOURCE_DIR@/utils/perf-training/lit.cfg")
diff --git a/clang/utils/perf-training/perf-helper.py b/clang/utils/perf-training/perf-helper.py
index 959bdba5c98cc..96fe571966b9e 100644
--- a/clang/utils/perf-training/perf-helper.py
+++ b/clang/utils/perf-training/perf-helper.py
@@ -42,14 +42,20 @@ def clean(args):
def merge(args):
- if len(args) != 3:
- print(
- "Usage: %s merge <llvm-profdata> <output> <path>\n" % __file__
- + "\tMerges all profraw files from path into output."
- )
- return 1
- cmd = [args[0], "merge", "-o", args[1]]
- cmd.extend(findFilesWithExtension(args[2], "profraw"))
+ parser = argparse.ArgumentParser(
+ prog="perf-helper merge",
+ description="Merges all profraw files from path into output"
+ )
+ parser.add_argument("profdata", help="Path to llvm-profdata tool")
+ parser.add_argument("output", help="Output filename")
+ parser.add_argument("path", help="Folder containing input profraw files")
+ parser.add_argument("--sample", action="store_true", help="Sample profile")
+ opts = parser.parse_args(args)
+
+ cmd = [opts.profdata, "merge", "-o", opts.output]
+ if opts.sample:
+ cmd += ["--sample"]
+ cmd.extend(findFilesWithExtension(opts.path, "profraw"))
subprocess.check_call(cmd)
return 0
@@ -69,11 +75,19 @@ def merge_fdata(args):
def perf(args):
parser = argparse.ArgumentParser(
- prog="perf-helper perf", description="perf wrapper for BOLT profile collection"
+ prog="perf-helper perf",
+ description="perf wrapper for BOLT/CSSPGO profile collection"
)
parser.add_argument(
"--lbr", action="store_true", help="Use perf with branch stacks"
)
+ parser.add_argument(
+ "--call-graph", action="store_true", help="Collect call graph"
+ )
+ parser.add_argument(
+ "--event", help="PMU event name, defaults to cycles:u",
+ default="cycles:u"
+ )
parser.add_argument("cmd", nargs=argparse.REMAINDER, help="")
opts = parser.parse_args(args)
@@ -82,12 +96,14 @@ def perf(args):
perf_args = [
"perf",
"record",
- "--event=cycles:u",
+ f"--event={opts.event}",
"--freq=max",
"--output=%d.perf.data" % os.getpid(),
]
if opts.lbr:
perf_args += ["--branch-filter=any,u"]
+ if opts.call_graph:
+ perf_args += ["-g", "--call-graph=fp"]
perf_args.extend(cmd)
start_time = time.time()
@@ -123,6 +139,23 @@ def perf2bolt(args):
return 0
+def perf2prof(args):
+ parser = argparse.ArgumentParser(
+ prog="perf-helper perf2prof",
+ description="perf to CSSPGO prof conversion wrapper",
+ )
+ parser.add_argument("profgen", help="Path to llvm-profgen binary")
+ parser.add_argument("binary", help="Input binary")
+ parser.add_argument("path", help="Path containing perf.data files")
+ opts = parser.parse_args(args)
+
+ profgen_args = [opts.profgen, f"--binary={opts.binary}"]
+ for filename in findFilesWithExtension(opts.path, "perf.data"):
+ subprocess.check_call(profgen_args + [f"--perfdata={filename}",
+ f"--output={filename}.profraw"])
+ return 0
+
+
def dtrace(args):
parser = argparse.ArgumentParser(
prog="perf-helper dtrace",
@@ -565,6 +598,7 @@ def genOrderFile(args):
"merge-fdata": merge_fdata,
"perf": perf,
"perf2bolt": perf2bolt,
+ "perf2prof": perf2prof,
}
diff --git a/llvm/CMakeLists.txt b/llvm/CMakeLists.txt
index 485c76b8bb936..4421ce7e59b92 100644
--- a/llvm/CMakeLists.txt
+++ b/llvm/CMakeLists.txt
@@ -879,6 +879,9 @@ set(LLVM_ENABLE_PER_TARGET_RUNTIME_DIR ${LLVM_ENABLE_PER_TARGET_RUNTIME_DIR_defa
set(LLVM_PROFDATA_FILE "" CACHE FILEPATH
"Profiling data file to use when compiling in order to improve runtime performance.")
+set(LLVM_SPROFDATA_FILE "" CACHE FILEPATH
+ "Sampling profiling data file to use when compiling in order to improve runtime performance.")
+
if(LLVM_INCLUDE_TESTS)
# Lit test suite requires at least python 3.6
set(LLVM_MINIMUM_PYTHON_VERSION 3.6)
diff --git a/llvm/cmake/modules/HandleLLVMOptions.cmake b/llvm/cmake/modules/HandleLLVMOptions.cmake
index 0699a8586fcc7..3754dff598451 100644
--- a/llvm/cmake/modules/HandleLLVMOptions.cmake
+++ b/llvm/cmake/modules/HandleLLVMOptions.cmake
@@ -1116,7 +1116,7 @@ endif()
option(LLVM_ENABLE_IR_PGO "Build LLVM and tools with IR PGO instrumentation (deprecated)" Off)
mark_as_advanced(LLVM_ENABLE_IR_PGO)
-set(LLVM_BUILD_INSTRUMENTED OFF CACHE STRING "Build LLVM and tools with PGO instrumentation. May be specified as IR or Frontend")
+set(LLVM_BUILD_INSTRUMENTED OFF CACHE STRING "Build LLVM and tools with PGO instrumentation. May be specified as IR, Frontend, CSIR, CSSPGO")
set(LLVM_VP_COUNTERS_PER_SITE "1.5" CACHE STRING "Value profile counters to use per site for IR PGO with Clang")
mark_as_advanced(LLVM_BUILD_INSTRUMENTED LLVM_VP_COUNTERS_PER_SITE)
string(TOUPPER "${LLVM_BUILD_INSTRUMENTED}" uppercase_LLVM_BUILD_INSTRUMENTED)
@@ -1149,6 +1149,15 @@ if (LLVM_BUILD_INSTRUMENTED)
CMAKE_EXE_LINKER_FLAGS
CMAKE_SHARED_LINKER_FLAGS)
endif()
+ elseif(uppercase_LLVM_BUILD_INSTRUMENTED STREQUAL "CSSPGO")
+ append("-fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fno-optimize-sibling-calls -fpseudo-probe-for-profiling"
+ CMAKE_CXX_FLAGS
+ CMAKE_C_FLAGS)
+ if(NOT LINKER_IS_LLD_LINK)
+ append("-fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fno-optimize-sibling-calls -fpseudo-probe-for-profiling"
+ CMAKE_EXE_LINKER_FLAGS
+ CMAKE_SHARED_LINKER_FLAGS)
+ endif()
else()
append("-fprofile-instr-generate=\"${LLVM_PROFILE_FILE_PATTERN}\""
CMAKE_CXX_FLAGS
@@ -1199,6 +1208,21 @@ if(LLVM_PROFDATA_FILE AND EXISTS ${LLVM_PROFDATA_FILE})
endif()
endif()
+if(LLVM_SPROFDATA_FILE AND EXISTS ${LLVM_SPROFDATA_FILE})
+ if ("${CMAKE_CXX_COMPILER_ID}" MATCHES "Clang" )
+ append("-fpseudo-probe-for-profiling -fprofile-sample-use=\"${LLVM_SPROFDATA_FILE}\""
+ CMAKE_CXX_FLAGS
+ CMAKE_C_FLAGS)
+ if(NOT LINKER_IS_LLD_LINK)
+ append("-fpseudo-probe-for-profiling -fprofile-sample-use=\"${LLVM_SPROFDATA_FILE}\""
+ CMAKE_EXE_LINKER_FLAGS
+ CMAKE_SHARED_LINKER_FLAGS)
+ endif()
+ else()
+ message(FATAL_ERROR "LLVM_SPROFDATA_FILE can only be specified when compiling with clang")
+ endif()
+endif()
+
option(LLVM_BUILD_INSTRUMENTED_COVERAGE "Build LLVM and tools with Code Coverage instrumentation" Off)
option(LLVM_INDIVIDUAL_TEST_COVERAGE "Emit individual coverage file for each test case." OFF)
mark_as_advanced(LLVM_BUILD_INSTRUMENTED_COVERAGE)
|
aaupov
requested review from
petrhosek,
zmodem,
wlei-llvm,
llvm-beanz,
htyu and
WenleiHe
|
✅ With the latest revision this PR passed the Python code formatter. |
|
Do you have benchmarks on how well this performs relative to standard IRPGO (or on top of it)? |
Couple of things to untangle here:
|
|
Linking the preprint of the paper accepted to CGO'24 which has CSSPGO performance comparison with AutoFDO and Instr PGO (Fig. 6): https://htyu.github.io/paper/Revamping_Sampling_based_PGO_with_Context_sensitivity_and_Pseudo_instrumentation_preprint.pdf |
Ah. Might've been getting this confused with another technique (maybe IRPGO+instrumented context sensitive PGO?). I remember Google saying in https://discourse.llvm.org/t/practical-compiler-optimizations-for-wsc-workshop-us-llvm-dev-meeting-2023/73998 that a valid pipeline was doing two rounds of profiling with different levels of instrumentation. Thanks for the link to the existing performance results.
Right. Not sure if you've seen #78879, but that should allow making that pretty easy to do. There's also #77347 proposing to change the default, but there hasn't really been a consensus on that one. Thank you for the additional information. Sorry for the derailment/noise. |
aaupov
force-pushed
the
clang-bolt
branch
2 times, most recently
from
53e8dc8
to
cb724ca
Compare
aaupov
force-pushed
the
clang-bolt
branch
2 times, most recently
from
efac760
to
2a5a3f2
Compare
Build on Clang-BOLT infrastructure to collect sample profiles for CSSPGO. Add clang/cmake/caches/CSSPGO.cmake to automate CSSPGO Clang build. Differential Revision: https://reviews.llvm.org/D155419
aaupov
added a commit
to aaupov/llvm-devmtg-2022
that referenced
this pull request
Sep 25, 2024
Can be used with llvm/llvm-project#79942
aaupov
added a commit
to aaupov/llvm-devmtg-2022
that referenced
this pull request
Sep 26, 2024
Can be used with llvm/llvm-project#79942
|
Ping @boomanaiden154, @WenleiHe, @wlei-llvm. I've updated the diff to use external training set (CLANG_PGO_TRAINING_DATA_SOURCE_DIR), and included performance testing results. |
|
Ping @wlei-llvm @WenleiHe |
aaupov
added a commit
that referenced
this pull request
Nov 12, 2024
Match inline trees first between profile and the binary: by GUID,
checksum, parent, and inline site for inlined functions. Map profile
probes to binary probes via matched inline tree nodes. Each binary probe
has an associated binary basic block. If all probes from one profile
basic block map to the same binary basic block, it’s an exact match,
otherwise the block is determined by majority vote and reported as loose
match.
Pseudo probe matching happens between exact hash matching and call/loose
matching.
Introduce ProbeMatchSpec - a mechanism to match probes belonging to
another binary function. For example, given functions foo and bar:
```
void foo() {
bar();
}
```
profiled binary: bar is not inlined => have top-level function bar
new binary where the profile is applied to: bar is inlined into foo.
Currently, BOLT does 1:1 matching between profile functions and binary
functions based on the name. #100446 will extend this to N:M where
multiple profiles can be matched to one binary function (as in the
example above where binary function foo would use profiles for foo and
bar), and one profile can be matched to multiple binary functions (e.g.
if bar was inlined into multiple functions).
In this diff, ProbeMatchSpecs would only have one BinaryFunctionProfile
(existing name-based matching).
Test Plan: Added match-blocks-with-pseudo-probes.test
Performance test:
- Setup:
- Baseline no-BOLT: Clang with pseudo probes, ThinLTO + CSSPGO
(#79942)
- BOLT fresh: BOLTed Clang using fresh profile,
- BOLT stale (hash): BOLTed Clang using stale profile (collected on
Clang 10K commits back), `-infer-stale-profile` (hash+call block
matching)
- BOLT stale (+probe): BOLTed Clang using stale profile,
`-infer-stale-profile` with `-stale-matching-with-pseudo-probes`
(hash+call+pseudo probe block matching)
- 2S Intel SKX Xeon 6138 with 40C/80T and 256GB RAM, using 20C/40T for
build,
- BOLT profiles are collected on Clang compiling large preprocessed
C++ file.
- Benchmark: building Clang (average of 5 runs), see driver in
aaupov/llvm-devmtg-2022
- Results, wall time, lower is better:
- Baseline no-BOLT: 429.52 +- 2.61s,
- BOLT stale (hash): 413.21 +- 2.19s,
- BOLT stale (+probe): 409.69 +- 1.41s,
- BOLT fresh: 384.50 +- 1.80s.
---------
Co-authored-by: Amir Ayupov <aaupov@fb.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Build on Clang-BOLT infrastructure to collect sample profile for CSSPGO.
Add CSSPGO.cmake and BOLT-CSSPGO.cmake to automate CSSPGO/+BOLT
Clang builds.
Note that
CLANG_PGO_TRAINING_DATA_SOURCE_DIRis required as built-intraining set is inadequate for collecting sampled profile.
Differential Revision: https://reviews.llvm.org/D155419
Test Plan:
Added BOLT-CSSPGO.cmake with same use as BOLT-PGO.cmake,
e.g. for bootstrapped ThinLTO+CSSPGO+BOLT, with CSSPGO profile collected
from LLVM build, and BOLT profile collected from Hello World (instrumentation):
Performance testing with Clang: