Proposal: Adopt the Include-What-You-Use practice for headers and its implications in CMakeList

[zuyu]

Propose to 1) adopt Include-What-You-Use (IWYU) for header files, and 2) link what-you-use in CMakeList.

Benefits

• reduce compilation time, especially for incremental builds
• reduce object file size by avoid linking unnecessary libraries
• avoid build breaks due to missing headers
  • fix: Add include cstdint explicitly #11594
• avoid linking errors
  • misc(parquet): IWYU and fix linking errors #11653
  • Fix velox_query_replayer linking error #11388
• avoid an inappropriate way to write CMake like add_executable(velox_query_replayer TraceReplayerMain.cpp TraceReplayRunner.cpp)
  • Cleanup velox/tool/trace/CMakeLists.txt #11406

Action Plans

We will first do IWYU for all source files, starting from the ones need minimal changes, might be velox/common. The header file order would follow the global-to-local in #7723. Also, use forward declaration, instead of including a header file, wherever possible.

Then make changes to CMakeList.

Link-what-you-use in CMakeList

Each header, along its cpp file, is a micro library, and only link what uses according to IWYU in source file. See examples blow.

We would provide Python scripts to

• automatically generate CMakeList by scanning the source code, useful for adding a new file
• check for the cyclic dependencies among the CMake linkings in CI
• validate CMakeList in CI
• run cpplint in CI

# Declare micro-libs:
add_library(velox_core_ExpressionEvaluator ../empty.cpp ExpressionEvaluator.h)
add_library(velox_core_PlanNode PlanNode.cpp PlanNode.h)

# Link dependencies:
target_link_libraries(velox_core_ExpressionEvaluator
                      velox_vector_ComplexVector)
target_link_libraries(velox_core_PlanNode
                      fmt
                      velox_connectors_Connector
                      velox_core_Expressions)
                      velox_core_QueryConfig)

# Tests:
add_executable(PlanNode_unittest "${CMAKE_CURRENT_SOURCE_DIR}/tests/PlanNodeTest.cpp")
target_link_libraries(PlanNode_unittest
                      gtest
                      gtest_main
                      velox_common_base_tests_GTestUtils
                      velox_core_PlanNode)
add_test(PlanNode_unittest PlanNode_unittest)

Although Velox uses velox_add_library and velox_link_libraries, we could adapt to that.

Note that we might have cyclic dependencies need to break, so expect some code breakdown, and we will discuss the design before we proceed.

[FelixYBW]

@PHILO-HE useful to Gluten?

[PHILO-HE]

@FelixYBW, I think Gluten can benefit from this work. Sometimes, incremental build for Velox requires too much time, even though ccache is already enabled.