Parabuild is a Rust tool that helps you compile complex (single file) projects in parallel, such as some C++/CUDA projects that heavily use templates (when you cannot achieve the best performance through make -j).
The following is an example of how to use parabuild-rust cli to compile a C++ project.
If you are new to rust-lang, you can install it by running the following command:
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | shThen you can install parabuild-rust by running the following command:
cargo install parabuildInstall lsof and rsync, which are required by parabuild-rust:
sudo apt update
sudo apt install -y lsof rsync
We use handlebars templating language to generate source file, here is an example:
// src/main.cpp
#include <iostream>
template <int n>
void print(){
std::cout << n << std::endl;
}
int main(){
print<{{N}}>();
return 0;
}We will use this file to organize a C++ project like this:
example_project
├── CMakeLists.txt
├── src
│ └── main.cppSuppose we want to compile the project with different N values, we can use the following command, where you can use xxx-bash-script to specify what needs to be executed during workspace initialization, compilation, and runtime:
parabuild \
example_project \
build/main \
--init-bash-script "cmake -S . -B build" \
--compile-bash-script "cmake --build build -B" \
--run-bash-script "./build/main" \
--template-file src/main.cpp \
--data '[{"N": 10}, {"N": 20}]' \
-j 1Give it a quick try from scratch by running the following commands:
git clone https://github.com/panjd123/parabuild-rust.git
cd parabuild-rust
cargo run -- \
tests/example_cmake_project \
build/main \
--template-file src/main.cpp \
--data '[{"N": 10}, {"N": 20}]'We have many customization options, please check parabuild --help
$ parabuild --help
A parallel build utility for template heavy projects.
Usage: parabuild [OPTIONS] <PROJECT_PATH> [TARGET_FILES]...
Arguments:
<PROJECT_PATH>
project path
[TARGET_FILES]...
target files in the project, which will be moved between build/run workspaces for further processing
e.g. `build/main,data_generate_when_build`
Options:
-t, --template-file <TEMPLATE_FILE>
template file in the project
-w, --workspaces-path <WORKSPACES_PATH>
where to store the workspaces, executables, etc
[default: workspaces]
--data <DATA>
json format data
-d, --data-file <DATA_FILE>
json format data file, when used together with the `--data` option, ignore this option
-o, --output-file <OUTPUT_FILE>
output the json format result to a file, default to stdout
--init-bash-script <INIT_BASH_SCRIPT>
init bash script
Default to `cmake -S . -B build -DPARABUILD=ON`
--init-bash-script-file <INIT_BASH_SCRIPT_FILE>
init bash script file, when used together with the `--init-bash-script` option, ignore this option
-i, --init-cmake-args <INIT_CMAKE_ARGS>
init cmake args, when used together with the `--init-bash-script` or `--init-bash-script-file` option, ignore this option
e.g. "-DCMAKE_BUILD_TYPE=Release"
--compile-bash-script <COMPILE_BASH_SCRIPT>
compile bash script
Default to `cmake --build build --target all -- -B`
--compile-bash-script-file <COMPILE_BASH_SCRIPT_FILE>
compile bash script file, when used together with the `--compile-bash-script` option, ignore this option
-m, --make-target <MAKE_TARGET>
make target, when used together with the `--compile-bash-script` or `--compile-bash-script-file` option, ignore this option
--run-bash-script <RUN_BASH_SCRIPT>
run bash script
If not provided, we will run the first target file in the `target_files` directly
--run-bash-script-file <RUN_BASH_SCRIPT_FILE>
run bash script file when used together with the `--run-bash-script` option, ignore this option
-s, --silent
do not show progress bar
-j, --build-workers <BUILD_WORKERS>
build workers
-J, --run-workers <RUN_WORKERS>
run workers
We have four execution modes:
1. separate and parallel
2. separate and serial (by default)
3. execute immediately in place
4. do not execute, only compile, move all the TARGET_FILES to `workspaces/targets`
The first one means we will move TARGET_FILES between build/run workspaces. Compile and run in parallel in different places like a pipeline.
The second behavior is similar to the first, but the difference is that we only start running after all the compilation work is completed.
The third method is quite unique, as it does not move the TARGET_FILES and immediately executes the compilation of a workspace in its original location.
To specify these three working modes through the command line:
1. positive numbers represent the first
2. negative numbers represent the second
3. pass `--run-in-place` to represent the third, we will ignore the value of this option
4. 0 represent the fourth
--run-in-place
run in place, which means we will not move the TARGET_FILES between build/run workspaces
--seperate-template
seperate template file, as opposed to using the same file to render in place
--no-cache
Clear the contents in `workspaces` before running
--without-rsync
do not use rsync, which means you will not be able to use incremental replication, which may require you to use `--no-cache` every time you modify the project
--makefile
Mark that you are actually working on a makefile project
pass `data` to `CPPFLAGS` environment variable in the compile bash script
e.g. when data is `{"N": 10}`, `CPPFLAGS=-DN=10`
--panic-on-compile-error
panic on compile error
--format-output
format the output when printing to stdout (only valid when `--output-file` is not provided)
-h, --help
Print help (see a summary with '-h')
-V, --version
Print versionFor more advanced usage, please refer to the documentation and complete example.
This project is licensed under the MIT License - see the LICENSE file for details.