Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use parallelization #129

Open
LecrisUT opened this issue Nov 24, 2023 · 7 comments
Open

Use parallelization #129

LecrisUT opened this issue Nov 24, 2023 · 7 comments

Comments

@LecrisUT
Copy link

LecrisUT commented Nov 24, 2023

The action should set the following environment variables, especially when presets are used:

CMAKE_BUILD_PARALLEL_LEVEL: $(nproc)
CTEST_PARALLEL_LEVEL: $(nproc)

(effectively running cmake --build -j $(nproc) and ctest -j $(nproc))

These could be controlled by:

parallel: auto | none
build_parallel: ${parallel:-auto | none}
test_parallel: ${parallel:-auto | none}
@lukka
Copy link
Owner

lukka commented Nov 24, 2023

@LecrisUT did you consider using jobs property in build and test presets of CMakePresets.json?

@LecrisUT
Copy link
Author

LecrisUT commented Nov 24, 2023

Yes, but there is no macro expansion from cmake to get the $(nproc) dynamically. Also jobs value takes precedence so it should be safe to set is as a default

@lukka
Copy link
Owner

lukka commented Nov 29, 2023

@LecrisUT The cmake invocation could be customized to satisfy any additional needs with the usage of the *AdditionalArgs and *CmdString inputs.

If those inputs are not enough, contribution to this project are welcome.

@LecrisUT
Copy link
Author

Yes, but $(nproc) is a bit tricky because it is not a static variable: https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#choosing-github-hosted-runners

But this should be a simple enough addition I can write it up

@lukka
Copy link
Owner

lukka commented Dec 21, 2023

@LecrisUT I am trying to understand which problem we are trying to solve. When defining the value of CMAKE_BUILD_PARALLEL_LEVEL is going to be useful? I'd expect that delegating the choice to the current build tool is always the best choice because it should be able to utilize all the cpu available cycles, by default. For example, when using cmake and ninja, there is no need to specify any special additional value to ensure the build is as fast as possible on the system it is running on.

Could you show some workflows on GH which gives good use of such variable?

@LecrisUT
Copy link
Author

For CMAKE_BUILD_PARALLEL_LEVEL it is more confusing than the test one. If we look at the -j documentation, there are a few caveats:

  • it is generator dependent, so it depends on the generator used in the preset. Realistically there are ninja and the OS defaults Unix makefile, and vscode, xcode? I have encountered situations where the CMake was generator dependent so probably the user's CI can be extensive.
  • which generators always build in maximum parallel by default?
  • which generators need the -j without value to specify to build in maximum parallel
  • which generators need -j $(nproc) explicitly? For those that have defaults, how is it calculated?
  • the most crucial point: using presets, there doesn't seem to be an option to pass -j without value

Setting CMAKE_BUILD_PARALLEL_LEVEL seems the most general way of solving this, even though it might not be as efficient as the automatic.

@LecrisUT
Copy link
Author

LecrisUT commented Dec 21, 2023

About the preset options, these options are available, but:

  • the accepted value is integer
  • cannot specify a null in order to pass a -j without value
  • cannot use macro expansion, and there are no macros for $(nproc)
  • cannot use arithmetics. This is useful for oversubsscribing the build tasks
  • generally the philosophy regarding presets that they want to implement is to keep it as simple as possible, especially if the CI infrastructure can cover it. It's a coin flip for this feature I would say, and I'm certain the arithmetic one won't go through
  • this feature if implemented, would appear in future CMake versions, but it is often needed to support older CMake versions as well, at least the latest OS LTS in order to make the project packageable there. The environment approach would cover all available versions

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants