-
Notifications
You must be signed in to change notification settings - Fork 6.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Zephyr Build info file #79118
Zephyr Build info file #79118
Conversation
Any reason not to use a |
Some comments when running $ west build -p always -b native_sim samples/hello_world
# output snippets
kconfig:
files:
- /home/pdgendt/zephyrproject/zephyr/boards/native/native_sim/native_sim_defconfig
- /home/pdgendt/zephyrproject/zephyr/samples/hello_world/prj.conf
user-files:
- /home/pdgendt/zephyrproject/zephyr/samples/hello_world/prj.conf
toolchain:
name: host
path:
zephyr:
version: 3.7.99
|
Only reason was to be somewhat aligned with the default Naming the file |
|
Thanks, appreciated 👍
Correct, and the reason for this is to provide a list of all files which are used for the kconfig, but also a dedicated list of those files which are considered user / application controlled and thus directly under the developers control. One may argue that external tool can concatenate those lists but the tricky part is that internally in CMake all files are being appended to a common new list later at which point we would need to split the lists / rework how they are constructed.
True for the
Doing so will have other implications, so should not be part of this PR. If we want this, then a dedicated issue / PR for this should be opened. Secondly, for the git SHAs we have the build meta file for Zephyr which already generates an output file at build time containing the SHA of Zephyr as well as each Zephyr module, so that feature covers much more than just a git describe of Zephyr itself. So no reason to duplicate information in the build info which we already in a safer way produces at build time. |
82dd714
to
d303d47
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some thoughts:
- Comments refer to
build.info
file but it's actuallyzephyr.build
. - In a sysbuild project (smp_svr) it seems to have the same file in multiple places:
kconfig:
files:
- /tmp/aa/zephyr/samples/subsys/mgmt/mcumgr/smp_svr/build/_sysbuild/empty.conf
- /tmp/aa/zephyr/samples/subsys/mgmt/mcumgr/smp_svr/sysbuild.conf
user-files:
- /tmp/aa/zephyr/samples/subsys/mgmt/mcumgr/smp_svr/sysbuild.conf
and
kconfig:
files:
- /tmp/aa/zephyr/boards/nordic/nrf5340dk/nrf5340dk_nrf5340_cpuapp_defconfig
- /tmp/aa/zephyr/samples/subsys/mgmt/mcumgr/smp_svr/prj.conf
- /tmp/aa/zephyr/samples/subsys/mgmt/mcumgr/smp_svr/build/smp_svr/zephyr/.config.sysbuild
user-files:
- /tmp/aa/zephyr/samples/subsys/mgmt/mcumgr/smp_svr/prj.conf
and
devicetree:
bindings-dirs:
- /tmp/aa/zephyr/dts/bindings
files:
- /tmp/aa/zephyr/boards/nordic/nrf5340dk/nrf5340dk_nrf5340_cpuapp.dts
- /tmp/aa/bootloader/mcuboot/boot/zephyr/app.overlay
include-dirs:
- /tmp/aa/bootloader/mcuboot/boot/zephyr/include
...
user-files:
- /tmp/aa/bootloader/mcuboot/boot/zephyr/app.overlay
- Another oddity:
kconfig:
files:
- /tmp/aa/zephyr/samples/subsys/mgmt/mcumgr/smp_svr/_AA/_sysbuild/empty.conf
- /tmp/aa/zephyr/samples/subsys/mgmt/mcumgr/smp_svr/_AA/_sysbuild/empty.conf
- /tmp/aa/zephyr/samples/subsys/mgmt/mcumgr/smp_svr/sysbuild.conf
user-files:
- /tmp/aa/zephyr/samples/subsys/mgmt/mcumgr/smp_svr/sysbuild.conf
- Would it make sense to also have the original zephyr commit as well as the version, so that it can be tracked too?
- This might include the files that were used but does not mean a build can be reproduced e.g. if a user includes their own command line overrides or if they used FILE_SUFFIX
@nordicjm thanks for the comments.
you mean commit messages ?
See explanation here: #79118 (comment)
yes, this is a sysbuild / kconfig reuse issue. The current This has nothing to do with build info but becomes more visible now.
See reply here #79118 (comment) why git commit or similar should be left out initially.
It's reproducible in the sense that you can see exactly what was used by the build system, but not in the sense that you have an explicit command that will produce exactly the list of files, as example if extra Zephyr modules are available. The purpose of this PR is to provide a solution which shows what was used, for example to be used by an IDE to present configuration files, or a downstream project which needs to use settings for own build invocation, see: #78727 for an example. Trying to provide a solution which provides a single command for reproducible builds is a different task, because a user might manually modify CMakeCache.txt after the first CMake invocation, but you have no way to know what changed and if a given setting can just be appended to original CMake invocation. Please pay attention to the description of the PR:
what is useful with this PR in relation to reproducible builds is the fact that you now have a file with the internal info from a given build which you may compare against a known expectation, and if the file you generate doesn't match expectation then you can compare and see what the difference are, which is much easier than comparing two elf files or two build.ninja files. And combined with build meta feature we already have: then we are one more step closer to really be able to provide real support for reproducible builds. |
3832b26
to
be99434
Compare
Done. |
@tejlmand can you rebase please? |
this is showing some issues that we need to fix...
why are we getting includes for unrelated modules and hals? looks like those are just being added for each build? |
agree, but fixing this is outside the scope of build info. |
right, it seems the dts part is coming from
in some of the hals. |
why do the qualifiers start with '/'? IMO this is wrong and misleading and conflict with what we have in the docs, the "/" is how you connect the various elements, not part of the qualifiers themselves. |
Other toolchains uses <toolchain>_TOOLCHAIN_PATH, align Zephyr SDK by setting ZEPHYR_TOOLCHAIN_PATH to be identical to the ZEPHYR_SDK_INSTALL_DIR. Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
Move Zephyr CMake script mode handling from package_helper.cmake into extensions.cmake. This ensures that all Zephyr CMake script which includes extensions.cmake will have the same functions stubbed or mocked and thus does not need to replicate this behavior. Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
The build_info function provides a generic and stable way of dumping build information to the <build>/build_info.yml file. The build info file is in YAML format and the keys in the file are intended to be stable, as to allow external tools to retrieve information regarding the build. The main differences to the CMakeCache.txt are: - Settings in the CMakeCache.txt are user controlled, whereas the information in the build info file is intended to be those values which are used by the build system regardless if those are specified by the developer or picked up automatically. - Internal build system variables are not present in the CMake cache and should not be, because their values are calculated when CMake runs. This also has the benefits of decoupling CMake variable names from build info keys. Several CMake variables has internal build system names, and the build system is free to rename those at its own discretion. Having dedicated key names ensures a stable API that external tools can rely upon. Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
For pristine builds 'west build' will now create a build_info.yml file containing the west build command including arguments. This is done to help users and external tools to recreate builds. Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
e6c5346
to
41a49d7
Compare
Store informations regarding the current Zephyr build. The following informations are stored during CMake configure: - Board information - Application source directory - Application configuration directory - Toolchain information - Devicetree files - Kconfig config files - Zephyr version Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
Save information regarding SVD file in use in vendor-specific section of the build info file. Information is stored under Nordic section. Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
41a49d7
to
93866cb
Compare
Thanks for catching, fixed 👍 |
Other toolchains uses <toolchain>_TOOLCHAIN_PATH, align Zephyr SDK by setting ZEPHYR_TOOLCHAIN_PATH to be identical to the ZEPHYR_SDK_INSTALL_DIR. Upstream PR: zephyrproject-rtos/zephyr#79118 Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
…o extensions.cmake Move Zephyr CMake script mode handling from package_helper.cmake into extensions.cmake. This ensures that all Zephyr CMake script which includes extensions.cmake will have the same functions stubbed or mocked and thus does not need to replicate this behavior. Upstream PR: zephyrproject-rtos/zephyr#79118 Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
The build_info function provides a generic and stable way of dumping build information to the <build>/build_info.yml file. The build info file is in YAML format and the keys in the file are intended to be stable, as to allow external tools to retrieve information regarding the build. The main differences to the CMakeCache.txt are: - Settings in the CMakeCache.txt are user controlled, whereas the information in the build info file is intended to be those values which are used by the build system regardless if those are specified by the developer or picked up automatically. - Internal build system variables are not present in the CMake cache and should not be, because their values are calculated when CMake runs. This also has the benefits of decoupling CMake variable names from build info keys. Several CMake variables has internal build system names, and the build system is free to rename those at its own discretion. Having dedicated key names ensures a stable API that external tools can rely upon. Upstream PR: zephyrproject-rtos/zephyr#79118 Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
For pristine builds 'west build' will now create a build_info.yml file containing the west build command including arguments. This is done to help users and external tools to recreate builds. Upstream PR: zephyrproject-rtos/zephyr#79118 Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
Store informations regarding the current Zephyr build. The following informations are stored during CMake configure: - Board information - Application source directory - Application configuration directory - Toolchain information - Devicetree files - Kconfig config files - Zephyr version Upstream PR: zephyrproject-rtos/zephyr#79118 Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
…le used. Save information regarding SVD file in use in vendor-specific section of the build info file. Information is stored under Nordic section. Upstream PR: zephyrproject-rtos/zephyr#79118 Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
@@ -114,3 +114,6 @@ set_ifndef(TOOLCHAIN_KCONFIG_DIR ${TOOLCHAIN_ROOT}/cmake/toolchain/${ZEPHYR_TOOL | |||
|
|||
set(HostTools_FOUND TRUE) | |||
set(HOSTTOOLS_FOUND TRUE) | |||
build_info(toolchain name VALUE ${ZEPHYR_TOOLCHAIN_VARIANT}) | |||
string(TOUPPER ${ZEPHYR_TOOLCHAIN_VARIANT} zephyr_toolchain_variant_upper) | |||
build_info(toolchain path VALUE "${${zephyr_toolchain_variant_upper}_TOOLCHAIN_PATH}") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Causing old compilation errors as ???_TOOLCHAIN_PATH is invalid.
-- Found toolchain: gnuarmemb (C:\gnu_arm_embedded_10)
CMake Error at C:/Users/BillyBob/zephyrGoodX/zephyrNew/cmake/modules/yaml.cmake:313 (string):
string sub-command JSON failed parsing json string: * Line 1, Column 1Bad escape sequence in string
See Line 1, Column 6 for detail.
I changed it to:
build_info(toolchain path VALUE "${TOOLCHAIN_ROOT}")
# build_info(toolchain path VALUE "${${zephyr_toolchain_variant_upper}_TOOLCHAIN_PATH}")
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey @WilliamGFish, could you open a pull request with the fix? This is the contribution guideline documentation. Thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, found out its a Linux / Windows path issue.
I'll raise a 'fix' via a PR.
This PR introduces a common build info file for Zephyr.
Some tools peeks into Zephyr's CMakeCache.txt file and relies on what is considered internal build variables.
This is bad for several reasons, especially as it suddenly puts unknown expectation on the Zephyr build system.
And it makes it harder to cleanup Zephyr's build system when we are doing clean ups in the build code.
Besides the lack of visibility, such as which variables are used externally / downstream, then it's also hard for developers that are integrating to Zephyr to know which settings they can expect to be constant.
Some settings are documented as input to Zephyr build system, like EXTRA_CONF_FILE, EXTRA_DTS_OVERLAY_FILE, etc. but are used together with internal logic on conf file selection, making it hard to extract knowledge on configuration file.
#78727 also present a use-case where internal build knowledge must be exported to Python
To support various use-cases, then this PR proposes a common interface which allows to store build values in a common yaml file.
The yaml file is proposed to be named and placed in
<build-dir>/build_info.yml
.A yaml schema is provided in https://github.com/zephyrproject-rtos/zephyr/blob/82dd714dc8dbcf0b1fa138cd521117ec49e3a2be/scripts/schemas/build-schema.yml.
This schema acts as the contract regarding information from the build system.
For example, the schema contains:
thus promising to place kconfig files in use under this key.
The build system are the free to have this information in an internal variable and rename this variable as it find fit.
To facilitate populating the build info file, then a new CMake function is created:
build_info(<key> VALUE <value>)
Example:
this allows the code to call
build_info()
close to the final usage of the internal CMake variable.Furthermore, having a
build_info()
function makes it easy to find internal settings that are exported, and any CMake variables expanded in the<value>
field can easily be renamed if build system internals are changed.If
build_info()
is called with an invalid key, likebuild_info(does not exists VALUE ok)
, then an error is thrown.This both helps to ensure that only keys described in the schema are allowed as well as catching typos.
It also ensures that in the event of a key is renamed / removed then usage of said key in
build_info()
will raise an error.To both minimize the need for Zephyr modules to require n-entries in the schema as well as supporting vendor specific information, then a free form
vendor-specific
section is provided. This section is considered under the responsibilities of the vendors using it.zephyr/scripts/schemas/build-schema.yml
Lines 92 to 99 in 82dd714