Can I recompute fs.hash() when the file changes?

[detly]

I have a project where I generate an archive of files for deployment, based in part on some templates (using gomplate). I do this with a custom target that runs the template processor, which takes variables via the environment.

My problem is that one of these variables is a file's SHA256 hash computed via fs.hash(), and meson does not update this variable when the file changes. Here's a simple example project:

project(
    'meson-test',
    meson_version: '>=1.3.2',
)

fs = import('fs')

magic = files('magic')
magic_hash = fs.hash(magic, 'sha256')

message(f'Hash: @magic_hash@')

target_env = {
    'HASH': magic_hash,
}

forcing_target = custom_target(
    build_by_default : true,
    depend_files : [magic],
    output : 'more-magic',
    command : ['sh', '-c', 'echo $HASH'],
    env : target_env
)

Create the magic file with eg. echo 'magic' > magic. Check the hash:

# sha256sum magic
3030ce6423e63c188f5d6d742c70ba4730096865f4f166838ffa0e3a8eec4117  magic

The first time this is set up and built, you get the hash:

# meson setup build
The Meson build system
Version: 1.3.2
Source dir: /root/meson-test
Build dir: /root/meson-test/build
Build type: native build
Project name: meson-test
Project version: undefined
Host machine cpu family: x86_64
Host machine cpu: x86_64
Message: Hash: 3030ce6423e63c188f5d6d742c70ba4730096865f4f166838ffa0e3a8eec4117
Build targets in project: 1

Found ninja-1.11.1 at /usr/bin/ninja
# ninja -C build
ninja: Entering directory `build'
[1/1] Generating more-magic with a custom command (wrapped by meson to set env)
3030ce6423e63c188f5d6d742c70ba4730096865f4f166838ffa0e3a8eec4117

But if the file is updated:

# echo 'more magic' > magic
# sha256sum magic
49309ee6a9430f0ec70ffe44e57b62194afa69bd1773cbe87c68cc53ef949d76  magic
# ninja -C build
ninja: Entering directory `build'
[1/1] Generating more-magic with a custom command (wrapped by meson to set env)
3030ce6423e63c188f5d6d742c70ba4730096865f4f166838ffa0e3a8eec4117

I think I understand what the problem is: meson's variables are computed at setup/configuration time. At build time, the configuration variables do not get recomputed except when meson.build itself changes. The targets are what get rebuilt, and may trigger other targets to get rebuilt, but not the configuration that expresses these dependencies.

However, even knowing that, I'm at a loss as to how to work around this. I don't think there's a way to "promote" a variable to a target or source, is there? Or to defer something like fs.hash() to build time? How should I approach this?

[eli-schwartz]

I think I understand what the problem is: meson's variables are computed at setup/configuration time. At build time, the configuration variables do not get recomputed except when meson.build itself changes. The targets are what get rebuilt, and may trigger other targets to get rebuilt, but not the configuration that expresses these dependencies.

Yes.

However, even knowing that, I'm at a loss as to how to work around this. I don't think there's a way to "promote" a variable to a target or source, is there? Or to defer something like fs.hash() to build time? How should I approach this?

One approach would be to run the sha256sum tool in your custom_target command.

Another approach is to do fs.read(magic) and discard the results. This will cause meson to add the "magic" file as a regen dependency -- which means that it's equivalent to meson.build itself changing. This is kind of a big stick to use since reconfigure can be relatively slow...

[detly]

One approach would be to run the sha256sum tool in your custom_target command.

I am leaning towards this. Is there a way to take the output and stuff it directly into a custom_target()'s environment? Or would I need to write it to an intermediate file first and... read it with fs.read() anyway?

[eli-schwartz]

If you use e.g. a python or sh script for the custom_target you can simply read the sha256 checksum in the same process that then sets an environment variable and forks another program that actually produces the output file.

[detly]

Okay, thanks for the info and ideas, it was genuinely helpful 👍 I am going to go and agonise over the weird economies of software.

Purely out of curiosity, what is fs.hash() for? I had assumed it was to trigger reconfiguration if something changed in a way Meson couldn't detect, but in retrospect that doesn't make much sense.

[eli-schwartz]

I am not exactly sure what fs.hash() is for. @scivision added it in #6150 but did not list the motivating reason.

[scivision]

Hash() was intended as a one-time configure time (or manual reconfigure) check. I used it on multi-gigabyte files so considering spinning HDD I didn't want to trigger meson reconfigure off a hash change. hash() was simple to implement internal to Meson as it's built into Python.