Some options in a mode do not expand environment variables past in

[tteggelit]

Given the following mode:

        {
            'name': 'jenkins-prebuilt',
            'options': [
                '--run',
                '--save-log-files',
                '--performance-report',
                '--purge-env',
                '--prefix ${AT_OUTPUT_DIR}',
                '--stage ${AT_OUTPUT_DIR}/stage/${JOB_BASE_NAME}/${BUILD_NUMBER}',
                '--output ${AT_OUTPUT_DIR}/output/${JOB_BASE_NAME}/${BUILD_NUMBER}',
                '--perflogdir ${AT_OUTPUT_DIR}/perflogs',
                '--report-file ${AT_OUTPUT_DIR}/report/report_{sessionid}.json',
                '--job-option v=JENKINS_BUILD=${JOB_BASE_NAME}-${BUILD_NUMBER}',
                '--setvar project=${AT_ACCOUNT}',
                '--setvar use_prebuilt=True',
                '--setvar queue=${QUEUE}',
                '--setvar num_nodes=${NODES}',
            ]
        },

I would expect the environment variables for each option to expand to what they're set to. For some (--stage, --prefix, --output, --perflogdir, and --report-file) they do expand, but some don't expand (--job-option, --setvar).

If environment variables are set prior to executing reframe:

export AT_OUTPUT_DIR=/flare/Aurora_AT/leggett/reframe/data
export NODES=1
...

Some are obviously set because paths are correct:

$ reframe -v -C /flare/Aurora_AT/leggett/alcf_reframe_testsuite/config/alcf.py --mode jenkins-prebuilt -c /flare/Aurora_AT/leggett/alcf_reframe_testsuite/tests/sow/FFT/fft_sow.py
[ReFrame Setup]
  version:           4.5.1
  command:           '/opt/aurora/24.180.1/spack/unified/0.8.0/install/linux-sles15-x86_64/gcc-12.2.0/reframe-4.5.1-exvucbt/bin/reframe -v -C /flare/Aurora_AT/leggett/alcf_reframe_testsuite/config/alcf.py --mode jenkins-prebuilt -c /flare/Aurora_AT/leggett/alcf_reframe_testsuite/tests/sow/FFT/fft_sow.py'
  launched by:       auroraat@aurora-uan-0014
  working directory: '/lus/flare/projects/Aurora_AT/leggett/reframe/data/tmp/sow/FFT/1/13'
  settings files:    '<builtin>', '/flare/Aurora_AT/leggett/alcf_reframe_testsuite/config/alcf.py'
  check search path: (R) '/lus/flare/projects/Aurora_AT/leggett/alcf_reframe_testsuite/tests/sow/FFT/fft_sow.py'
  stage directory:   '/flare/Aurora_AT/leggett/reframe/data/stage/FFT_TEST/13'
  output directory:  '/flare/Aurora_AT/leggett/reframe/data/output/FFT_TEST/13'
  log files:         '/lus/flare/projects/Aurora_AT/leggett/reframe/data/tmp/sow/FFT/1/13/reframe.log', '/lus/flare/projects/Aurora_AT/leggett/reframe/data/tmp/sow/FFT/1/13/reframe.out'

But the test is erroring out because ${NODES} doesn't seem to resolve but is instead being passed as literal text:

WARNING: skipping test 'FFTTest': value error: invalid literal for int() with base 10: '${NODES}' (rerun with '-v' for more information)

[vkarak]

That's indeed inconsistent and we should treat more uniformly the environment variable expansion.