add (experimental) support for general options in Easystack

[deniskristak]

related to: easystack file pull request

Additional command-line arguments to EasyBuild can be passed at any level (may even be included outside of software scope by specifying a keyword that matches the long argument name, e.g.

robot: True
software:
  GROMACS:
    parallel: 6
    toolchains:
      foss-2020a:
        versions:
          2020.1:
          2020.3:
            from_pr: 1234

Will be equivalent to running

eb GROMAS-2020.1-foss-2020a.eb --robot --parallel=6
eb GROMAS-2020.3-foss-2020a.eb --robot --parallel=6 --from-pr=1234

The deepest nested value will always take priority, i.e.

software:
  GROMACS:
    parallel: 6
    toolchains:
      foss-2020a:
        versions:
          2020.1:
          2020.3:
            parallel: 12

will be equivalent to running

eb GROMAS-2020.1-foss-2020a.eb --parallel=6
eb GROMAS-2020.3-foss-2020a.eb --parallel=12

[boegel]

robot: True
software:
  GROMACS:
    parallel: 6
    toolchains:
      foss-2020a:
        versions:
          2020.1:
          2020.3:
            from_pr: 1234
            include_easyblocks_from_pr: 123
            debug: True
            trace: True

# current approach:
# eb example.yaml
eb GROMACS-2020.1-foss-2020a.eb GROMACS-2020.3-foss-2020a.eb --robot --parallel 6
# 1) eb /path/to/GROMACS-2020.1-foss-2020a.eb --parallel 6
# 2) eb /path/to/GROMACS-2020.3-foss-2020a.eb --parallel 6 --from-pr 1234 --include-easyblocks-from-pr 123 --debug --trace

High-level overview of things to change to support using custom configuration options for specific installations

• [REQ] parse_easystack returns list of (easyconfig filename, cfg options for this easyconfig file)
• det_copy_ec_specs and categorize_files_by_type have to be aware of cfg options per easyconfig file
• [REQ for --from-pr] det_easyconfig_paths has to take into account things like from-pr cfg option per easyconfig file
• determined_paths -> paths
• [REQ] parse_easyconfigs has to store cfg options per easyconfig file in EasyConfig instance
• [REQ for --robot] resolve_dependencies has to pass down cfg options to dependencies
  • this would result in different behavior compared to eb command (cfr. --hidden --robot)
• [REQ] build_and_install_software has to "activate" cfg options per installation
  • orig_build_options = BuildOptions()
  • replace_singleton(BuildOptions, ec2opts[name_of_easyconfig_file]) (del Singleton._instances[BuildOptions])
  • after every install: restore original build options

[boegel]

Support for updating build options on the fly was added in #3684, which should help a bit with this...

[casparvl]

I had a look at this and am trying to understand what you mean by e.g.

[REQ for --from-pr] det_easyconfig_paths has to take into account things like from-pr cfg option per easyconfig file

I guess this is about the fact that this line normally reads the --from-pr from the global options. In the case of an EasyStack file, where we might have options per EasyConfig, this is not enough. Would your idea be to simply add an additional (optional) argument to det_easyconfigs_paths that conatins the from_pr that is specified per EasyConfig? And then change the behaviour of det_easyconfigs_paths so that if that argument is defined, it determines --from-pr at the per-easyconfig level?

Another question that came to my mind: does a --from-pr specified for a specific EasyConfig come instead of, or in addition to globally defined flags? I.e. what if I have an EasyStack my_easystack,py file that specifies --from-pr=101 for a certain EasyConfig, and I call with eb my_easystack.py --from-pr=102: should the from_prs vector on this line then simply be two elements long? (for that particular easyconfig, of course it would be only 102 for the others)

[casparvl]

@boegel and I had a discussion on how to proceed, and what to think about. The notes are included below:

robot: True
software:
  GROMACS:
    parallel: 6
    toolchains:
      foss-2020a:
        versions:
          2020.1:
          2020.3:
            from_pr: 1234
            include_easyblocks_from_pr: 123
            debug: True
            trace: True

EasyBuild 4.5.0, parse_easystack returns:

res = (['GROMACS-foss-2020a-2020.1.eb', 'GROMACS-foss-2020a-2020.3.eb'], {})

What we would like parse_easystack to return in order to support EasyConfig-specific options is:

orig_paths = [...]
opts_per_ec = {
    'GROMACS-foss-2020a-2020.1': {
        'parallel': 6,
        'robot': True,
    },
    'GROMACS-foss-2020a-2020.3': {
        'debug': True,
        'trace': True,
        'from_pr': 1234,
        'include_easyblocks_from_pr': 123,
        'parallel': 6,
        'robot': True,
    },
}

Some functions that are called based on the results that might be affected:

orig_paths, opts_per_ec = parse_easystack

det_ec_copy_specs(orig_paths, from_pr_list, opts_per_ec={})

det_easyconfig_paths(categorized_paths['easyconfigs'], opts_per_ec={})

Once we have the dictionary of EasyConfig options as a result of parse_easystack, we could use it like this:

opts_per_ec.get(key, {})

Actions

• [DETAIL] Updating det_copy_ec_specs: only relevant for --copy-ec => create new issue
• categorize_files_by_type: no need to be aware of options => no-op
• [ESSENTIAL] det_easyconfig_paths:
  • has to be made aware of opts_per_ec for from-pr per ec
• [DETAIL] How to handle conflicting command line args & EasyStack options? => don't solve now, make ticket. For now, implement so that EasyStack always wins.
• per-ec options need to be installed:
  • [ESSENTIAL] Add feature to update_build_option(key, value) function to return the original key/value (so that we can restore later).
  • [ESSENTIAL] also implement update_build_options(dict) utility function to pass opts_by_ec[ec_filename] dict + return original values (in a dict, aggregated from the returns of update_build_option)
  • [ESSENTIAL] when parsing easyconfigs (see parse_easyconfigs in framework/easyconfig/tools.py)
    • Add update of build options at

      easybuild-framework/easybuild/framework/easyconfig/tools.py

      Line 387 in f1f421b

      path = os.path.abspath(path)

    • Do the parse + store custom options in EasyConfig instance (self.custom_build_opts)
    • Restore original build options with another call to update_build_options
  • [DETAIL] when resolving dependencies (in resolve_dependencies, right before robot_find_easyconfig is called in while loop,

    easybuild-framework/easybuild/tools/robot.py

    Line 403 in f1f421b

    # find easyconfig, might not find any

    )
    • only relevant to pick up custom robot_paths for a specific easyconfig
  • [ESSENTIAL] when installing easyconfigs (see build_and_install_software in main.py)
    • Update build opts (this time using ec.custom_build_opts) at

      easybuild-framework/easybuild/main.py

      Line 125 in f1f421b
    • Restore original build options at the end of the iteration

Note that we should probably split this over multiple PRs:

• changes to update_build_option and update_build_options could be one
• updates to det_easyconfig_paths could be another, as this would implement support for --from-pr
• parse_easyconfigs and build_and_install_software could be one, or maybe two PRs (not sure if just having support in parse_easyconfigs` already gives something)

special cases

• --from-pr: needs to be taken into more places (det_easyconfig_paths)
• --include-easyblocks-from-pr: included easyblocks are set up very early (directly after option parsing)
  • update_build_option needs to re-do setup of included easyblocks (from PR)
• robot-paths specified for specific easyconfig

issues

• semantics of --copy-ec + --easystack
• combining of configuration options from env/eb-arg + easystack file => which wins?
• semantics of using --try-* options in easystack file (see tweak function)

[boegel]

This is basically done in #4057, so closing...