Representing ecflow .def files in yaml

[benjamin-cash]

I thought it might be easier to discuss this via github than slack. I'm interested in how one would represent ecf/defs/aqm_cycled.def as a yaml file.

One option looks something like this:

suite_nco_aqm:
  limit:
    support: 10
    model: 10
  edit:
    SLEEP: 10
    FRED: 'bill'
  family_primary:
    edit:
      aqm_ver: 'v7.0.1'
      PACKAGEHOME: '/lfs/h2/emc/physics/noscrub/%EMC_USER%/nwdev/packages/aqm.%aqm_ver%'
      NET: 'aqm'
      RUN: 'aqm'
      PROJ: 'AQM'
      PROJENVIR: 'DEV'
      MACHINE_SITE: 'development'
      ENVIR: 'prod'
      QUEUE: 'devhigh'
      QUEUE_ARCH: 'dev_transfer'
      OUTPUTDIR: '/lfs/h2/emc/ptmp/%EMC_USER%/ecflow_aqm/para/output/prod/today'
    family_00:
      edit:
        CYC: '00'
      task_cycle_end:
        inlimit:
          - support
        edit:
          ECF_FILES: '%PACKAGEHOME%/ecf'
        cron: '23:00'

This is taking advantage of the fact that the ecflow api will, for example, accept a dictionary for Edit(), and thus you can add everything in the edit: block in a single command. Not everything can be done that way though, so you get some inconsistencies in representation. Another option would look more like:

suite_nco_aqm:
  limit_support: 10
  limit_model: 10
  edit_SLEEP: 10
  edit_FRED: 'bill'
  family_primary:
    edit_aqm_ver: 'v7.0.1'
    edit_PACKAGEHOME: '/lfs/h2/emc/physics/noscrub/%EMC_USER%/nwdev/packages/aqm.%aqm_ver%'
    edit_NET: 'aqm'
    edit_RUN: 'aqm'
    edit_PROJ: 'AQM'
    edit_PROJENVIR: 'DEV'
    edit_MACHINE_SITE: 'development'
    edit_ENVIR: 'prod'
    edit_QUEUE: 'devhigh'
    edit_QUEUE_ARCH: 'dev_transfer'
    edit_OUTPUTDIR: '/lfs/h2/emc/ptmp/%EMC_USER%/ecflow_aqm/para/output/prod/today'
    ...

This would use the same _name construction everywhere.

[benjamin-cash]

@christinaholtNOAA - Any thoughts on these two directions?

[maddenp-noaa]

Hi @benjamin-cash -- Sorry for the delay, the team has been consumed with other work and I, at least, have not had time to look into this.

First thing, though: The "ecf/defs/aqm_cycled.def" link you gave above doesn't work. Could you edit it? I'd be interested in looking at the source configuration file for comparison to what you show here.

Without knowing more, my immediate impression is that the first example makes better use of YAML's hierarchical nature, which I think is preferable to encoding structure in the key names as in the second example. I'd be wary of case sensitivity in key names, though in cases where there's an easy one-to-one mapping between UW YAML keys and component or workflow-engine keys it might be worth accepting this. Currently, however, UW YAML only uses lower-case, snake_case keys, and we map to component-required names where necessary. So, we might consider that approach.

We might also want to consider if/how we could re-use information from the platform: block to avoid duplicating information that ecFlow might need. It could make sense to extend platform: to cover more use cases. We can of course use Jinja2 to propagate a value from one block to another to keep the configuration DRY, but platform information is such a central concept that factoring it out might make sense. And in some cases probably not: For example, one would probably like to be able to submit different parts of a workflow to different queues (serial vs parallel queues, for example), so a single queue value probably doesn't make sense.

I see a trade-off between designing a concise, DRY config language in UW YAML that can then be expanded by the tool into the full configuration required by the component -- hard, but potentially rewarding in terms of ease-of-use; and basically mirroring the component configuration in UW YAML and doing less translation/expansion -- easier, but not necessarily very helpful to users, as they have to do much of what they would have done in the component configuration directly, except in YAML now (for better or worse -- maybe better, but there's also a mediating tool in play that has to strive to be bug-free).

We have a series of tasks in our backlog to implement an ecFlow tool similar to the Rocoto tool we already have. Anything we figure out here could be captured for use in that work.

[benjamin-cash]

@maddenp-noaa - Getting a bit to the motivation for this question, I'm very interested in moving to ecflow for my own work and being able to clearly describe my workflow suites via yaml. I know that's a longer term goal of the uw team, and I would definitely like whatever I do be a step in that direction. Otherwise it is just another kludge that later gets discarded.

[benjamin-cash]

Hi @maddenp-noaa - Thanks for chiming in, and no worries on the slow reply. I know (or at least believe) this is not one of the tasks the team is currently working, so it's a bit on the back burner.

Not sure why the link got mangled, here it is again (hopefully): https://github.com/ufs-community/ufs-srweather-app/blob/production/AQM.v7/ecf/defs/aqm_cycled.def

[maddenp-noaa]

Was that .def file generated from ecFlow's Python API? There seems to be a lot of repetition that I'd hope would flow out of loop-like constructs. If so, I'd wonder what the API calls look like and how we'd capture the arguments passed to them in the most concise way possible. Is that what you had in mind with your examples, or was that potentially just a small piece of a much larger config (the linked-to one is 959 lines)?

[maddenp-noaa]

This class handles that. Its constructor receives a UW YAML that's validated against a corresponding JSON Schema, then starts building an in-memory XML tree starting with the workflow: key's value and calling (sometimes recursively) functions to map YAML (now parsed to Python dict) sub-trees onto Rocoto XML, creating new XML nodes and setting their attributes and text fields as needed.

how would I describe generating a looped set of tasks like in the snippet above?
Which snippet exactly? The Jinja2 one? If so, I'd start with something like:

• An ecFlow def is defined as a suite, so call the API to create a suite. You could encode information in the key name, but I'd recommend considering (despite what we've done for Rocoto) using a dedicated UW YAML key name like suite, and having a name: key beneath it to provide the specific name. This way, names are easily schema-checkable without using regular expressions.
• Apparently a suite may or must include an "edit", so locate that key, and pass its value (the dict under edit:) to e.g. and add_edit() function, along with the suit object, and let that function take care of adding to the suite whatever it needs to based on the block, by making API calls. Then the function returns to the caller which steps forward to:
• A suite may or must include a limit, so pass the suite object along with the limit: block to `add_limit() and let it do what it knows how to do. Return.
• It looks like a bunch of "family" objects get added to the suite object, maybe via a double loop or a cartesian product of "ldate" and "mdate" values. Pass appropriate values to add_family() and let it add those to the suite.
• Etc.

This should end up consuming all of the YAML input to produce a corresponding suite object. Or so I'd hope.

[benjamin-cash]

Hi @maddenp-noaa - Sorry, I didn't phrase that question very well. I agree 100% with what you've written here, and have even written some bad code that implements a lot of what you are describing. One thing I'm not sure about is how you would describe those nested jinja2 loops in the yaml config file, and I was curious if you had an example of how you did that for rocoto.

I think that for some elements of the suite you will have to encode information in the name, because you can have multiple tasks or families at the same level, unless maybe tags can be used to declare that something is a Family (for example) and treated as such? It looks like CROW used something like that here? https://github.com/NOAA-EMC/CROW/blob/2079a681188ea49ad73f548de093216c828e8750/crow/config/from_yaml.py

[maddenp-noaa]

There are some places like here and here where some arbitrary number of child items (children of a metatask in the first case, which may be tasks, other metatasks, or variables; and task dependencies in the second case) are looped over and the XML tree built out based on those. But they child items are explicitly enumerated in the UW YAML config, which I think is a different use case.

I'm not sure exactly what LDATE and MEM mean here, but if LDATE is cycle information, maybe it could be expressed in YAML as a start/step/num triplet (or start/stop/step, or similar), from which any set of (consistently-spaced) cycles could be generated; and if MEM is something about ensemble members, I support that could just be an integer. So, in Python, maybe something like

>>> from itertools import product
>>> from datetime import datetime, timedelta
>>> 
>>> start, step, num = datetime(2024, 4, 30, 0), 6, 8
>>> members = 4
>>> ldates = [start + timedelta(hours=step * i) for i in range(num)]
>>> mems = range(members)
>>> 
>>> [f"item_{ldate.strftime('%Y%m%d%H')}_{mem}" for ldate, mem in product(ldates, mems)]
['item_2024043000_0', 'item_2024043000_1', 'item_2024043000_2', 'item_2024043000_3', 'item_2024043006_0', 'item_2024043006_1', 'item_2024043006_2', 'item_2024043006_3', 'item_2024043012_0', 'item_2024043012_1', 'item_2024043012_2', 'item_2024043012_3', 'item_2024043018_0', 'item_2024043018_1', 'item_2024043018_2', 'item_2024043018_3', 'item_2024050100_0', 'item_2024050100_1', 'item_2024050100_2', 'item_2024050100_3', 'item_2024050106_0', 'item_2024050106_1', 'item_2024050106_2', 'item_2024050106_3', 'item_2024050112_0', 'item_2024050112_1', 'item_2024050112_2', 'item_2024050112_3', 'item_2024050118_0', 'item_2024050118_1', 'item_2024050118_2', 'item_2024050118_3']

That's just a made-up representation of some cycle-based items, with 4 members for each of 8 6-hourly cycles. Something like those values (but probably just the datetime and int objects) could be fed to a function that added whatever these represent to a suite, or family, or whatever.

Sorry if that's obvious; I'm not sure it's helpful or addressed the question.

I think that for some elements of the suite you will have to encode information in the name, because you can have multiple tasks or families at the same level

Instead of e.g.

families:
  family_a:
    x: 1
    y: 2
  family_b:
    x: 11
    y: 22
  etc.

You could also have

families:
  - name: a
    x: 1
    y: 2
  - name: b
    x: 11
    y: 22
  - etc.

i.e. use a YAML sequence (Python list of dicts here) instead of a YAML mapping (just Python dict) for these. It would be more difficult to refer to a specific element by name if that were a requirement, but the list could be searched for the desired name: value, which would presumably be unique. But if it's just a matter of representing all the required information and looping over it to set up a workflow, a list of items might be fine and doesn't cause a key conflict. IIRC there was some problem with representing the Rocoto workflows that precluded this, but I'm not sure at the moment. You may also have a deal-breaker use case. But if you can avoid encoding information in keys and keep it in values, some things will be easier.

[benjamin-cash]

I think the problem with these approaches is that you cannot guarantee that everything is contained within families, or that all tasks within a family can be grouped together in a single block. ecflow, so far as I understand, acts on the contents of the .def file in sequence. So I think it's possible that you could need to have something like:

families:
  - name: a
     x: 1
     y: 2
  - name: b
     x: 11
task:
  - name: c
     x: 33
families:
   - name: d
   etc 

If nothing else, there is nothing stopping you (as far as I can tell) in ecflow from having a setup like that, where you have multiple families or tasks at the same level interrupted by something else. Presumably that is also true in rocoto, which is why the task_ construction was required.

[benjamin-cash]

@christinaholtNOAA - This is where I think it would be really useful to see more examples of how ecflow is used with UFS, particularly by NCO. In looking at the aqm_cycled.def example I don't see any cases where a family is specified, then something else, and then a family again, but I don't know if that is always going to be true.

[benjamin-cash]

The reason for the case sensitivity in key names like PACKAGEHOME is because of the way ecflow.Edit is set up. It will accept a dictionary, setting the variable names to the keys and the values to the values. So if you are representing a workflow that has case sensitive variable names you either need case sensitive keys or a translation scheme. In the aqm .def file it looks like suite/family/task names are all lower case snake, and variable names are upper case snake, but I don't know how universal that is.

[maddenp-noaa]

We have assumed that there will be cases where we'll need mixed- or upper-case keys: shell environment variables for example. I think this is an example where we wouldn't consider the environment-variable names "UW YAML" per se. We might have a block like

environment_variables:
  FOO: 1
  BAR: 2

Strictly speaking, we could also express this like:

environment_variables:
  - name: FOO
    value: 1
  - name: BAR
    value: 2

but I'm not sure it's worth the effort. Not controlling the key space might weaken our ability to do schema checking and early error detection, but the handful of places where we might want to permit arbitrary key names may be small enough to never cause issues. So, I think we can accept this, but probably want to nest any free-form names under a special key, beneath which the schema will be permissive. Otherwise, if we have to say "any key at all is fine here", it becomes difficult to distinguish typos and give users helpful advice.

[benjamin-cash]

There are some places like here and here where some arbitrary number of child items (children of a metatask in the first case, which may be tasks, other metatasks, or variables; and task dependencies in the second case) are looped over and the XML tree built out based on those. But they child items are explicitly enumerated in the UW YAML config, which I think is a different use case.

It sounds then like how to specify this kind of loop in the UW YAML config files has not been defined, is that correct?

[benjamin-cash]

In my experience it depends - for example, the way the Frontera system is set up it doesn't work particularly well with cylc and some of the functionality to restart and retry is non-functional and you end up manually picking up failed cases.

But it's also easy to imagine a situation where I want to run a specific set of non-contiguous dates on purpose and not due to an error. All El Nino and La Nina years, for example. So in addition to start/stop/step it might be good to allow for a list of some kind.

[maddenp-noaa]

Running an arbitrary set of cycles is a perfectly reasonable requirement. A UW YAML min-language for ecFlow might accept either a start/stop/step kind of definition, or an arbitrary-list-of-cycles definition. A schema can require that one of these be provided, but accept either.

[benjamin-cash]

How would you capture in the yaml that an entry is a function of the loop variable? There are a couple of different examples of this in https://github.com/ufs-community/ufs-srweather-app/blob/production/AQM.v7/ecf/defs/aqm_cycled.def. In lines 624 to 842, there is a nice regular increase in what could be thought of us a loop variable (f000, f001, ...) and that same variable is used as a task variable (edit FHR '000' ...). But just above that is a case where the loop variable is used in a more complex way (538 to 621) where the format of the entry depends on the value and the entries are also functions of that value.

[maddenp-noaa]

Until I have time to study the ecFlow definition language and understand what I'm seeing, I think I'm getting to the end of what I can usefully say about this. I would guess/hope that identifiers (e.g. jaqm_post_f000, 000_rdy) are arbitrary and only contain would-be loop-variable information because someone decided to name them that way, because it was a natural choice and maybe easy for humans to understand. If the numerical information embedded in the identifiers is interpreted in some way, I feel like that would be a disappointing design choice, though maybe I'll feel differently when I study ecFlow.

Without understanding their concepts of task, trigger, event, etc., I vaguely assume that what's being represented is the dependencies between and readiness criteria for tasks. I'd hope that some of this could be deduced from a more compact YAML representation (I'm avoiding your central question: How?) that could be expanded into this massive chunk of definitions, full of apparent repetition despite e.g. some differences between cycles. For this particular case, it seems like most of the config could "compress" into a small amount of YAML. But, then, is that even close to a general mini-language for configuring ecFlow? Probably not. The question is how, if it's even possible, to capture enough of ("all of" may be out of reach) the expressibility of ecFlow in a more compact YAML form to address enough potential UFS use cases to be worth it.

I mean, looking at that aqm_cycled.def, I guess you could manually edit it to, say, not process leadtime 7 for the 00Z cycle, and to not process leadtime 11 for the 12Z cycle. You could tweak that def to say that, but figuring out how to capture that requirement in a "compressed" YAML representation might be difficult. You might add an optional skip_leadtimes YAML key for that. But then you might have to keep adding extra optional UW YAML keys to control every unlikely-but-possible behavior. The complexity could end up approaching the contents of aqm_cycled.def -- in which case, you might as well just write that, or have an extremely literal UW YAML that was a 1:1 mapping to that def. Painful. But if we can capture a subset of what ecFlow def files can say that's useful enough for likely UFS use cases, that could be a win. If a user has a strange use case that isn't worth codifying, they can always hand-code their .def file, or generate it with the Python API.

Back to the question (maybe): On line 623 I see family post, so I assume there's some kind of "family" object that can be created with the ecFlow API. Then, presumably we'd have some kind of start/stop/step description in YAML that describes the leadtimes. We might create a Python range from that spec, loop over the leadtimes, and then attach to the "family" object a "task" object for each leadtime. We might have some kind of base name, like jaqm_post for all post tasks, and then tack on the leadtime to form task name jaqm_post_f000, jaqm_post_f001, etc. Each "task" object would, in turn, get "edit" and "trigger" objects attached to it via API calls. I don't understand yet how these tasks get tied to executable code, but I guess there's some external logic that gets called to do jaqm_post work, perhaps configured via environment variables?

I'm not sure if I've answered any part of your question!

[benjamin-cash]

That's fair, and you've already given a lot of time to this considering that it is not on the team's plate for this sprint or PI (so far as I know). I think answering most of these questions will require input from stakeholders about what is required. Fortunately, it looks like that file is under active revision, specifically for compliance with NCO requirements from the commit messages. Maybe we could get one or both of the maintainers to meet with the group at some point to explain those choices and what the constraints are.