Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Follow dask task API more closely? #151

Open
khaeru opened this issue Nov 12, 2024 · 0 comments
Open

Follow dask task API more closely? #151

khaeru opened this issue Nov 12, 2024 · 0 comments
Labels
discuss Under/solely for discussion

Comments

@khaeru
Copy link
Owner

khaeru commented Nov 12, 2024

Building on #149:

Dask recently introduced an internal classes like GraphNode, Task, etc. and uses these in its internal graph representation. This mirrors in some ways handling done in genno and downstream, e.g. message_ix_models.workflow.Step. For instance, it reflects a need or desire to attach the key to the task directly (rather than having (key, task) pairs only via a container like a dict).

We could consider whether to:

  • Generate dask.Task items within genno, so that dask's internal convert_legacy_graph() call does not need to transform the genno.Computer.graph.
  • Adjust genno.Key to be a subclass of one of the dask.typing.Key classes: str, bytes, float, int, or tuple of same. For instance, Key could be made a subclass of tuple with contents (name, (dims,), (tags,)).

However, most of these features are not advertised as part of the documented/public dask API, so it may be unwise to rely on their stability. Another approach (probably not desirable, because of maintenance overhead) would be to vendorize parts of dask that are used by genno.

@khaeru khaeru added the discuss Under/solely for discussion label Nov 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
discuss Under/solely for discussion
Projects
None yet
Development

No branches or pull requests

1 participant