Restructure P2PShuffle extensions #7390
Conversation
distributed/shuffle/_utils.py
Outdated
| from __future__ import annotations | ||
|
|
||
| from typing import NewType | ||
|
|
||
| ShuffleId = NewType("ShuffleId", str) |
There was a problem hiding this comment.
maybe we can put this in the init directly. pretty important top level variable
| # circular dependency | ||
| from distributed.shuffle._worker_extension import ShuffleWorkerExtension | ||
|
|
||
| ShuffleId = NewType("ShuffleId", str) |
There was a problem hiding this comment.
I've moved this into _shuffle now since the P2PShuffleLayer is responsible for all ID- or naming-related things. Some helpers from the ShuffleSchedulerExtension moved here as well which allows us to rely on constants instead of hard-coded strings within methods.
Moving ShuffleId into __init__ would create some cyclic typing dependencies that I'm not too fond of.
Unit Test ResultsSee test report for an extended history of previous test failures. This is useful for diagnosing flaky tests. 18 files + 4 18 suites +4 8h 0m 2s ⏱️ + 2h 8m 51s For more details on these failures, see this check. Results for commit ba2b38c. ± Comparison against base commit c4af791. ♻️ This comment has been updated with latest results. |
ShuffleSchedulerExtensionandShuffleWorkerExtensioninto individual submodulesShuffleStatedataclass to encapsulate data previously tracked in various collectionsShuffleIdinto_shufflepre-commit run --all-files