Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Invalid TaskState released #6191

Closed
mrocklin opened this issue Apr 24, 2022 · 0 comments · Fixed by #6404
Closed

Invalid TaskState released #6191

mrocklin opened this issue Apr 24, 2022 · 0 comments · Fixed by #6404
Labels
deadlock The cluster appears to not make any progress

Comments

@mrocklin
Copy link
Member

https://github.com/dask/distributed/runs/6148662781?check_suite_focus=true

Traceback (most recent call last):
  File "/Users/runner/work/distributed/distributed/distributed/worker.py", line 3991, in validate_task
    self.validate_task_released(ts)
  File "/Users/runner/work/distributed/distributed/distributed/worker.py", line 3962, in validate_task_released
    assert not any(ts.key in has_what for has_what in self.has_what.values())
AssertionError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/runner/work/distributed/distributed/distributed/core.py", line 625, in handle_stream
    handler(**merge(extra, msg))
  File "/Users/runner/work/distributed/distributed/distributed/worker.py", line 1727, in handle_free_keys
    self.transitions(recommendations, stimulus_id=stimulus_id)
  File "/Users/runner/work/distributed/distributed/distributed/worker.py", line 2635, in transitions
    self.validate_task(ts)
  File "/Users/runner/work/distributed/distributed/distributed/worker.py", line 3999, in validate_task
    raise AssertionError(
AssertionError: Invalid TaskState encountered for <TaskState 'ndarray-c380d2dc1cdf941cb55bb0ea78c929d7' released>.
Story:
[('ndarray-c380d2dc1cdf941cb55bb0ea78c929d7', 'ensure-task-exists', 'released', 'compute-task-1650839033.713147', 1650839033.725312)]

This came out of test_stress_scatter_death

cc @fjetter @crusaderky @gjoseph92

@mrocklin mrocklin added the deadlock The cluster appears to not make any progress label Apr 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
deadlock The cluster appears to not make any progress
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant