[UI] When a Ray worker node dies, it makes Ray dashboard "cluster" page and "logs" page broken

[WeichenXu123]

What happened + What you expected to happen

When a Ray worker node dies, it makes Ray dashboard "cluster" page and "logs" page broken

Normal "cluster page" containing dead node should be like:

But now it becomes:

and when Ray worker node dies, "logs" page is completely broken and show nothing.

Versions / Dependencies

This bug is introduced in Ray nightly build version , Ray 2.35 version is good.

Reproduction script

1. install Ray nightly build version
2. start a Ray head node like:

ray start --head --node-ip-address=127.0.0.1 --port 9988 --dashboard-host=0.0.0.0

3. start a Ray worker node like:

ray start --address 127.0.0.1:9988 --block &

4. Find the Ray worker node processes group, and kill (-9) all processes in the group

ps -ef|grep "ray start --address 127.0.0.1:9988"  # Get process id of "ray start --address 127.0.0.1:9988" command
kill -9 -{xxx} # kill the process **group** of id printed in above command

5. Wait a few seconds, Open Ray dashboard cluster page. error occurs.

Checking nodes?view=summary request response, it got error like:

Traceback (most recent call last):
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-8ac7e00d-2d10-430a-b39d-ad4447d3a5ed/lib/python3.11/site-packages/ray/dashboard/optional_utils.py", line 224, in _update_cache
    response = task.result()
               ^^^^^^^^^^^^^
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-8ac7e00d-2d10-430a-b39d-ad4447d3a5ed/lib/python3.11/site-packages/ray/dashboard/modules/node/node_head.py", line 364, in get_all_nodes
    all_node_summary, nodes_logical_resources = await asyncio.gather(
                                                ^^^^^^^^^^^^^^^^^^^^^
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-8ac7e00d-2d10-430a-b39d-ad4447d3a5ed/lib/python3.11/site-packages/ray/dashboard/datacenter.py", line 171, in get_all_node_summary
    return [
           ^
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-8ac7e00d-2d10-430a-b39d-ad4447d3a5ed/lib/python3.11/site-packages/ray/dashboard/datacenter.py", line 172, in <listcomp>
    await DataOrganizer.get_node_info(node_id, get_summary=True)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-8ac7e00d-2d10-430a-b39d-ad4447d3a5ed/lib/python3.11/site-packages/ray/dashboard/datacenter.py", line 150, in get_node_info
    node_info["status"] = node["stateSnapshot"]["state"]
                          ~~~~^^^^^^^^^^^^^^^^^
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-8ac7e00d-2d10-430a-b39d-ad4447d3a5ed/lib/python3.11/site-packages/ray/dashboard/utils.py", line 435, in __getitem__
    proxy = self._proxy[item] = make_immutable(self._dict[item])
                                               ~~~~~~~~~~^^^^^^
KeyError: 'stateSnapshot'

Issue Severity

None

[WeichenXu123]

CC @rkooo567 any ideas ?

[rkooo567]

can you share the console log when this happens? Also cc @alanwguo @nikitavemuri

[WeichenXu123]

can you share the console log when this happens? Also cc @alanwguo @nikitavemuri

which log file do you need ? Ray node console log shows nothing abnormal.

and the UI logs page is broken.

[rkooo567]

I meant the UI log page (assuming there must be an exception there? )

[WeichenXu123]

I meant the UI log page (assuming there must be an exception there? )

no.. nothing displayed, it turns out to be an blank page

[WeichenXu123]

This is the console error when loading "logs" page. @rkooo567 any insights ?

[WeichenXu123]

This is the error when loading "cluster" page:

[WeichenXu123]

@rkooo567
The HTTP request nodes?view=summary failed with error:

Traceback (most recent call last):
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-8ac7e00d-2d10-430a-b39d-ad4447d3a5ed/lib/python3.11/site-packages/ray/dashboard/optional_utils.py", line 224, in _update_cache
    response = task.result()
               ^^^^^^^^^^^^^
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-8ac7e00d-2d10-430a-b39d-ad4447d3a5ed/lib/python3.11/site-packages/ray/dashboard/modules/node/node_head.py", line 364, in get_all_nodes
    all_node_summary, nodes_logical_resources = await asyncio.gather(
                                                ^^^^^^^^^^^^^^^^^^^^^
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-8ac7e00d-2d10-430a-b39d-ad4447d3a5ed/lib/python3.11/site-packages/ray/dashboard/datacenter.py", line 171, in get_all_node_summary
    return [
           ^
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-8ac7e00d-2d10-430a-b39d-ad4447d3a5ed/lib/python3.11/site-packages/ray/dashboard/datacenter.py", line 172, in <listcomp>
    await DataOrganizer.get_node_info(node_id, get_summary=True)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-8ac7e00d-2d10-430a-b39d-ad4447d3a5ed/lib/python3.11/site-packages/ray/dashboard/datacenter.py", line 150, in get_node_info
    node_info["status"] = node["stateSnapshot"]["state"]
                          ~~~~^^^^^^^^^^^^^^^^^
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-8ac7e00d-2d10-430a-b39d-ad4447d3a5ed/lib/python3.11/site-packages/ray/dashboard/utils.py", line 435, in __getitem__
    proxy = self._proxy[item] = make_immutable(self._dict[item])
                                               ~~~~~~~~~~^^^^^^
KeyError: 'stateSnapshot'

[WeichenXu123]

@anyscalesam Who should we assign this bug to ? thanks! This is a critical bug I think.

[jjyao]

@rynewang it's likely due to the recent dashboard optimizations.

[jjyao]

I think it's due to #47367