-
Notifications
You must be signed in to change notification settings - Fork 5.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[core] Exception: Failed to read dashboard.err file #34504
Comments
I think it might be a path issue. I ran into issues with starting ray with your repro. Seems some parts of the ray wasn't handling relative path well (the Plasmastore) Could you try using the abs path for the temp dir and see if that works for you while I work on a fix for this? |
I also has same issue. |
Hi, I'm also facing the same issue. I'm using only one node and don't even need ray, only vLLM but internally it initializes a ray session and gets stuck indefinitely here:
Is there some way to disable ray in only vLLM scripts or mitigate this issue? |
cc @anyscalesam |
It works for me that I just uninstall |
Initial issue seems to no longer exists on latest version of ray, #36431 merged after this was opened also helps assure that this won't come up again now that we're always requiring absolute dir. @arshiya031196 Feel free to open a separate issue with the versions of the packages you're using and how you're running vllm if you're also still getting this issue on the latest vllm. |
What happened + What you expected to happen
Trying to run the following code for ray start:
And I keep getting the error:
Couldn't read dashboard.log file. Error: [Errno 2] No such file or directory: './temp_link/session_2023-04-17_18-41-09_060425_2337608/logs/dashboard.log'. It means the dashboard is broken even before it initializes the logger (mostly dependency issues). Reading the dashboard.err file which contains stdout/stderr.
and then:
Exception: Failed to read dashboard.err file: cannot mmap an empty file
I have checked that the
./temp_link/session_2023-04-17_18-41-09_060425_2337608/logs/
directory does exist but there is no dashboard.log file. I have an issue in launching the Ray cluster even if I set 'include-dashboard' as FalseI know that others experienced similar issues here (#26320) and I tried using the following fix:
However, my issue continues to exist.
Versions / Dependencies
Using linux
Python 3.10.6
ray 2.3.1
grpcio 1.49.1
Reproduction script
ray start --head
--port $RAY_PORT
--dashboard-port $((RAY_PORT + 1))
--include-dashboard True
--object-store-memory 10000000000
--num-cpus 0 --num-gpus 0
--temp-dir ./temp
Issue Severity
High: It blocks me from completing my task.
The text was updated successfully, but these errors were encountered: