[Ray Core] "/tmp/ray" should not be hard coded for dashboard/observation.

[luxunxiansheng]

What happened + What you expected to happen

When I config the --temp-dir for the ray start , the output does redirect to the new folder. However, the generated artifacts found in the /session/latest/metrics/ are still using the "/tmp/ray " . Ideally all the places once using "/tmp/ray" should be replaced with the new temp-dir.

Versions / Dependencies

ray 2.8

Reproduction script

As described above.

Issue Severity

Medium: It is a significant difficulty but I can work around it.

[scottsun94]

@rkooo567 @rickyyx how hard is it fix this one?

[rkooo567]

I think 1~2 days of work... But the team is very overloaded lately.

@luxunxiansheng Would you be interested in contribution here?

[MengjinYan]

@scottsun94 Thanks for reporting the issue!

We tried with a local ray cluster setup with --temp-dir set. At the same time, we setup the local Grafana & Prometheus and execute some jobs on the local setup as well. Unfortunately, from what we see, all the files/artifacts are created under the new folder we specified in --temp-dir and we were not able to reproduce the issue you mentioned.

I'm wondering if you can help provide more detailed steps to reproduce the issue you are facing? It will be very help for our further investigation!!

Also, one thing to note is that, when starting the ray cluster, the default grafana/prometheus configs are generated with /tmp/ray as the base directory. You might need to update the paths before setup the services.

[scottsun94]

@scottsun94 Thanks for reporting the issue!

We tried with a local ray cluster setup with --temp-dir set. At the same time, we setup the local Grafana & Prometheus and execute some jobs on the local setup as well. Unfortunately, from what we see, all the files/artifacts are created under the new folder we specified in --temp-dir and we were not able to reproduce the issue you mentioned.

I'm wondering if you can help provide more detailed steps to reproduce the issue you are facing? It will be very help for our further investigation!!

Also, one thing to note is that, when starting the ray cluster, the default grafana/prometheus configs are generated with /tmp/ray as the base directory. You might need to update the paths before setup the services.

1. I didn't report this issue. @luxunxiansheng did.
2. I thought that @luxunxiansheng was referring to this "the default grafana/prometheus configs are generated with /tmp/ray as the base directory" in the original issue? Shall we change the generated configs based on --temp-dir?

[MengjinYan]

Thanks for the clarification, @scottsun94! And sorry for the mistag. I misunderstood the ask. I'll take a further look.